VMware vShield for Endpoint & TrendMicro Deep Security


VMware vShield for Endpoint & TrendMicro Deep Security

Implementing new technology can be tricky, during a project we encountered some strange behavior with VMware vShield for endpoints & TrendMicro Deep Security.

At first we were in the dark about the whereabouts of issues we were experiencing. the issues we noticed were performance related.  Logon times were slow and starting applications like word could take up to 30 seconds. In a pilot environment we never noticed these troubles, issues seemed to appear out of the blue.

The first thing you do when troubleshooting this kind of performance issues is looking at the differences. One main difference with the pilot environment was that Trend Micro Deep Security anti-malware had been switched on. We noticed strange behavior also once in the pilot when we tested this, it broke the RDS licensing process.

The Environment

VMware Tools version8.6.5.13851 build 731933
Microsoft Windows     2008 R2
Citrix XenApp 6.5
Trend Micro Deep Security 8.5
VMware vSphere    5

Issues we experienced were;

  • Copying between virtual machines was slow;
  • Copying inside the virtual machine was slow;
  • Starting applications was slow.

First we tested with the Anti-Malware function in TM off in the source and/or destination virtual machine to see if that was causing the issue

With the Anti-malware function on the speed we reached to copy files was between 4 and 20MBps.

With the Anti-malware function switch off at the destination virtual machine we reached 80MBps. (excluding the folder also did the trick.)

With the Anti-malware function switched off at both side we reached >100MBps

Now we established that it had something to do with Anti-Malware and therefore with Trend Micro.

Trend Micro is talking to the virtual machine through VMware Tools vShield driver. we quickly looked at the pilot enviroment to see if we had this setup running there also. We didn’t, and didn’t experience the issue there.

We did some additional tests…. in production and pilot.

Production with Anti-malware switched on

Click on published app:desktop12:13:45
RES initializing12:13:55
All completed12:14:2840sec
Load Intranet12:14:3560sec
Start Word14sec

Production without Anti-malware 

Click on published app: desktop12:22:00
RES initializing12:22:11
All completed12:22:4931sec
Load Intranet12:22:4646sec
Start Word4sec

Pilot environment

Click on desktop12:36:15
RES initializing12:36:21
All completed12:36:4120sec
Load Intranet12:36:4025sec
Start Word3sec

Pilot was faster in every way… hmm that is not good.

Problem identified

We did some test… and found that the vShield driver installed with VMware Tools was the issue. these tests were conducted by Erik van Veenendaal (PQR).

Somehow this driver creates overhead or has issues talking with Trend Micro Deep security. Without the driver no issues were reported but that wasn’t the way we wanted to run a production environment.

New Driver

VMware also got the message that there was an issue, a call was opened. and after a while we got a new vShield driver, a unsigned driver. We did some new tests to see the differences.

Fileset 2: XenApp installation files

4,43Gb

3299 Files

761 Folders

Old driver

99.4922193

96.9084219

No driver

61.8032854

60.5241506

New patch driver

62.1181336

62.9257556

As you can see, with the new driver we get a very huge improvement.

A newer driver that is signed

We got a newer driver, perhaps the one that will be integrated in VMware tools. Who knows. We did a final test to see the performance.

 Logon times

TestTime
Old driver and TM AM on~60 sec
Old driver and TM AM off~45 sec
New driver and TM AM on~54 sec
New driver and TM AM on + some changes~45 sec
New driver driver and TM AM off~33 sec
New driver not loaded (TM AM on or off no difference)~33 sec

Startup times test

TestBoot TimeBytes Read
Start streamed Citrix XenApp server with TM AM on45 sec472.357 KB
Start streamed Citrix XenApp server with TM AM off + changes32 sec237.284 KB
Start streamed Citrix XenApp server with TM AM off25 sec191.005 KB

Starting MS Word 

TestTime
New driver and TM AM on in XenApp + RES Workspace Manager session~20 sec
New driver and TM AM off in XenApp + RES Workspace Manager session~4 sec
New driver and TM AM on, on the server console~4 sec

Hopefully with 5.1 the new VMware tools will be integrated and Trend Micro Deep security can be used again without issues.

We’re waiting……….

p.s. one more thing i need to add, the numbers you see here are not the defenitive numbers, the reason is that we had these issues while building the production environment. we had to do tuning etc and couldn’t run on a ideal environment. The numbers should go down when we finish tuning… still we need a stable driver to get these numbers


2 Responses

  1. […] Endpoint Driver available to improve Deep Security 8 performance 7 09 2012 Thanks to http://www.joulupukki.nl/wordpress/?p=523 for alerting me to an issue I didn’t even realise I […]

  2. Rob F says:

    Thanks for that, I did some further digging and saw this here: http://www.vmguru.nl/wordpress/2012/10/trend-micro-deep-security-9-on-its-way/

    Performance improvements
    In the previous version load balancing the Deep Security Manager was not optimal. Often one of the nodes did most of the work while the others were doing nothing. With Deep Security 9 the load can be more evenly distributed across the managers (If I remember correctly in combination with an external load balancer).
    File scanning is also improved. In DS8 files were scanned each time they were accessed, regardless of the fact that the files were scanned on another VM. If I understood the Trend Micro people correctly the scanning is now based on a hash. When a file is accessed in VM1 a hash is calculated and the file is scanned. If the same file is accessed it won’t be scanned again if the hash is the same. Preliminary figures indicate that it could be a tenfold performance boost. Deep Security 9 now also uses ESXi level caching and deduplication.

Leave a Reply

https://tracking.cirrusinsight.com/869c29e2-3a9b-48c5-9232-0b95e7993ae8/controlup-com-pixel-php