VMware vShield for Endpoint & TrendMicro Deep Security
Implementing new technology can be tricky, during a project we encountered some strange behavior with VMware vShield for endpoints & TrendMicro Deep Security.
At first we were in the dark about the whereabouts of issues we were experiencing. the issues we noticed were performance related. Logon times were slow and starting applications like word could take up to 30 seconds. In a pilot environment we never noticed these troubles, issues seemed to appear out of the blue.
The first thing you do when troubleshooting this kind of performance issues is looking at the differences. One main difference with the pilot environment was that Trend Micro Deep Security anti-malware had been switched on. We noticed strange behavior also once in the pilot when we tested this, it broke the RDS licensing process.
The Environment
| VMware Tools version | 8.6.5.13851 build 731933 |
| Microsoft Windows | 2008 R2 |
| Citrix XenApp | 6.5 |
| Trend Micro Deep Security | 8.5 |
| VMware vSphere | 5 |
Issues we experienced were;
- Copying between virtual machines was slow;
- Copying inside the virtual machine was slow;
- Starting applications was slow.
First we tested with the Anti-Malware function in TM off in the source and/or destination virtual machine to see if that was causing the issue
Now we established that it had something to do with Anti-Malware and therefore with Trend Micro.
| Click on published app:desktop | 12:13:45 | |
| RES initializing | 12:13:55 | |
| All completed | 12:14:28 | 40sec |
| Load Intranet | 12:14:35 | 60sec |
| Start Word | 14sec | |
Production without Anti-malware
| Click on published app: desktop | 12:22:00 | |
| RES initializing | 12:22:11 | |
| All completed | 12:22:49 | 31sec |
| Load Intranet | 12:22:46 | 46sec |
| Start Word | 4sec | |
Pilot environment
| Click on desktop | 12:36:15 | |
| RES initializing | 12:36:21 | |
| All completed | 12:36:41 | 20sec |
| Load Intranet | 12:36:40 | 25sec |
| Start Word | 3sec | |
Pilot was faster in every way… hmm that is not good.
Problem identified
We did some test… and found that the vShield driver installed with VMware Tools was the issue. these tests were conducted by Erik van Veenendaal (PQR).
Somehow this driver creates overhead or has issues talking with Trend Micro Deep security. Without the driver no issues were reported but that wasn’t the way we wanted to run a production environment.
New Driver
VMware also got the message that there was an issue, a call was opened. and after a while we got a new vShield driver, a unsigned driver. We did some new tests to see the differences.
3299 Files
761 Folders
Old driver
99.4922193
96.9084219
No driver
61.8032854
60.5241506
New patch driver
62.1181336
62.9257556
As you can see, with the new driver we get a very huge improvement.
A newer driver that is signed
We got a newer driver, perhaps the one that will be integrated in VMware tools. Who knows. We did a final test to see the performance.
Logon times
| Test | Time |
| Old driver and TM AM on | ~60 sec |
| Old driver and TM AM off | ~45 sec |
| New driver and TM AM on | ~54 sec |
| New driver and TM AM on + some changes | ~45 sec |
| New driver driver and TM AM off | ~33 sec |
| New driver not loaded (TM AM on or off no difference) | ~33 sec |
Startup times test
| Test | Boot Time | Bytes Read |
| Start streamed Citrix XenApp server with TM AM on | 45 sec | 472.357 KB |
| Start streamed Citrix XenApp server with TM AM off + changes | 32 sec | 237.284 KB |
| Start streamed Citrix XenApp server with TM AM off | 25 sec | 191.005 KB |
Starting MS Word
| Test | Time |
| New driver and TM AM on in XenApp + RES Workspace Manager session | ~20 sec |
| New driver and TM AM off in XenApp + RES Workspace Manager session | ~4 sec |
| New driver and TM AM on, on the server console | ~4 sec |
Hopefully with 5.1 the new VMware tools will be integrated and Trend Micro Deep security can be used again without issues.
We’re waiting……….
p.s. one more thing i need to add, the numbers you see here are not the defenitive numbers, the reason is that we had these issues while building the production environment. we had to do tuning etc and couldn’t run on a ideal environment. The numbers should go down when we finish tuning… still we need a stable driver to get these numbers
[…] Endpoint Driver available to improve Deep Security 8 performance 7 09 2012 Thanks to http://www.joulupukki.nl/wordpress/?p=523 for alerting me to an issue I didn’t even realise I […]
Thanks for that, I did some further digging and saw this here: http://www.vmguru.nl/wordpress/2012/10/trend-micro-deep-security-9-on-its-way/
Performance improvements
In the previous version load balancing the Deep Security Manager was not optimal. Often one of the nodes did most of the work while the others were doing nothing. With Deep Security 9 the load can be more evenly distributed across the managers (If I remember correctly in combination with an external load balancer).
File scanning is also improved. In DS8 files were scanned each time they were accessed, regardless of the fact that the files were scanned on another VM. If I understood the Trend Micro people correctly the scanning is now based on a hash. When a file is accessed in VM1 a hash is calculated and the file is scanned. If the same file is accessed it won’t be scanned again if the hash is the same. Preliminary figures indicate that it could be a tenfold performance boost. Deep Security 9 now also uses ESXi level caching and deduplication.