Office 2013 on Windows 10, CPU usages through the roof
In a current project we face an issue, of course our sales department would like to only talk about our successes but I think sharing our issues will show how we deal with these issues. Helping others before they jump into the same issues is good I think. In this project we have a VMware Horizon 7 Windows 10 VDI environment. Our environment is setup with the following components:
- VMware Horizon View 7.02
- RES ONE Workspace 2016 SR1
- Microsoft Office 2013 SP1
- Skype for Business 2016
- Palo alto Traps
- 3PAR SSD storage
- VMware vSphere 6.x
- Some hardware to run the stuff on, HP ones with 48 logical cores and 256RAM
So in the morning when users log on, the 3PAR has to deliver a couple of thousand IOPS… it barely wakes up. The virtual machines when they log on use about 80% if their memory but after things settle down memory settles to about 50-70% and nearly goes up. Of course that depends on the users activities but as we looked pretty hard at the environment last days it was steady all day long.
We have an issue with Microsoft Office 2013 in this environment, we did some debugging… actually we did a lot of debugging and we found that there are left over Office processes running that are clogging the processor time. Because of some software the customer uses, its from a company know for lacking development, we needed to run Office 2013. The environment is also running Skype for Business which of course is a higher version.
The issue we noticed was that Microsoft Excel, Word or Outlook would take 80% or more of CPU time in a VDI desktop. These desktops would eat up 5000MHz on a host and when a few desktops would join the group the host would raise an alert. It made sure the hosts were overloaded and users complained about slowness due to CPU Wait times.
Below you see a screenshot of Outlook taking 96% of CPU time.. and trust me this user (she is called mpenni… and I the only one who find that funny) is not using Outlook at all. She is working in another application unaware Outlook is doing this.
When looking in the VMware environment this desktop was taking 5600MHz of CPU time.
We looked that everything that starts an Office process;
- Auto start when starting a session – ruled out, process is killed nicely when closing the application.
- Preview of Outlook when clicking on a file – ruled out, process stays open as long as Outlook is open
- Files received in Skype – ruled out, processes are killed when application is closed
- Office calls from 3rd party applications – BINGO!
When users call Office applications from 3rd party, so when a application requests Outlook to send an email something interesting happens. Not the current Outlook process was started but a new process started. This user was entering multiple entries in the system and each one of them would trigger a new Outlook process. When we first looked we noticed that she had 8 Outlook processes open next to the one she was working in.
All 8 processes of Outlook were not used and just using resources, for some reason the requesting application was unable to close the process it started or Windows was unable to stop it or Outlook was unable to close itself.. this is still something we are looking into.
So this is where we stand, we know where the issue is coming from. We just don’t know yet why it is there, we need to find the reason why the processes are staying open. We are looking to deploy Office 2016 but there are a lot of other apps that need Office and the plugins and templates are not inter-exchangable just like that… life is not that simple.
Today we took a day to take a good look at the image and we found that there were some things that could be improved.
One of the things was that there was a Adobe Reader DC version being used that was not supported on Microsoft Windows 10. The version being used was 2015/333 and the version supported is 2015/339. So we now updated the image (dit a complete reinstall) and added the patch to make Adobe Reader DC 2015 version 339.
Also we saw that there was Adobe Pro X being used, after asking around only one guy might be using it, so it was stripped from the image. Adobe Pro is known for the plugin and integration and when its only used for 1 person I don’t want it bugging 700 others.
Next was a few viewers like Irfanview, which they use for TIFF files. The way Irfanview was installed was that it was assigned to all file types. I think Windows is very capable of handeling most images, so I reinstalled it and set the filetype to only TIFF.
Some other applications like Filezilla, GIMP and some codecs were found that were not needed. All were left out o the rebuild, of course we added Filezilla for them with now in a portable form.
We started this project months ago with a small list of applications, they were out of our scope. Now seeing what is deployed in the image we talked about layering or virtualizing applications. There are a large number of heavy applications that are only used by a dozen users. all these apps are in one image now, far from ideal.
We looked at the setup of the environment, we ain’t got no IOPS issue that’s for sure and memory is not an issue either. We had some spare memory so the new VDI’s are deployed with 6GB, 2GB more than before. Of course more memory is not solving CPU shortage but as we got the memory what ain’t hurting might be helpful.
Hardware – clients
Next we looked at the client side, the users work in SFF clients. So they got a basic SFF Core i5 with 4GB RAM and two 1920×1200 resolution screens. Those screens are a bit different than we sized with. We noticed also with Office 2013 that the fonts are a bit off as this resolution is high. Office likes a lower resolution but we can’t offer that as that is not the ideal resolution for the screens.
Conclusion after three days of tuning
So after three days we are at a point that we know for sure that:
- The hardware the VDI is running on is doing fine, no issues there.
- The Tuning of Windows 10 as described in this article – link – is ok. I enabled Windows search for now and looked at the other settings that were mentioned in the comments.
- The applications installed in Windows 10 are up to date and supported, freaky apps are removed.
In general what I wrote about resources and Windows 10 is valid. Windows 10 needs about double the resources Windows 7 did. If you don’t take that in account you will have unhappy users. Of course the number depends on the applications, hardware, client hardware, video handling and so on but a thumb rule is that you need at least 1000-1200 MHz to run a basic desktop. If you add Skype for Business, high resolution screens you need to add some more power.
Is this a bummer? Its not, we started the project last year April with a design, the design has changed since then often, during that time the customer realized they wanted Skype for Business, different client hardware, very heavy application on-premises instead of a cloud version and so on. They know and we know the game has changed and with a different game the rules are different.
Next up is going back to the drawing board and see what kind of resources we miss. The good thing is, we just need hardware. licenses are covered. Hardware is easy.
What will we do from here?
We have deployed a Office 2016 desktop pool and the customer is testing this so that we can go with that version asap. Office 2013 is not the best option on Windows 10 I noticed and read. They use Skype for Business that comes with Office 2016 so we have a mismatch in Office versions. I strongly believe matching the Office versions for all products is better. As we notice in a small test Office 2016 runs better and smoother in Windows 10.
I deployed two pools today, one with Office 2013 and one with 2016.. next week more metrics.
one more test I’m gonna setup is a pool with vGPU, perhaps one of the freaky applications is pushing the CPU for graphics. for now we just use an APEX 2800 for offloading but that is not a vGPU.
I’l keep you posted.
Main thing you should take from this update is; Check if apps are supported, check if you use plugins and if they are needed/supported/doing crazy things/don’t install useless apps and if you need them virtualize or layer them.
Lately we got the environment running at a better level, we increased the RAM to 6GB instead of the 4GB we had initially. With the 6GB the CPU’s on the hosts are not overloaded anymore or less at least. With the 6GB the Windows 10 VDI desktops run smoother and it seems that we can cope with this. Of course only 50 – 55 desktops per host is not a good scaling of your environment so that’s something still open.
We did a lot of tuning in the BIOS of the VDI desktop and a lot of fiddling around with the virtual machine settings but none really did anything with the performance. in the BIOS we turned of all serial and parallel ports for a starter and anything else that had anything to do with communication we could think of.
What we are still investigating is that a data-warehouse system they use (not really data-warehouse but got it comes close) where they store all citizens requests and cases is calling of Office applications like Outlook and Word to push out emails or create documents. What we noted is that it will call Outlook to open but once it is done seems to fail to send the request to close it. It’s either that or that Outlook for some reason never sends the signal that it is finished. This is under investigation but hard to solve if the vendor of the system is not on the same level as you and me on VDI.
At this moment if someone asks me if Windows 10 is VDI ready, I say that Microsoft has a FAT client only policy and Windows 10 is designed to run in a FAT client environment. I’ve seen environments that run smoother but if I look at the CPU usage there (MHz of course) it still is more that a heavy user would use and that is insane. Of course with a shit load of hardware you get your performance but with a shit load of hardware you can run anything… I guess a blog about the Windows10 strategy of Microsoft is due. Luckily I noted that somehow I got myself registered for a Microsoft Tech summit later this month in Amsterdam, lets see what they have to say on this.