Can’t Fork – VMware ESXi 5.5


Can’t Fork – VMware ESXi 5.5

Getting a message from a customer with the title “Heap
globalCartel-1 already at its maximum size of 7869288. Cannot expand. Could
not start VMX: msg.vmk.status.VMK_NO_MEMORY
” is not something you look forward to, at least I don’t.
I got this message yesterday stating he can’t start the VDI Golden image and he sees this error with several VDI desktop also. So another day in the world of troubleshooting… 
A quick Google search leads you to some, not to many, articles including the knowledge base article of VMware. The article is found here.
@IBeerens also wrote a blog about this, he wrote a script to determine the versions of the driver used on all the hosts, if you need a script to find the versions take a look here.

Debug

So I went to the customer and took a dive into the issue, first I logged on through ILO to see what the console was reporting.
That looks like an issue to me, the environment is still running though users are mildly impacted at this moment. The issue is around HP-AMS, which provides ILO with a richer data set that is OS specific (as mentioned on the HP site).
First question that pops in my mind is, do I need it… hell no, let’s remove it.
Wait… let’s first see if there is a solution for this, perhaps VMware fixed it already.
There are two solutions to fix this it seems.
One of them is to update the driver to version 10.0.1. To do this you need to download the driver. If you visit the HP site you will notice that this driver is not downloadable there. To find the driver you need to visit the vibsdepot, browse to the following address and extract it.

After you downloaded the bundle and extracted it, you need to install it.
  1. Download the zip file to the data store and create folder called update to make it easier
  2. Shutdown all virtual machine or move them to another host
  3. Enable SSH access to the host
  4. Access the host with SSH
  5. Enter  maintenance mode: esxcli system maintenanceMode set –enable on
  6. Change directory to the data store with the update: cd /vmfs/volumes/datastore/updates
  7. Execute the update: esxcli software vib update –v /vmfs/volumes/…./hp-ams-550.10.0.1-07.1198610.vib
  8. Reboot the host
  9. Restart virtual machines
The second option is to use the customized VMware ISO for HP, the driver is included there. The last version is from 04-11-2014. you can find this ISO or ZIP from here.
Of course deploying this zip would be the same as mentioned before only with the -depot option instead of the -v option and of course a reference to the zip file. you could add -dry-run to simulate it first but you gotta go someday.
  1. Download the zip file of the customized HP VMware ESXi 5.5 U2 image to the data store and create folder called update to make it easier
  2. Shutdown all virtual machine or move them to another host
  3. Enable SSH access to the host
  4. Access the host with SSH
  5. Enter  maintenance mode: esxcli system maintenanceMode set –enable
  6. Change directory to the data store with the update: cd /vmfs/volumes/datastore/updates
  7. Execute the update: esxcli software vib update –depot /vmfs/volumes/…./VMware-ESXi-5.5.0-Update2-2068190-HP-5.77.3-Nov2014-depot.zip
  8. Reboot the host
  9. Restart virtual machines
The third option, is there a third option?? Yes there is, is to remove the HP-AMS driver and do a reboot. This is the least favorite option but surely an option. It’s the least damaging option for anything new installed is a new risk for new bugs..
So now we just need a maintenance window to execute one of the options, a bit of luck, lot’s of coffee and we’re ready to go. Hope this helps some of you also running into this issue.

Leave a Reply

https://tracking.cirrusinsight.com/869c29e2-3a9b-48c5-9232-0b95e7993ae8/controlup-com-pixel-php