How to recover your VMware Horizon view environment when someone killed the vCenter database


How to recover your VMware Horizon view environment when someone killed the vCenter database

Today was one of those days you wish weekend was there after the day ended. I had an appointment with the customer to finish the VMware AirWatch environment implementation. Arriving in the morning I heard that no one could log on. quickly opening a browser and looking at the administrator console showed the issue. 300 desktops in error mode and a red alert next to the VMware vCenter server. The administrators informed me that they asked someone to do some work on the vCenter database. The database got corrupted a bit due to several power outages in the past 6 months. The recovery of the database resulted in data loss and now VMware vCenter was not operating correctly. Mind you this vCenter server is still a Windows server based one with a external SQL database.

Without a VMware vCenter database you can’t recover desktops having issues. VMware Horizon View is dependent on the vCenter server. So my planning changed, no AirWatch but recovering Horizon View. I thought it might be helpful to write little blog about how to solve this kind of disaster. There are several thoughts that crossed my mind and I’d like to share them with you.

The root cause

The root cause for all the desktops in error state was the VMware vCenter not functioning properly. My first thought was to restore the vCenter database but the customer didn’t trust the database backup. The power outages did a lot of damage and they wondered if we would restore a bad database.

The symptoms

The symptoms were that all desktops, pooled ones that refresh on logout, were in error state and unusable. The VMware Horizon connection servers couldn’t connect to the vCenter server, the server was in error state.

Communications

For who is wondering let me go into the communications a bit. The Horizon View Connection server connects to the vCenter server and the Composer server. The Composer handles the desktops pools and acts as the director to make sure you get the desktops, the desktops are in the Active Directory and so on. The vCenter server is used to deploy the virtual machines, power on, power off and so forth. Virtual machines are stored in the Connection server LDAP database together with the desktop pools. The same desktops are also stored in the Composer database, in my case a SQL database. The vCenter server database also holds the data of the virtual machines.

There is one rule in a VMware Horizon environment you have to live by, NEVER, NEVER remove a virtual machine from the vCenter side. VMware Horizon View will take care of that for you. If you don’t follow that rule your database will be inconsistent. So to recover you need to make sure the databases are consistent.

Thoughts

My initial thoughts were, how can I recover the environment without having to dig into the databases? Let me show you what crossed my mind;

  • Deleting a desktop pool can’t be realised without the  vCenter server.
  • Removing a virtual machine can’t be realised without the vCenter server.
  • Recovering a vCenter database (needed to deploy or recover VM’s) is not a walk in the park
  • Virtual machines are stored in the LDAP database, the composer database and the vCenter database.
  • Messing up in one of the databases will break everything resulting in a complete reinstall.

So these were my thought, I took a coffee and decided that cleaning the database was the best approach. I also decided to leave the vCenter alone and deploy a new VMware vCenter appliance.

Steps to recover

Let me walk you through the steps to recover from a lost vCenter server.

  1. Deploy a new vCenter server. I deployed the vCenter Appliance and used the internal database.
    1. Make sure you add a DNS entry for the new appliance first before deploying or you will be deploying twice…. like I did.
  2. Clean up the LDAP Database on the connection server
    1. Connect to the Connection server
    2. Open ADSIEdit
    3. Click on Action and Connect, name it e.g. cleanupdesktop
    4. Select the distinguished Name and type: DC=VDI,DC=VMware,DC=INT
    5. Type, at the computer field, Localhost:389
      1. Desktop pools
        1. Next you browse to Server Groups, within Server groups are the desktop pools.
        2. Delete all the desktop pools in the Server groups.
        3. Next you browse to the Applications, the dekstops won’t be gone from the Administrative console unless you also delete them here.
      2. Desktops
        1. Next you browse to the Servers (they are not server but the desktops are stored here)
        2. Delete all the desktops here.
    6. Now your desktops and desktop pools are removed from the LDAP database.
    7. Only do this in a disaster, if you just need to delete one desktop find the article to do this, you need to create some queries to find the desktop.
  3. Clean up the Composer database on the SQL server
    1. Open the SQL management console
    2. Browse to the VMware Horizon composer database
    3. Browse to the Tables
    4. Select table dbo.SVI._VM_Name
      1. Select edit first 200 rows
      2. Delete all of them
      3. If you have more dekstops to remove, repeat until they are all deleted
    5. Select table dbo.SVI_Computer_Name
      1. Select edit first 200 rows
      2. Delete all of them
      3. If you have more dekstops to remove, repeat until they are all deleted
    6. Select table  dbo.SVI_SC_PDISK_INFO
      1. Select edit first 200 rows
      2. Delete all of them
      3. If you have more dekstops to remove, repeat until they are all deleted
    7. Select table dbo.SVI_SC_BASE_DISK_KEYS
      1. Select edit first 200 rows
      2. Delete all of them
      3. If you have more dekstops to remove, repeat until they are all deleted
    8. Select table  dbo.SVI_TASK_STATE
      1. Select edit first 200 rows
      2. Delete everything here, often this is empty.
    9. Select table dbo.SVI_SIM_CLONE
      1. Select edit first 200 rows
      2. Delete all of them
      3. If you have more dekstops to remove, repeat until they are all deleted

Again if you just need to delete one desktop, don’t do it like this. With one desktop to delete you need to find the ID and delete just that one. I was doing a recover from a disaster and so everything was to be deleted.

After these steps your environment is clean again, next step is to build the new environment. Follow the next steps to bring the life back in your environment.

  1. vCenter server / appliance
    1. Browse to the vCenter vSphere client page
      1. Create a Datacenter, Cluster, add licences and you’re good to go.
      2. Add all the hosts from the old environment (mind you the hosts are still connected to the old vCenter server)
      3. Delete all the stale virtual machines (except perhaps the golden images) from the hosts.
        1. You might have to power them off first as they are all started)
    2. Open the VMware Horizon administrator page
      1. Browse to Server configuration
      2. Open the TAB vCenter server
      3. Add the vCenter server and the composer so that in the end you have two vCenter servers listed.
      4. Delete the old vCenter server entry
    3. Golden Image
      1. Make sure the Golden image you need to deploy the desktop pools is available on the new vCenter environment
    4. Go back to the VMware Horizon administrator page
      1. Browse to the Catalog
      2. Deploy a new desktop pool or desktop pools

After a few minutes, depending on your resources desktops will be available and you recovered from a disaster. Get a coffee you deserved it 🙂


6 Responses

  1. Jeffy Gezzy says:

    WOW, what a procedure. I have been Using XenDesktop with PVS for years and if you deleted a VM from vCenter you just received an error. You could then easily deleted it in Desktop Studio.
    You would think VMware would get their act together and have an easier solution if a VM was removed with vCenter.

    • Rob Beekmans says:

      Yeah it was a bit of work. With VMware normally (when you don’t kill the vCenter) are fine, it will recover itself. seen it happen often that it would delte a vm and recover.
      This was exceptional as the vcenter was unrecoverable and so the link between the vm running and where it was running was lost.

      A well lesson learned.

  2. Typo in: DC=VDI,DC-VMware,DC=INT

    Should be: DC=VDI,DC=VMware,DC=INT

  1. April 5, 2017

    […] Read the entire article here, How to recover your VMware Horizon view environment when someone killed the vCenter database […]

Leave a Reply

https://tracking.cirrusinsight.com/869c29e2-3a9b-48c5-9232-0b95e7993ae8/controlup-com-pixel-php