One of the most annoying events in a virtual farm is not having the root password of a host. Such was a recent case of a VMware 4.0 ESXi host. I tried all known passwords and variants and asked around but none worked and of course nobody changed the password.

It would have been easy if this was an ESX host as all you would need to do is to crash boot the system and place it into single user mode. The problem with ESXi is there is no GRUB loader so this is not possible.

If you look on the VMware site, you will find instructions to reset the root for ESX but for ESXi you have a nice message.

Reinstalling the ESXi host is the only supported way to reset a password on ESXi.  Any other method may lead to a host failure or an unsupported configuration due to the complex nature of the ESXi architecture. ESXi does not have a service console and as such traditional Linux methods of resetting a password, such as single-user mode do not apply.

It’s fine for VMware to suggest this approach but the problem is you will loose all your VMs. The author probably thinking of ESX where backups are rather simple. You can do that for ESXi but even if you have backups, it’s still time to reinstall and restore them and the user may not like the time needed.

The good thing is you don’t have to do this as there are two ways you can recover a lost password.

1) You can run a repair of the OS.

2) You can use a Linux Live CD.

I thought the easiest approach would be to run a repair since all it would do was reset the configuration of the system and leave the VMFS datastore alone.  The VMs would be forgotten but you can add them back with the client.

One of the things you will read from the documentation is that the VMFS datastore is preserved if it’s still the same patitition when you set up the host or its on a another disk.

This is not the case as it will be explained later.

There is a size issue if the VMFS location is on the boot disk and is located beyond the 900 MB partition and the partition table is corrupt. The VMFS datastore can not be recovered automatically by the repair process and you will need help from VMware.

This was not the case for this host.

Make sure you use the original install CD for the repair. If you don’t have it, get the version number from the login screen on the host and get the appropriate ISO from VMware and burn a new CD.

You will need to powercycle the host so inform your users.

 1) Insert the ESXi 4.0 Installation CD
 2) Power Cycle the machine. Depending on your how your BIOS is set for boot order 
    make the change for the CDROM via setup or boot option if your BIOS has it.
 3) The Installation will proceed.  Don't worry about it installing as a new machine 
    as the Repair option will appear. Press R to repair
 4) Accept the EULA by pressing F11.
 5) You will get a screen to select the disk with the OS.
    Now VMWARE defaulted all disks and if you read about how the VMFS partitions are 
    left alone this may not be the case.

    As you will also read:

     If you do not choose the same installation disk, the damaged ESXi 4.0 image 
     is not fixed and a new image is installed.

    Such was the situation I found.  Rather then selecting the OS disk I just 
    defaulted and went on.

    For your situation, highlight the OS disk and press Enter.

    Here you will get the confusing warning of the data on the selected disk is 
    about to be overwritten and if there were no changes to the partitions, your 
    VMFS data stores will be preserved.

    Again such is not the case if you leave all drives selected.

 6) If you have the right disk, press Enter.  Otherwise, press the Backspace and 
    select the correct disk.
 7) Now you will get your last chance to back out as you will get a confirmation
    request. If all is in order, press F11 to start the recovery.

The process will run and you will end with one of the two messages:

Repair Complete
The ESXi 4.0 image was repaired successfully and the partition table was restored. The installer recreated the partition table to recover your VMFS partitions or custom partitions. The repair operation added these entries in the partition table.

If you have this then you simply reconfigure the general configuration and then add the missing VMs

Repair Incomplete
The ESXi 4.0 image was repaired successfully, but the partition table could not be restored. The installer could not recreate the partition table to recover your VMFS partitions or custom partitions. You must manually add the partition entries to the partition table to recover your data. Call VMware support for help.

This was the message I received and it was misleading. It completed but nothing was changed as I still could not login via the root account.

Now comes the problem with ESXi, it’s not supported since it is free to use. This would be a paid support call and it would be a critical problem if it was decided to contact support.

What to do?

Before any attempt of recovery, I needed access to the system. Since the host did boot, the original OS was still usable.

To be continued.