Tag Archive: Troubleshooting

One thing I have noticed on the Net; information about Citrix can be lacking at times.

I had an instance where a resource pool was established and the thinking behind it was wrong. I wanted to destroy it. I started Xencenter and accessed the doomed pool. I removed all the extra Xenserver hosts by simply right-clicking the entry and selecting “Remove Server from pool”

When it came time to kill the master, there wasn’t an option. I tried the xe options of eject and forgetting but they gave errors.

As mentioned, the Net was lacking on information. So I basically started hunting around thinking “it can’t be a reload!” I didn’t want to go through patching again.

I looked at the Xencenter window tab for Pool and sure enough there was an interesting entry!

Make into standalone server

I selected the master host, clicked “Pool” and selected the option. It ran really fast and the pool entry disappeared with the server showing up at the bottom of list of resources. I expected it to reboot but it didn’t. I gave it a reboot for good measure and then added the ex-master as a host to the correct pool.



My adventures with Xenserver found a couple interesting problems today. The first I will leave for another post. This one involved trying to install an update: XS65E1003.

I was trying to install it via the updates wizard and when it came time to run the Prechecks; it would fail with the message:

Checking server side status ... failed with an unknown error.

Ever so helpful.

I ran the update a couple more times just to see if there was a bad download but the error continued.

The messages log didn’t show anything obvious then again I could have missed it.

I checked the Net to see if there was anything similar especially for the unknown error. I found similar errors but not exact.

I was reading through a problem with xenserver 6.2 and saw an interesting suggestion. From the shell enter:

xe vm-cd-eject --multiple --force

I figured why not? It couldn’t hurt as this pool was not in service yet.  It gave a few errors. Nothing is the CD/DVD drive and couldn’t talk to a host. I did notice there wasn’t a message for each system.

I re-ran the install and sure enough; it passed the prechecks.

Strange but expected considering Xenservers error messaging at times.

Normally, I would give a link to where I found the command to credit the person but for the life of me; I can’t find the page anymore. If I find it later, I will add a link.

I am migrating virtual machines from VMware workstation to our cluster. People were trying to save money and use the workstation product instead of getting another cluster.

I am using VMware vCenter Converter Standalone and one migration died. After correcting a couple things; I tried to rerun but received the following error:

A general system error occurred: Unexpected element tag “ConvertManager” seen while parsing serialized DataObject of type converter.AgentInstanceContent at line7, column 63 while parsing return value of type converter.ServiceInstanceContent, version converter.version.version2 at line 7, columen 0 while parsing SOAP body at line 6, column 0 while parsing SOAP envelope at line 2, column 0 while parsing HTTP response for method retrieveConent on object of type converter.ServiceInstance at line 1, column 0

I checked a few things but no obvious issues. This particular VM was part of a group with a similar configuration and they migrated successfully.

Since it crashed, I checked the programs control panel and found that VMware vCenter Converter Agent was installed.

The agent is installed as part of a conversion and is removed at completion. Since the effort failed, it was left in place. Usually, I can rerun without a problem but every so often the mentioned error happens.

I removed the agent (no reboot needed) and was able to initiate a conversion.

Software hive will not load

When a company is acquired; it tends to bring in problems from their past for you to clean up.

Such was the case of a Windows XP 64bit virtual machine. It would not boot. It would hang during the startup, crash and reboot.

There was no BSOD to trap which was rather odd considering the way XP handled such things so it had to be multiple problems.

The first issue was to try and boot the VM. With a physical machine, you simply booted off the installer CD. Luckily, VMware allows for this by giving you the ability to mount an ISO via the virtual cdrom drive.

One nice thing about the 64bit OS? Didn’t have to load drivers via an flp file as I previously had to do with a 32bit install.

I was able to boot off the cdrom and got into the recovery console. It did show the Windows install which I selected. Once it loaded and gave me a prompt; I ran a chkdsk /r which found a few problems and corrected them.

After that I tried a reboot and got a BSOD which quickly disappeared.

A funny side note: there is a way to slow down the reboot so you can read the message. The Net is full of articles about how to handle this. For a physical setup, this makes sense. However, I found several articles for virtual machine problems. Sometimes in debug mode; people overlook the obvious.  Why not take a screen capture?

When the VM tried to load again and the BSOD appeared; I took a screen capture and found the message stated:

STOP: c0000218 {Registry File Failure}
The registry cannot load the hive (file):
or it's log or alternate.
It is corrupt, absent, or not writable.

I researched this and found some solutions which required many steps and there was a question if this VM was really even needed as the problem had been around for awhile.

The VM needed to be up in order to answer the question. I did find a simple solution which allowed this to happen.

  1. Boot to your CD and choose R (Recovery Console), at the command prompt, type the following:
  2. ren c:\windows\system32\config\software software.old
  3. copy c:\windows\repair\software c:\windows\system32\config


This VM did need a reinstall of java and the gpupdate /force needed to be run.

Now the users are reviewing the need of the VM.

One of the most annoying events in a virtual farm is not having the root password of a host. Such was a recent case of a VMware 4.0 ESXi host. I tried all known passwords and variants and asked around but none worked and of course nobody changed the password.

It would have been easy if this was an ESX host as all you would need to do is to crash boot the system and place it into single user mode. The problem with ESXi is there is no GRUB loader so this is not possible.

If you look on the VMware site, you will find instructions to reset the root for ESX but for ESXi you have a nice message.

Reinstalling the ESXi host is the only supported way to reset a password on ESXi.  Any other method may lead to a host failure or an unsupported configuration due to the complex nature of the ESXi architecture. ESXi does not have a service console and as such traditional Linux methods of resetting a password, such as single-user mode do not apply.

It’s fine for VMware to suggest this approach but the problem is you will loose all your VMs. The author probably thinking of ESX where backups are rather simple. You can do that for ESXi but even if you have backups, it’s still time to reinstall and restore them and the user may not like the time needed.

The good thing is you don’t have to do this as there are two ways you can recover a lost password.

1) You can run a repair of the OS.

2) You can use a Linux Live CD.

I thought the easiest approach would be to run a repair since all it would do was reset the configuration of the system and leave the VMFS datastore alone.  The VMs would be forgotten but you can add them back with the client.

One of the things you will read from the documentation is that the VMFS datastore is preserved if it’s still the same patitition when you set up the host or its on a another disk.

This is not the case as it will be explained later.

There is a size issue if the VMFS location is on the boot disk and is located beyond the 900 MB partition and the partition table is corrupt. The VMFS datastore can not be recovered automatically by the repair process and you will need help from VMware.

This was not the case for this host.

Make sure you use the original install CD for the repair. If you don’t have it, get the version number from the login screen on the host and get the appropriate ISO from VMware and burn a new CD.

You will need to powercycle the host so inform your users.

 1) Insert the ESXi 4.0 Installation CD
 2) Power Cycle the machine. Depending on your how your BIOS is set for boot order 
    make the change for the CDROM via setup or boot option if your BIOS has it.
 3) The Installation will proceed.  Don't worry about it installing as a new machine 
    as the Repair option will appear. Press R to repair
 4) Accept the EULA by pressing F11.
 5) You will get a screen to select the disk with the OS.
    Now VMWARE defaulted all disks and if you read about how the VMFS partitions are 
    left alone this may not be the case.

    As you will also read:

     If you do not choose the same installation disk, the damaged ESXi 4.0 image 
     is not fixed and a new image is installed.

    Such was the situation I found.  Rather then selecting the OS disk I just 
    defaulted and went on.

    For your situation, highlight the OS disk and press Enter.

    Here you will get the confusing warning of the data on the selected disk is 
    about to be overwritten and if there were no changes to the partitions, your 
    VMFS data stores will be preserved.

    Again such is not the case if you leave all drives selected.

 6) If you have the right disk, press Enter.  Otherwise, press the Backspace and 
    select the correct disk.
 7) Now you will get your last chance to back out as you will get a confirmation
    request. If all is in order, press F11 to start the recovery.

The process will run and you will end with one of the two messages:

Repair Complete
The ESXi 4.0 image was repaired successfully and the partition table was restored. The installer recreated the partition table to recover your VMFS partitions or custom partitions. The repair operation added these entries in the partition table.

If you have this then you simply reconfigure the general configuration and then add the missing VMs

Repair Incomplete
The ESXi 4.0 image was repaired successfully, but the partition table could not be restored. The installer could not recreate the partition table to recover your VMFS partitions or custom partitions. You must manually add the partition entries to the partition table to recover your data. Call VMware support for help.

This was the message I received and it was misleading. It completed but nothing was changed as I still could not login via the root account.

Now comes the problem with ESXi, it’s not supported since it is free to use. This would be a paid support call and it would be a critical problem if it was decided to contact support.

What to do?

Before any attempt of recovery, I needed access to the system. Since the host did boot, the original OS was still usable.

To be continued.