Latest Entries »

I had a case where I was building an image for windows 2008. The host uses 10 GB NICs and for some reason, they were not recognized.

I would add them to the configuration, reboot and nothing happened. I installed Xentools and nothing happened. Thinking it might be the age old driver issue but I did not find any on the vendor site. Setup issue?

To get around this “feature;” you have to delete all NICs and then add the one you want.

A quick reboot and the network was available.


One thing I have noticed on the Net; information about Citrix can be lacking at times.

I had an instance where a resource pool was established and the thinking behind it was wrong. I wanted to destroy it. I started Xencenter and accessed the doomed pool. I removed all the extra Xenserver hosts by simply right-clicking the entry and selecting “Remove Server from pool”

When it came time to kill the master, there wasn’t an option. I tried the xe options of eject and forgetting but they gave errors.

As mentioned, the Net was lacking on information. So I basically started hunting around thinking “it can’t be a reload!” I didn’t want to go through patching again.

I looked at the Xencenter window tab for Pool and sure enough there was an interesting entry!

Make into standalone server

I selected the master host, clicked “Pool” and selected the option. It ran really fast and the pool entry disappeared with the server showing up at the bottom of list of resources. I expected it to reboot but it didn’t. I gave it a reboot for good measure and then added the ex-master as a host to the correct pool.


My adventures with Xenserver found a couple interesting problems today. The first I will leave for another post. This one involved trying to install an update: XS65E1003.

I was trying to install it via the updates wizard and when it came time to run the Prechecks; it would fail with the message:

Checking server side status ... failed with an unknown error.

Ever so helpful.

I ran the update a couple more times just to see if there was a bad download but the error continued.

The messages log didn’t show anything obvious then again I could have missed it.

I checked the Net to see if there was anything similar especially for the unknown error. I found similar errors but not exact.

I was reading through a problem with xenserver 6.2 and saw an interesting suggestion. From the shell enter:

xe vm-cd-eject --multiple --force

I figured why not? It couldn’t hurt as this pool was not in service yet.  It gave a few errors. Nothing is the CD/DVD drive and couldn’t talk to a host. I did notice there wasn’t a message for each system.

I re-ran the install and sure enough; it passed the prechecks.

Strange but expected considering Xenservers error messaging at times.

Normally, I would give a link to where I found the command to credit the person but for the life of me; I can’t find the page anymore. If I find it later, I will add a link.

I am migrating virtual machines from VMware workstation to our cluster. People were trying to save money and use the workstation product instead of getting another cluster.

I am using VMware vCenter Converter Standalone and one migration died. After correcting a couple things; I tried to rerun but received the following error:

A general system error occurred: Unexpected element tag “ConvertManager” seen while parsing serialized DataObject of type converter.AgentInstanceContent at line7, column 63 while parsing return value of type converter.ServiceInstanceContent, version converter.version.version2 at line 7, columen 0 while parsing SOAP body at line 6, column 0 while parsing SOAP envelope at line 2, column 0 while parsing HTTP response for method retrieveConent on object of type converter.ServiceInstance at line 1, column 0

I checked a few things but no obvious issues. This particular VM was part of a group with a similar configuration and they migrated successfully.

Since it crashed, I checked the programs control panel and found that VMware vCenter Converter Agent was installed.

The agent is installed as part of a conversion and is removed at completion. Since the effort failed, it was left in place. Usually, I can rerun without a problem but every so often the mentioned error happens.

I removed the agent (no reboot needed) and was able to initiate a conversion.

With our new XenApp effort; we are going to attempt VDI for our Linux users. Previously, we were using Sun Terminal servers but Citrix stopped supporting it awhile ago and now we have to change.

Our XenServer is version 6.5. Before you attempt to install xentools on any of your guests; make sure you have installed Service Pack 1 and all the suggested patching. Otherwise, you might find yourself having to update xentools again.

Installing Xentools is easier then I remember from a failed XenServer project a couple years ago. Well, the XenServer effort failed due to many mistakes by Citrix  and I probably flushed my brain of this experience as fast as possible.

To install Xentools on RedHat.

  1. Start XenCenter.
  2. Login into your Redhat guest as root.
  3. From XenCenter; click the link to install xentools or right-click your guest and select install xentools. This will load the ISO on the DVD.
  4. From your Redhat guest; enter:   mount /dev/dvd /mnt
  5. Enter: cd /mnt/Linux
  6. Enter:  ./
  7. Reboot after it finishes.

Surprisingly simple to do.

A couple years ago we attempted to use XenServer to offset the costs of VMware. We received the usual sales presentations but what was interesting was the main pitch seemed to be we were foolish to use VMware. Not exactly a great tactic but we were mandated to give it a go.  Xenserver had is quirks and it’s possibilities but ultimately it failed.

Citrix had promised 10 gig Ethernet support. They kind of stretched the definition as yes the NIC would work but the speeds were terrible. At best we saw 3 gig. This was due to their drivers being 32 bit and they wouldn’t give any idea of when this would be addressed.

Support was abysmal. Basically; they received their initial order which as only a small amount of licenses and basically disappeared. There was no after sales support to make sure things worked. It’s almost like they thought we weren’t going to blast VMware out of the company right away so why bother?

Training was terrible. A coworker and I went to a week course on Xen Administration. The Citrix training materials were awful! There was no editorial review at all. Paragraphs out of order. Labs would have 10 steps and even though they were numbered right the steps were really 1,2,5,6,3,8,4,7,9,10. My coworker stopped going but I tried to tough it out. The teacher tried his best to correct everything but much time was wasted. Citrix never responded to my complaints over the student material.

There were other issues but ultimately I was the only one left with a small cluster. It had to be abandoned when the company refused to re-up the licenses.

Fast forward a couple years.

We are going to redesign our XenApp farms. It needs an uplift and we are going to redesign our access. One issue is the old Sun terminal servers. Citrix stopped supporting this product awhile ago but we kept using it as the Linux alternative at the time was terrible. Now, we have Linux with VDI which means the use of XenServer.

The demo farm was pretty easy to setup as it is small and only meant to reintroduce ourselves and see what the new products will yield in our environment.

Xenserver hasn’t changed much.  It’s version 6.5 and so far it looks like it added missing features from 5.x and strengthened other areas. The 10 gig drivers should yield proper speeds as they are now 64 bit.

The Linux side for guests is still lacking. Citrix was mainly Windows for many years but seriously; shouldn’t you be able to setup a guest and install the OS via pxeboot? We cheated and allocated a 2012 shell with it looking to the network. When the guest booted, it looked to the network and our Kickstart server installed the OS. Mind you this is not a clean way to handle this. The virtual stuff in the background will treat the guest as a windows box. For our demo purposes; the work load will not be that heavy.  We won’t use this approach when we start setting up the production farm.

Thinking back to the class; the sad thing is I have learned more with 3 hours of the sales engineers time then I did with the class.

Overall, we are still in the setup phase but it’s looking good so far.

Software hive will not load

When a company is acquired; it tends to bring in problems from their past for you to clean up.

Such was the case of a Windows XP 64bit virtual machine. It would not boot. It would hang during the startup, crash and reboot.

There was no BSOD to trap which was rather odd considering the way XP handled such things so it had to be multiple problems.

The first issue was to try and boot the VM. With a physical machine, you simply booted off the installer CD. Luckily, VMware allows for this by giving you the ability to mount an ISO via the virtual cdrom drive.

One nice thing about the 64bit OS? Didn’t have to load drivers via an flp file as I previously had to do with a 32bit install.

I was able to boot off the cdrom and got into the recovery console. It did show the Windows install which I selected. Once it loaded and gave me a prompt; I ran a chkdsk /r which found a few problems and corrected them.

After that I tried a reboot and got a BSOD which quickly disappeared.

A funny side note: there is a way to slow down the reboot so you can read the message. The Net is full of articles about how to handle this. For a physical setup, this makes sense. However, I found several articles for virtual machine problems. Sometimes in debug mode; people overlook the obvious.  Why not take a screen capture?

When the VM tried to load again and the BSOD appeared; I took a screen capture and found the message stated:

STOP: c0000218 {Registry File Failure}
The registry cannot load the hive (file):
or it's log or alternate.
It is corrupt, absent, or not writable.

I researched this and found some solutions which required many steps and there was a question if this VM was really even needed as the problem had been around for awhile.

The VM needed to be up in order to answer the question. I did find a simple solution which allowed this to happen.

  1. Boot to your CD and choose R (Recovery Console), at the command prompt, type the following:
  2. ren c:\windows\system32\config\software software.old
  3. copy c:\windows\repair\software c:\windows\system32\config


This VM did need a reinstall of java and the gpupdate /force needed to be run.

Now the users are reviewing the need of the VM.

There was a “secret” movement of server equipment in our hardened site. I call it secret since notification of this plan was minimal. One of servers hit was a VMware ESXI host which happened to have two important VMs.

After the host was restored to the network with a new IP; I was surprised with this message when I tried to connect to it:

The VMRC console has disconnected.. attempting to reconnect

I tried a few things but nothing worked. This ESXI host was running version 5.0 and I remembered having an error message when trying to the VMware client wanted to update.

Due to the critical nature of the VMs; I did not have enough time to debug the issue. I ended up using another server and installed the VMware client from this host (ie http:/esxi host).

After that; I was able to connect to the host, power up the VMs and apply IP changes.

I have VMware workstation but I rarely use it as I have access to a large ESX cluster. However, we do acquire companies and sometimes companies do things like run production from the workstation product as it’s cheaper.

We are iterating the company and of course we are phasing out the workstation installs. You would think VMware would make it really simple to migrate to the ESX product as it’s more expensive. But, such is not the case.

We used the vCenter converter and ran a powered on migration.  Slow but does the job.  We had the originals left over and wanted them deleted so a user didn’t notice them and think “oh wait, these VMs are supposed to be running”

If you use workstation, you would probably remember how to delete the VMs. However, at mentioned I rarely use it and the interface isn’t as intuitive as the VMware Client.  There is no simple right-click delete.

To delete a VM, you:

1) Start VMware Workstation
   • If remote, connect to the VMware Workstation installation.
2) Click the virtual machine.
3) Click the VM menu, select Manage and Click Delete from Disk.
4) Click Yes.

To free up the space in VMware Workstation, empty the Recycle Bin. Note: You will not have this option from remote access.

Cloning and SIDs

A big issue of Windows cloning is the duplication of the machine Security Identifier (aka SID). The SID represents all security principals on a computer (machines, domain computer accounts, users and groups). In the past simply duplicating a computer saw the SID copied to the other computer or in this case the virtual machine.

If you run sysprep, you don’t have to worry about it. However, the computing world being as it is there are always attempts to speed the process and avoid steps if possible.

One of these attempts was a nice little utility called NEWSID by Mark Russinovich. It was small, simple and it appeared to do the job. I admit to using it a couple times.

However, as with many things you learn later that it’s not the best thing to use. In this case, newsid starts having problems if you install applications and add them to your cloning process.  Mark Russinovich has a blog and a good entry about it so I won’t go into detail.

NEWSID was retired in 2009 and really shouldn’t be used anymore. Options are sysprep (expected by Microsoft) and third party (VMware SID generation).

I have used the cloning wizard of VMware and found it does generate a new SID. This was verified by using Mark’s program psgetsid. This is part of pstools and if you aren’t already using them, you should get them!

However, I am curious to how clean the VMware SID generation is and will go back and review some clones to make sure things are good.