XenServer Upgrade 6.0.2 to 6.2 : the vm is incompatible with the cpu features of this host

The Solution

In my case I ended up needing to force the VM migration via the command line on each server. From xencenter, connect to the host that you're getting the error on and bring up the console. Enter the following command.

xe vm-migrate host=<xenserverhost> force=true vm=<vmname>

#<xenserverhost> is the host you're migrating to - should be the one of the ones that has already been updated. You're not allowed to migrate to an un-upgraded host, so forcing it to do so could break everything. Be careful.

#<vmname> is the name that shows up in xencenter, there's also an option to do it by UUID

This will force the machine to migrate to the new host. Keep in mind I am running this in a homogeneous environment, so I had some assurance that compatibility wasn't going to be an issue. If you're running on non-homogeneous hardware you'll probably want to test with a non-critical (or at least less critical VM) before you start migrating everything.

The Cause/Full Story

Upgrading servers is always a time and a half. One of the reasons I like Citrix XenServer is the ability to do rolling upgrades on server pools without shutting down any VMs. It works great in theory, but theories are flimsy things because they exist on paper - a substance not known for ability to withstand stress (slightly tortured analogy).

I ran into a problem today upgrading some of my XenDesktop VM hosting XenServers from version 6.0.2 (+ various updates) to version 6.2. This version (and indeed all versions back to 5.6) are supposed to be compatible with the rolling upgrade feature in xencenter. 

I made all the backups, and followed all of the "before upgrading" steps listed in the guide, then started the install using the rolling pool upgrade wizard in xencenter. 

The first server (the master) went great. All the VMs migrated off the server, everything installed and it came back up just like it should. However when I went to start the second server update, none of the VMs could migrate off with the error "the vm is incompatible with the cpu features of this host"

Which was bull, since most of the VMs had just been running on that server, and were currently running on a server that was identical hardware. So after a bit of googling I found the follow commands.

xe host-cpu-info

xe host-reset-cpu-features

xe host-set-cpu-features

Running the first command on the upgraded server and an non-upgraded server I could see that the CPU feature set had changed (for some reason). I tried the second command, to see if resetting would work, but even after a server reboot the feature set was the same. I tried manually setting the feature set using the third command. That seemed to work, the "feature set after reboot" changed to the correct feature set. However after a reboot the feature set had not actually changed. 

Running low on options I went to the command line to try to migrate (GUIs are for babies, etc. etc.). Trying to migrate the servers manually using the "xe vm-migrate" command yielded the same error. But running xe help vm-migrate showed me there was a "force=" parameter I could set.

Luckily I had a few currently-not-in-use VMs I could use to test. The test machines were able to migrate using the force option. I was able to force migrate all of the VMs without any crashes or other problems. After this I was able to finish the upgrade on the other pool members. For the record these were all windows 7 machines (and one server 2008 R2) with the 6.0.2 XenTools installed.