VMware VI3 HA, or How I learned to stop fearing patching ESX Servers

Our implementation of ESX in the past have typically taken the approach of “if it’s not broke, don’t fix it.” Consequently, I typically have not been very agressive in our patching to the various vulnerabilities and enhancements that VMware releases. Part of that had to be the difficulty any patch solution is for VMware with 16 or so patches all manual installed. It is certainly daunting enough to encourage slothfulness.

We however were hit with a bug (http://www.vmware.com/support/vi3/doc/esx-2066306-…) related to VM’s going unresponsive after VMotion. Our servers — now totalling 21 ESX servers — had to be fixed. Luckily, I stumbled upon this Virtrix article on how to automatically patch your ESX servers from HTTP Patch Depot, and after a minor correction, I was rapidly installing my patches on my ESX servers at the headquarters datacenter. Remote datacenters proved less resilient to WAN slowness with the script. For those, MichaelJKnight’s script did exactly the right job. We now have a patch strategy for ESX servers using VMware VI3 HA and VMotion to failover the running machines by putting it into Maintenance mode.

We did run into problems however with one system that would not run after the patch process. Booting normally led to kernel panic — “kernel panic not syncing:VFS: unable to mount root fs on unknown-bloc (0).” Booting into Debug mode or Service Console Only, the server worked fine. It checked the file system, and must have fixed the problem. It now stopped as it attempted to load initrds, and booting into debug or troubleshooting mode warns that the config was unable to be saved.

My choices were somewhat clear – call support or rebuild. I called VMware support, and this was apparently a common problem with the latest patches. Here was the fix:

  • “esxcfg-boot -p” – to reload the PCI data
  • “esxcfg-boot -r” – to refresh initrd information
  • “esxcfg-boot -b” – to setup boot information

I have definitely put these in the toolbox for next time.

Technorati : , , ,


No comments yet to VMware VI3 HA, or How I learned to stop fearing patching ESX Servers