on Mar 14th, 2008A First
This has been a long, crazy week. Wednesday night we realized there was some additional fallout from the disk problem we had Sunday night with our Exchange server. The VMware ESX rebuild that formatted the raw device mappings also deleted the partition tables off of three vmfs volumes on one of our SANs. So our most import ESX cluster did not recognize the LUNs as vmfs partitions. I’ve been “playing IT” for almost 11 years now. This isn’t my first rodeo when it comes to an “oh $%&#” moment. But this was the first time that I was actually scared of what might happen. In researching the issue I came across more than one person saying “do not power off the host or you’ll lose the guests.” Having roughly 25 business critical or important servers sitting on these volumes with the threat of losing them was enough to make my knees wobble. Luckily crisis was averted. VMware support was able to recreate the partition tables and save the data. They did confirm that powering off the hosts or guests could have resulted in bad things. We got lucky. Lesson is to disconnect the SAN before rebuilding an ESX server. There is also some documentation on the web about having the HBA drivers not load during the install that would protect the SAN volumes as well. Sitting here tonight having just watched Kansas beat my Huskers in the Big 12 tournament this Miller Lite tastes extra good after a week like this!
A tribute to yet another idiot in the news, former New York Governor Eliot Spitzer. Borrowed from Wedding Crashers! (Rated R for content and language)