Patching an inherently dangerous activity

We’ve been busy patching for the recent vulnerability. This has led to touching a lot of servers that had been left to rust in recent past. We have 60-some servers that meet the criteria (Windows 2000 SP4) in the most vulnerable group, ignoring of course all the Windows SP2 & 3 servers unable to be patched currently. We had an interesting occurance with our Ciscoworks server though. After the patch was applied and the system rebooted, the system blue screens with inaccessible boot device. We boot the machine with our Bart’s PE Builder CD and discover the entire systemroot aka Windows directory had been deleted. The directory was there, but all files and subdirectories were deleted. It made for an interesting recovery or explanation for it was even possible.

Planning for a SAN migration from an EMC Symmetrix to a larger Clariion led to patching drivers, firmware, and SAN application on all our SAN attached servers. Again, patching servers led to all sorts of storage-related fun. Machines suddenly not seeing their SAN space or cluster problems. Some of the problems were self-induced as using dynamic drivers to span multiple (10!) luns is without a doubt a Bad Idea ™.

Just like Luminol in CSI, patching systems just exposes all the flaws in your systems you wish would have stayed hidden.

Share

No comments yet to Patching an inherently dangerous activity

  • Just a continuation of this thought, we had further patching activities today, and we successfully implemented the BSOD generator (aka Windows 2000 SP4 Rollup patch) twice. We had the “Address 804A6467 base at 80400000, DateStamp 427b58bb – ntoskrnl.exe” infamous error on each machine, but are actually getting pretty good at having Bart’s PE Builder standing by to swap the scsiport.sys file with the previous copy.

Statistics