PowerCli, vmdk’s “Device or resource busy,” and host MAC – oh my…

We have been having a number of problems with our backup software and its vStorage backups locking disks and thereby creating multiple phantom snapshots.   We are having this problem so often I thought I better post our quick fix to it, which involved some PowerCli goodness.

  • Determine the locked files. Go to the directory, and touch all the files in the directory to determine which are locked

touch *

  • Discover which host locked the vmdk disk.

Run:

vmkfstools -D servername.vmdk

Indicates not locked or locked by service console process (may post later on how to solve this one)

[root@phovmwareXX vm]# vmkfstools -D vm-flat.vmdk
Lock [type 10c00001 offset 80693248 v 66, hb offset 3219456
gen 11609, mode 2, owner 00000000-00000000-0000-000000000000 mtime 1100232]
Addr <4, 135, 1>, gen 3, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 2144047674368, nb 255581 tbz 0, cow 0, zla 3, bs 8388608
Indicates host with MAC 3c:4a:92:78:0c:2a
[root@phovmwareXX vm]# vmkfstools -D vm-ctk.vmdk
Lock [type 10c00001 offset 67147776 v 35, hb offset 3219456
gen 11611, mode 1, owner 4d921417-f871d30a-27df-3c4a92780c2a mtime 743957]
Addr <4, 102, 3>, gen 4, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 8179408, nb 1 tbz 0, cow 0, zla 1, bs 8388608
  • Discover which host has the specific MAC address.  I created a simple little PowerCli script to that for me.


#Enter which cluster the locked file is within
$cluster = Read-Host "Enter Cluster"
#Enter MAC Address of host (00:00:00:00:00:00 format)
$MAC = Read-Host "Enter Desired MAC (00:00:00:00:00:00 format)"
get-cluster cluster4 | get-vmhost | Get-VMHostNetworkAdapter | where {$_.Mac -match "$MAC" } | select-object VMHost, Name, MAC, IP, BitRatePerSec | ft -autosize

This should tell you immediately which host has locked it open.  You can PuTTy to the host and attempt to kill any process with it locked (lsof or ps -auwwwx | grep filename ).  Alternatively, you can put the host into maintenance mode and reboot.

Share

Comments are closed.

Statistics