Storage vMotion Fails Because Veeam Disables It

Our Providence address:
999 Main Street, Suite 715
Pawtucket, RI 02860
401-272-6688
Our Boston address:
800 South Street, Suite 300
Waltham, MA 02453
855-679-2971

Storage vMotion Fails Because Veeam Disables It

Added on: 02.26.13, by Andy Ward

When the current version of Veeam kicks off a job, it locks the VM’s ability to be Storage vMotioned. This is a good thing, since a backup job will likely push the datastore IOPs up and cause some unusual latency compared to normal operation  - which can trigger SDRS to move VMs to other LUNs.  When the Veeam job completes, it's supposed to remove the lock, but if the backup fails, the lock will sometimes remain in place. In fact, if the backup failed a month ago and has succeeded ever since, the lock might still be in place.

The following image is what you’ll see when your SvMotion fails. When this happens, it references the VM with a numerical ID as depicted below.


VMware states that you can remove the VM from inventory and add it back in to resolve this. While that solution does work at times, it does not appear to consistently work in all cases.

If you look at the VPX_DISABLED_METHODS table in your vCenter database, you can see all of the VMs that currently have this condition.  To find out what VMs they actually are, run the following command against the SQL database:

select * from VPX_VM WHERE FILE_NAME LIKE '%Virtual-Machine-Name%'

 


To confirm that the condition does exist for a particular VM:

select * from VPX_DISABLED_METHODS WHERE ENTITY_MO_ID_VAL = 'vm-##'


To delete the table entry and return the VM to normal operation, stop the vCenter services and run the following against the vCenter database:

delete from VPX_DISABLED_METHODS WHERE ENTITY_MO_ID_VAL = 'vm-##'


A similar issue can be experienced, but with a different root cause - in some cases Veeam still has the snapshot VMDKs mounted to the Veeam server. Make sure that Veeam doesn’t have the VMDKs mounted on Veeam when first encountering this issue. Also make sure that if you do remove from inventory or remove the lock with the steps provided here, be cogniscent of the fact that there may be a snapshot opened by Veeam - having the snapshot mounted in two places is not a good idea.

Update 09.05.13

The following is a response from Veeam to this article:

"Veeam re-enables these flags which would allow Storage vMotion to happen when everything operates normally, but this workaround helps if something goes out of sequence"

Veeam support says:

This is actually part of the VDDK code when we start the job. There’s a VDDKPrepareForAccess call that’s made that disables svmotion so the files aren’t moved while being processed. SRM or anything that does image level backups with VDDK would do the same thing.. As long as the job stops successfully it should be released with no issue. All versions of Veeam have done this. If you kill the backup server and proxies or something in the middle of a job so we can’t clean it up, that flag can be left on the VM, but just running the job again will release this lock.

View All Blog Articles