Backuping up a running VM using snapshots

[Kartweel]

Hi,

I just wanted to check, is it safe to back up a running VM using the following sequence?

Take a VM snapshot
Take a filesystem snapshot (eg. VSS, or ZFS snapshot).
Copy the VBox files from the filesystem snapshot
Delete (merge) the VM snapshot back

So I know there are pauses while the VM is snapshotted, but it isn't too bad.

My concern is that the filesystem snapshot, depending on writes to the VDI, may mean that the VM snapshot file & the original VDI are out of sync?. Is this possible, or is it safe to back up this way?
I could pause the VM while the snapshot happens, but that creates a pause that is long enough to make network clients disconnect.

Thanks!

Ryan

[mpack]

No, that is not safe. Your backup would be of a crash consistent state. There is no telling what state all of the control xml files will be in. The only safe way to back up a VM is to shut it down (not suspend) and then copy the VM folder.

[Kartweel]

What about using 2 snapshots like on this blog? http://root42.blogspot.com.au/2010/08/b ... hines.html
Or is this bogus?

[Kartweel]

Ok, so thinking about it... I think it is OK, because I am actually not using the current state of the system, so I don't really need the differencing VDI which is the only file that will be changing. I'm only needing the .sav file and the original VDI. I think that is right... I might test my theory by not even backing up the differencing VDI and see if I can restore... (I know it will look for it, but I should be able to delete it from the vbox xml file)
So it would just be if the xml file is changing, and the chances of that changing while backing up is pretty much 0, and it stores the previous version anyway.

[mpack]

If you are really keen to proceed with that then I advise also taking real backups on a regular basis so you'll have something to fall back on when your scheme results in corrupted control files and your VM won't start.

[emoll]

I think should be a crazy way, but i have tested the following backup way.


1.- during the night, 00:00h, make a VBoxMachine folder copy with cobian backup, i tested it with windows copy too, without any shadow setting, directly.
2.- I have no error on the copy process,
3.- i test the copied vboxmachine, moving to the original place,
4.- i have the error that the machine last time is no correctly shutdown.
5.- but no more system error, data integrity seems ok, and work. Seems like if accidentally powered off.

Anyone can tell me if this is a real solution?

[Kartweel]

I've done it like that in the past also and it has worked ok, but not the best. I think the problem is that without a snapshot it isn't crash consistent. So you might have file system corruption in the guest VM. It could go unnoticed for a long time and then suddenly the entire OS is dead. Try running a disk scan on one of your backups after you restore it to see if there are any file system issues. So I don't think it is a safe way to back up, you'll get caught out sooner or later.

[emoll]

Thanks a lot,

and if before copying i do a vboxmanage pause machine?

should be a secure way to backup virtual machines without shadowcopys?

[Kartweel]

The link I posted a few posts earlier uses VM snapshots and just copies files, I don't think it uses shadow copies. Pausing would probably make it better, but I'm not sure sorry. If the VM is running and you aren't backing up the current state, it will still be in a crash consistent state when you restore it (Which is better than copying when it is running, but could still cause problems. Most file systems can recover from crash consistent state, but the applications and os might have trouble recovering, it is just like pulling the power plug on a running computer). So you are probably better off stopping it and saving the state, copying, then starting again, or using the method in that link.

[scottgus1]

One of the best reasons I've found to run essential systems in a VM is the ability to transfer the VM to different hardware with a compatible processor and the VM fires up just as if it's still on the old hardware. So if my main Intel Xeon server box that runs my SBS2003 VM fails, for example, I can put the backup VDI's on the Core-I7 PC I use and fire them right back up again, total server restoration in less than 5 minutes. (Try that with NT Backup )
My backups involve shutting down the whole VM to a nicely shut-down-by-the-OS powered-off state, then copying the VDI's (these VM's are in Version 3 folder format, so I can't copy the entire folder like I could with version 4, but I have backups of the VM settings too, and I have figured out how to restore to a different PC using just the VDI's and the backed-up VM settings, and have done so successfully.) These backups can go on different hardware, and as long as the processor is a same-capability or newer Intel, they'll start right up just fine. (Switching to an AMD-based physical machine can be done, but requires reconfiguring the VM and isn't a less-thsn-5-minute process...)

I have asked this "snapshot backup" question before (viewtopic.php?f=1&t=47588&p=214961&hili ... ot#p214961), and I concur with Mpack's answer. It is not a good idea to use snapshoots for backup.

Using a snapshot to facilitate an online backup stores the state of the virtual processor(s) with the backup (if I read correctly). Unless you are putting the restored VM back on precisely the same processor type as the one that the VM was running on when the backup was made, the stored processor stuff might not match the new processor. I don't know what would result if such a thing happened, but I surmise it wouldn't be pretty. It might have no problems. But I wouldn't trust it for essential VMs that I'd grieve for if I lost them. A full backup with the VM powered down is much more restorable.

[Kartweel]

That's a very good point regarding hardware compatibility!. Even if a network config is slightly different or something you wouldn't be able to restore it. Regarding your CPU, you can expose a synthetic cpu to the guest os, so it is portable across amd / intel, etc. But I haven't tried it ... The situation is similar to teleporting VMs apparently? (Which is figure it is basically a snapshot, transfer and restore).

So I might keep snapshot backuping during the week, then do a shutdown backup on the weekend. Depending on the machine... I think ppl will get annoyed if I keep shutting down some of the VMs all the time .

In that link, I don't see how a "live" backup from within the VM is any good, unless it is application specific. eg. SQL server backup, not an entire OS image backup. Otherwise it will still be in a crash consistent state, which is no better than just snapshotting the VDI. Also you lose your vbox config. But he didn't really explain "live" backup very well.

Thanks for your input!

[mpack]

In that link, I don't see how a "live" backup from within the VM is any good

It would rely on guests implementing something like Windows VSS service.

Versions of Windows since XP have supported a Volume Snapshot Service (not to be confused with VirtualBox snapshots). Basically live backup starts by sending a signal to all running apps to flush their file buffers, then the OS creates a virtual image of the disk, then it unlocks the drive again allowing apps to resume running while backup software backs up that entire virtual image or only selected files from same.

It isn't a complete solution: not all apps understand the VSS "flush" signal, and not all of those actually do a full flush. However, generally it isn't a bad solution. Technically it's a crash consistent state, but it's less "crashy" than the proposal above. It's even more reliable if you shut down all apps (inside the guest) first.

[Kartweel]

I understand what you mean now. Yep disk image with flushed buffers for vss aware applications is definitely better than a snapshot without flushing buffers, which is really quite dangerous.

[emoll]

so much interesting posts,

then you say that is good idea to make a diskshadow for the volume that contain the VM's and then copy all the VM's folder with the machine online?

but still have the risk like if I power off accidentally, no?

Thanks for your help

[Kartweel]

He means back up the VM internally. So shadow copy and back up the image of the VM internally (maybe to a shared folder on the host). If you shadow copy the host, it won't go through the applications inside the VM to flush their buffers. So it would be better to do it inside the VM. Shadow copy is slightly better than accidental power off because the VSS aware applications are in a state where they should recover correctly. Shadow copy on the host and copy the VMs would not be as good, still crash consistent, but the applications running in the guest may not be in a state where they can recover.

So I think order of goodness of backups
- Shutdown guest and copy files (Full recoverability)
- Running VM - Shadow copy from within the VM (Crash consistent, VSS aware application should be "safe")
- Running VM - Save VM state and copy files (Needs to restart on similar hardware to recover saved state, otherwise crash consistent)
- Running VM - Host Snapshot and copy files (Crash consistent, just like pulling the power plug)
- Running VM - Copy VM files with no snapshot (Very dangerous, will most likely result in a corrupted disk image)

Does this sound right to everyone?