Cleaning up snapshots and cloning

[gdseas]

Hello everyone,

I am running an Ubuntu mail server as a guest on an Ubuntu host using VBox 4.1.12. I have two questions that I am having trouble with.

(1) I use daily snapshots to make sure that if my current snapshot gets corrupted, I can quickly roll back to yesterday's snapshot. I wrote a small script to both generate and then clean-up snapshots older than a week. However, after my script runs: VBoxManage snapshot <SYSTEMNAME> delete <OLDSNAPSHOT>, the snapshot goes away but the pesky .sav files associated with the deleted snapshot remain. Is it safe to delete these .sav files?

(2) I would really like to clone the mail server every night but I can't afford the many hours of downtime that are required while the guest is being cloned. My guest is 1TB in size so it takes *hours*. As far as I can tell from browsing the forum and the internet, there is no way to "fast" clone a VM -- unless I have missed something.

I even thought of creating a mirror VM, sync the contents between the mail server and the mirror (using rsync), shut down the mail server, swap IPs and then clone the mail server while the mirror is handling traffic...and repeat the next day. It just seems such a clumsy way...

Do you brilliant people have any idea or scheme that would work for my case? Any help would be very much appreciated!

Best-
Greg

[mpack]

First of all, snapshots are not backups. On the contrary they create a greater risk of total loss than taking no backups at all.

Basically, for real backups your choices are to shut down the VM and take a backup: incremental backups really shouldn't take that long. Or, you can run a backup tool inside the VM.

Also, I don't think you understand what deleting a snapshot entails. It basically removes a marker from the timeline, which removes your ability to revert to that moment on the timeline. It does not affect the current state or invalidate the .sav file. The latter is a hibernation file and nothing much to do with the disks anyway.

[gdseas]

First of all, snapshots are not backups. On the contrary they create a greater risk of total loss than taking no backups at all.

Understood. And I have seen such statements before even though nobody bothers to explain why this is the case (and I still don;t get it even after RTM)... This is why I do regular guest-OS based backups...*But* doing snapshots saved me twice when we lost power in the building beyond UPS capacity, servers went down abruptly corrupting the current snapshot. I was up and running with the snapshot from the night before in 30 seconds!

Basically, for real backups your choices are to shut down the VM and take a backup: incremental backups really shouldn't take that long. Or, you can run a backup tool inside the VM.

I am afraid I don't understand what you mean. Take an incremental back up of the OS *within* the VM or *of* the VM? I do full/incremental backups on a weekly/daily schedule of the guest OS but if my server dies I would rather launch a saved cloned VBox instance rather than having to go through the motion of setting up a new server and recovering user data and mailboxes from full backup + incrementals. If you are proposing of taking an incremental back up of the VM then I still need to bring down the VM for an occasional full backup which takes hours -- a luxury I don't have.

Also, I don't think you understand what deleting a snapshot entails. It basically removes a marker from the timeline, which removes your ability to revert to that moment on the timeline. It does not affect the current state or invalidate the .sav file. The latter is a hibernation file and nothing much to do with the disks anyway.

Thanks but I am not sure my question is answered though. So, I can delete the .sav files or not ? What is the impact?

Thank you for your help!

[mpack]

A snapshot chain is like a patch list. The B file tells you how to patch A to create the B state. In turn the C file tells how to patch the B state to create the C state... and so on. Damage any one of these - or the control file - and you lose that and everything downstream of it. Lose any one and the VM will refuse to boot and require expert repairs to recover even a part of it. So where before taking your "backup" you had a single point of failure, you now have multiple points of failure and still no redundancy, which is worse than no backup at all, especially considering that a snapshot chain is much harder to repair than a stand alone VDI. All real backups exploit redundancy in which data damage doesn't matter because the data is duplicated elsewhere. If you don't have that then it isn't a backup.

Incremental backup meant, shut down the VM and run a tool which only copies data which has changed since the last backup. Mind you that might not be practical. Me, I'd just see how long a full backup would take, and decide if that's preferable to losing the lot after the first power cut.

On the .sav file, I did answer you: deleting a snapshot has no bearing on the current state. So if the current state has been hibernated then the .sav file is still needed. However the .sav file shouldn't exist if the VM was fully shut down instead of being hibernated.


I note that you ignored the suggestion of running a live backup tool inside the VM. Perhaps you missed it. This is after all what I imagine is usually done on a physical PC running that kind of system.

[gdseas]

A snapshot chain is like a patch list. The B file tells you how to patch A to create the B state. In turn the C file tells how to patch the B state to create the C state... and so on. Damage any one of these - or the control file - and you lose that and everything downstream of it. Lose any one and the VM will refuse to boot and require expert repairs to recover even a part of it. So where before taking your "backup" you had a single point of failure, you now have multiple points of failure and still no redundancy, which is worse than no backup at all, especially considering that a snapshot chain is much harder to repair than a stand alone VDI. All real backups exploit redundancy in which data damage doesn't matter because the data is duplicated elsewhere. If you don't have that then it isn't a backup.

Best explanation on snapshots -- now it makes sense ! thank you!

Incremental backup meant, shut down the VM and run a tool which only copies data which has changed since the last backup. Mind you that might not be practical. Me, I'd just see how long a full backup would take, and decide if that's preferable to losing the lot after the first power cut.

On a Linux machine the only thing I can imagine like this is rsync. But which files do I do this too? Is is the .vmdk files ????? What files do I need to back-up?

On the .sav file, I did answer you: deleting a snapshot has no bearing on the current state. So if the current state has been hibernated then the .sav file is still needed. However the .sav file shouldn't exist if the VM was fully shut down instead of being hibernated.

Sorry didn't catch the hibernated piece.

I note that you ignored the suggestion of running a live backup tool inside the VM. Perhaps you missed it. This is after all what I imagine is usually done on a physical PC running that kind of system.

I don't think I ignored anything unless I misunderstood. As I mentioned: "I do full/incremental backups on a weekly/daily schedule of the guest OS" (using tar or filesystem dump)....Is there some other kind of backup I should be doing that I am missing?

Thank you for your help!

[Perryg]

If this is a mission critical mail server then I can hope that this is all the machine is being used for.
If that is the case all you need to backup on a daily basis is the data storage (which can be done with rsync and cron).

Just make sure that you have a good clone of the now state (yes I know it will be down for this) and clone after a system, or mail server update. After all the system and program don't really need to be backed up all the time. I usually only do updates once a month during the maintenance cycle and clone the machine then. Tell users of the monthly maintenance window and you might even be able to get by with once every two months or even quarterly.

Now after the cron job you can copy the rsync backup to a different media, whether it is the host or some other spin disk media while the machine is hot.

Lastly, please listen to Don. He and I have both been hit with snapshots and have listened to all the crying from others here about loosing it all. They are great for an immediate shot when updating to make sure you don't loose the guest due to a bad install, but then merge the snapshot and only have one flat file to deal with.

[gdseas]

Perfect! thank you.

Yes it is a mission critical and dedicated to the specific task server.
I am planning now on the downtime to get the clone. I will get rid of the snapshots and continue the guest backups via rsync and tar (using cron).


Warm regards!

[Perryg]

One other thing. Backup the guests *.vbox file and the VirtualBox.xml file when you do your system clone. These are the control files for VirtualBox and the guest. You usually loose the *.vbox file on a hard down, and why they started the *.vbox-prev as a safety.

[gdseas]

Well, I return, but unfortunately, not victorious...

I scheduled the outage for this morning and started cloning bright and early. After removing the snapshots and powering down the VM, I issued:

Code: Select all

VBoxManage clonevm MailServer --options keepallmacs --name MailServer-10JAN13 --basefolder "/mnt/data0/VMClones"

and the ...wait started.

After 4 hours the process was still at 0%. I could see my disk getting filled-up (CPU was low) but nothing else happening. So I thought I take a couple of measurements and attempt to project how long this will take. Well, it wrote on average about 145.7 bytes/sec. After doing the math, it would take 2,045 hrs (or 85 days or 2.8 months) to clone my 1TB VM

When users started gathering outside my office with pitchforks and torches, I decided to terminate the process and restart the VM and services. So cloning won't work for us...<sigh>

Is there a way to just power-off the VM and copy files to a backup disk? My VMs use VMDK for storage. What files do I need to copy?
Is there any information on how I can use the copied files (import? attach disk to new VM) to recover if the original VM dies?

Thanks for all your help (again)!

[Perryg]

You can do a direct CP/Paste of the guests folder. This accomplishes basically the same thing and should be faster.

[gdseas]

You can do a direct CP/Paste of the guests folder. This accomplishes basically the same thing and should be faster.

Will try this then. So this is the directory with the .vmdk files plus the .vbox file as you indicated before, right? (BTW, I do not seem to have a VirtualBox.xml file anywhere... is this normal?)

Would restarting the machine in the same host would be the same as copying the files back and powering-on?

Thanks!

[Perryg]

Will try this then. So this is the directory with the .vmdk files plus the .vbox file as you indicated before, right? (BTW, I do not seem to have a VirtualBox.xml file anywhere... is this normal?)

Yes the guests folder has the *.vbox, *.vdi or vmdk, subfolders with logs, & snapshots. The VirtualBox.xml file is in a hidden folder on Lunux. ./VirtualBox in your user space.

Would restarting the machine in the same host would be the same as copying the files back and powering-on?

I don't understand this.

[gdseas]

Would restarting the machine in the same host would be the same as copying the files back and powering-on?

I don't understand this.

Sorry, let me clarify.
After I copy the files and my production VM dies (gets corrupted, rendered unusable), can I just copy the files from backup to the right place (overwrite the broken VM files) and just power-on the VM? Or there is a different way to bring the backups in VBox to use them?

Thanks!

[mpack]

Yes, you can simply overwrite the corrupted VM files with good ones, only proviso is of course that VBox must be completely shut down at the time, i.e. no GUI running and no VMs running.

[AllenL]

Lastly, please listen to Don. He and I have both been hit with snapshots and have listened to all the crying from others here about loosing it all. They are great for an immediate shot when updating to make sure you don't loose the guest due to a bad install, but then merge the snapshot and only have one flat file to deal with.

Hi,

I believe this is an old thread but have one question which was not indicated in this post.

It was mentioned above that the .sav files are created during snapshots and must not be deleted. If I have merged all snapshots into one flat file (no more snapshots), can I now delete the .sav and the rest of the other snapshot files in the folder?

AllenL