Backup Plan

[dksoba]

Situation: I started using VirtualBox on a Debian machine and before I knew it I was running several non-test virtual machines (an OwnCloud instance, a DokuWiki install, etc.) that I've come to depend on (there are 2-3 users using these virtual machines lightly). I currently back up these by shutting them down, copying their associated files, and starting them back up. I'm doing this all manually. I'd like to do this automatically. I can afford to have the VMs go down for awhile during backups (an hour a night would be do-able). But I can't really afford to manually do backups (they just won't get done as often or regular as they should). I can afford to test my backups periodically as well (I guess this should never be fully automated).


I've done a bit of research and what I've learned so far is the following:

1. It's ill advised to backup virtual machine files while the virtual machine is running (I actually tried this and found out that my virtual machine that I copied was out of date).
2. Snap shots shouldn't be used for backups.
3. From my experiments, ACPI shutdown will not wait for commands to finish running. To demonstrate this, I used this command:

Code: Select all

rm ~/test_touch/* && touch ~/test_touch/test_touch_start_$(date +"%s").txt && sleep 30 && touch ~/test_touch/test_touch_end_$(date +"%s").txt

and issued an ACPI shutdown during that command. The result was only 1 file being written to disk (indicating that the shutdown did happen before the command was done running).

My automated backup plan (which is likely to change based on feedback) is as follows:
Execute this for each VM:
1) Use ACPI shutdown to shut down the VM. I'm not sure if this is safe to do with a LAMP stack or OwnCloud stack running but it's my current idea.
2) Verify the VM is shut down.
3) Make a copy of the necessary VM files (maybe using rsysnc or duplicity to minimize the amount of time this takes, although these VMs are less than 10gb and running on solid state media).
4) Start up the VM.
5) Archive the copy of the VM files that I previously copied to off site storage (aka Dropbox, Backblaze B2, or a RaspberryPI running at home, etc.). I will keep one copy locally as well.
6) Repeat this at set time intervals (once a day, or once a week, etc.)

Questions I have:

1) I know this is probably fairly subjective, but what's the best way to shutdown a VM running a turnkey appliance? What if I have an OwnCloud server running and someone is syncing while the shutdown happens, should it fail gracefully (I guess this is more dependent on OwnCloud's failure mechanisms)?
2) Instead of making a copy of the virtual machine files, I could make an LVM snapshot (this would require setting up my virtual machines on LVM devices, but I've been learning more about LVMs and practicing so I might be able to do this). Thoughts on this? Is there any advantage other than reducing the amount of time my virtual machines have to stay offline?
3) Am I going about this in the wrong manner? Is there some glaringly obvious solution I've missed?
4) My backup script could just run the "halt -p" command via SSH (using PKA of course).

Threads/pages I've read on the matter:
Threads/pages I've read on the matter:
1) viewtopic.php?f=1&t=48807 (Basically shutting down is the best method for a fully recoverable backup).
2) https://www.howtoforge.com/virtualbox-c ... -snapshots (Can't seem to find the date posted. Doesn't seem to account for the fact that not all data is guaranteed to be flushed out at the time of the LVM snapshot).
3) https://www.howtoforge.com/using-raw-de ... ualbox-vms (How to install your VMs on LVM raw devices).
4) viewtopic.php?p=29277#p29277

Now that I have switched from Windows to Linux for my host of preference, I now use a more advanced technique based on hosting my VDIs on a LVM partition. My backup utility uses a VBoxManage guestcontrol execute command to sync the guest file system and then established a temporary readonly snapshot. I then use rsync to mirror this frozen copy of the VM to a separate host before releasing the snapshot. This gives me the ability to hot-copy any VM.

Not really sure how safe this is. Also not clear to me how VBoxManage guestcontrol execute is used. From the VirtualBox documentation it seems that this runs a command of your choosing on the guest.

Thanks in advance for your advice,

Matt

Edit1: Checked "Notify me when a reply is posted."
Edit 2: Added links to additional information.

[EnrgyXprt]

BUMP ?

Great question, looking for same solution but from a windows host...