vbbu backup script

Guideloom · Post by **Guideloom** » 12. Jun 2019, 21:28

For the past month or so, I've been working on a backup script for a Linux based virtualbox host.
We've been using it in our production system for about a month now, and are quite happy with it.

We call it vbbu -> virtualbox back up
If anyone wants to give it a try, I'd love feedback. As of 20190619, version is v2.16.

https://github.com/guideloom/vbbu

Have at it

20190709 update: v2.18 released
20190710 update: v2.21 released
20190726 update: v2.26 released
20190805 update: v2.27 released

Guideloom · Post by **Guideloom** » 9. Jul 2019, 21:12

Update to v2.18.

A few scripting changes to allow data based backup folder names, and a new "daystokeep" option (aka: how long to keep a backup for)
Requires : https://github.com/guideloom/gl_functions.sh to be installed

Guideloom · Post by **Guideloom** » 10. Jul 2019, 19:51

update to v2.21

quashed various bugs.

Guideloom · Post by **Guideloom** » 26. Jul 2019, 20:47

v2.26 released.
- fixed logging indentation
- added disk usage of a backup

usage:

Code: Select all

Usage: ./vbbu [--verbose] [--syslog] [--syslogid SYSLOG_ID_STRING] [--dryrun] [--help|-h]
          [--list PATH_TO_VM_FILE_LIST] [--state running|stopped|paused|saved|poweroff] [--type ova|clone]
          [--exportdir PATH_TO_VM_EXPORT_FOLDER] [--backupdir PATH_TO_VM_BACKUP_FOLDER] [--confdir PATH_TO_CONF_FILES
          [--acpi] [--noconf] [--nodays] [--runbackup]
          [--versions N] [--daystokeep N] [VMNAME|VMUUID]...

 Version : 2.26
       --verbose     = print lines as they run. Useful for debugging only
       --syslog      = send output to syslog as well as stdout [Default: 0]
       --syslogid    = syslog id string to send to syslog [Default: vbbu]
       --list        = full path to list of VMs to backup
       --noconf      = do not use config files. Master conf file/vm conf files under conf folder (/etc/vbbu.d) are ignored
       --nodays      = ignore days option in conf files. Translation: run every day. [Default: off]
       --state       = only backup VMs whose status is one of running|stopped|paused|saved|poweroff. [Default: not set, aka any]
       --type        = type of backup to create. One of ova|clone. [Default: ova]
       --exportdir   = path to temporary export directory, [Default: /mnt/lv001-r0/backup/vms]
       --backupdir   = path to final backup directory. [Default: /mnt/usb1/backup/vms]
       --versions    = number of versions to keep in BACKUPDIR. [Default: 2]
       --daystokeep  = number of days to keep backups for. Ones older are removed. [Default: 0]
                       Note: if daystokeep is set, this OVERRIDES the --versions option.
       --acpi        = issue acpishutdown instead of savestate. Fixes bug in vbox 5.X sometimes causes kernel panic on vm restart.
       --dryrun      = Limited run. Display commands, and do not run them. [Default: off]
       --help        = this help info
       --runbackup   = Actually run. Safety switch. Prevents accidently running backups and pausing VMs

       VMNAME|VMUUID = VM to backup. Can list more then one. If not set, fallback to list.

  Note: Options can also be set in /etc/vbbu.conf or /etc/vbbu.d/VMNAME.conf

Sample logging:

Code: Select all

Jul 25 23:00:01 dev01 vbbu: -- [Buildroot 2018.02] VMs with space in their names are not supported at this time. Skipping
Jul 25 23:00:01 dev01 vbbu: -- [vl001] cannot backup. VM day mismatch. [VM days:Wed Sat] [Today:Thu or 25]
Jul 25 23:00:01 dev01 vbbu: -- [vl002] cannot backup. VM day mismatch. [VM days:Wed Sat] [Today:Thu or 25]
Jul 25 23:00:01 dev01 vbbu: -- [vpn1] cannot backup. VM set to never backup. [VM days:never]
Jul 25 23:00:02 dev01 vbbu: -- [vw001] Start backup [State:running] [Days:Mon Thu] [Type:ova] [Shutdown:savestate]
Jul 25 23:00:02 dev01 vbbu:     Begin VM savestate
Jul 25 23:00:57 dev01 vbbu:     End VM savestate. 00:00:55
Jul 25 23:00:57 dev01 vbbu:     Begin Clone : [vw001-20190725-230002-vboxbu]
Jul 25 23:00:57 dev01 vbbu:      Disk free before clonevm /mnt/lv001-r0/backup/vms : 1710904MB
Jul 25 23:14:53 dev01 vbbu:       Exportdir size: 82892MB
Jul 25 23:14:53 dev01 vbbu:      Disk free after clonevm /mnt/lv001-r0/backup/vms : 1628012MB
Jul 25 23:14:53 dev01 vbbu:     End Clone export. 00:13:56
Jul 25 23:14:53 dev01 vbbu:     Begin VM restore state
Jul 25 23:15:07 dev01 vbbu:     End VM restore state. 00:00:14
Jul 25 23:15:07 dev01 vbbu:     Begin VM register for OVA export : [vw001-20190725-230002-vboxbu] [running]
Jul 25 23:15:07 dev01 vbbu:     End VM register for OVA export. 00:00:00
Jul 25 23:15:07 dev01 vbbu:     Begin OVA export: [vw001-20190725-230002.ova]
Jul 25 23:15:07 dev01 vbbu:      Disk free before OVA export /mnt/lv001-r0/backup/vms : 1628012MB
Jul 25 23:52:33 dev01 vbbu:       OVA size: 52248MB
Jul 25 23:52:33 dev01 vbbu:      Disk free after OVA export /mnt/lv001-r0/backup/vms : 1575764MB
Jul 25 23:52:33 dev01 vbbu:     End OVA export. 00:37:26
Jul 25 23:52:33 dev01 vbbu:     Begin VM unregister from OVA export : [vw001-20190725-230002-vboxbu] [running]
Jul 25 23:52:33 dev01 vbbu:     End VM unregister from OVA export. 00:00:00
Jul 25 23:52:33 dev01 vbbu:     Removing folder (+14 days old) /mnt/usb1/backup/vms/vw001/vw001.20190708-220002
Jul 25 23:52:34 dev01 vbbu:     Begin VM move from export to backup : [vw001]
Jul 25 23:52:34 dev01 vbbu:      Disk free before move to backup /mnt/usb1/backup/vms/vw001/vw001.20190725-230002 : 4895272MB
Jul 25 23:58:11 dev01 vbbu:       Backup folder size: 52248MB
Jul 25 23:58:12 dev01 vbbu:      Disk free after move to backup /mnt/usb1/backup/vms/vw001/vw001.20190725-230002 : 4843024MB
Jul 25 23:58:12 dev01 vbbu:     End VM move.  00:05:37
Jul 25 23:58:12 dev01 vbbu: -- [vw001] End backup [State:running] 00:58:10
Jul 25 23:58:12 dev01 vbbu: -- [vl005] cannot backup. VM day mismatch. [VM days:Mon] [Today:Thu or 25]
Jul 25 23:58:12 dev01 vbbu: -- [vl006] cannot backup. VM day mismatch. [VM days:Tue] [Today:Thu or 25]
Jul 25 23:58:12 dev01 vbbu: -- [vl003] cannot backup. VM day mismatch. [VM days:Sat Mon Wed Fri] [Today:Thu or 25]
Jul 25 23:58:12 dev01 vbbu: -- [vl004] cannot backup. VM day mismatch. [VM days:Sun] [Today:Thu or 25]
Jul 25 23:58:13 dev01 vbbu: -- [vl007] Start backup [State:running] [Days:Sun Tue Thu] [Type:ova] [Shutdown:savestate]
Jul 25 23:58:13 dev01 vbbu:     Begin VM savestate
Jul 25 23:58:22 dev01 vbbu:     End VM savestate. 00:00:09
Jul 25 23:58:22 dev01 vbbu:     Begin Clone : [vl007-20190725-235813-vboxbu]
Jul 25 23:58:22 dev01 vbbu:      Disk free before clonevm /mnt/lv001-r0/backup/vms : 1710904MB
Jul 26 00:41:39 dev01 vbbu:       Exportdir size: 261723MB
Jul 26 00:41:39 dev01 vbbu:      Disk free after clonevm /mnt/lv001-r0/backup/vms : 1449181MB
Jul 26 00:41:39 dev01 vbbu:     End Clone export. 00:43:17
Jul 26 00:41:39 dev01 vbbu:     Begin VM restore state
Jul 26 00:41:41 dev01 vbbu:     End VM restore state. 00:00:02
Jul 26 00:41:41 dev01 vbbu:     Begin VM register for OVA export : [vl007-20190725-235813-vboxbu] [running]
Jul 26 00:41:41 dev01 vbbu:     End VM register for OVA export. 00:00:00
Jul 26 00:41:41 dev01 vbbu:     Begin OVA export: [vl007-20190725-235813.ova]
Jul 26 00:41:41 dev01 vbbu:      Disk free before OVA export /mnt/lv001-r0/backup/vms : 1449181MB
Jul 26 02:29:11 dev01 vbbu:       OVA size: 140291MB
Jul 26 02:29:11 dev01 vbbu:      Disk free after OVA export /mnt/lv001-r0/backup/vms : 1308890MB
Jul 26 02:29:11 dev01 vbbu:     End OVA export. 01:47:30
Jul 26 02:29:11 dev01 vbbu:     Begin VM unregister from OVA export : [vl007-20190725-235813-vboxbu] [running]
Jul 26 02:29:11 dev01 vbbu:     End VM unregister from OVA export. 00:00:00
Jul 26 02:29:11 dev01 vbbu:     Removing folder (+7 days old) /mnt/usb1/backup/vms/vl007/vl007.20190716-220931
Jul 26 02:29:13 dev01 vbbu:     Begin VM move from export to backup : [vl007]
Jul 26 02:29:13 dev01 vbbu:      Disk free before move to backup /mnt/usb1/backup/vms/vl007/vl007.20190725-235813 : 4990130MB
Jul 26 02:49:14 dev01 vbbu:       Backup folder size: 140291MB
Jul 26 02:49:14 dev01 vbbu:      Disk free after move to backup /mnt/usb1/backup/vms/vl007/vl007.20190725-235813 : 4849839MB
Jul 26 02:49:14 dev01 vbbu:     End VM move.  00:20:01
Jul 26 02:49:14 dev01 vbbu: -- [vl007] End backup [State:running] 02:51:01
Jul 26 02:49:14 dev01 vbbu: -- [qemu01] cannot backup. VM day mismatch. [VM days:Sun] [Today:Thu or 25]
Jul 26 02:49:15 dev01 vbbu: -- [vl008] cannot backup. VM day mismatch. [VM days:Mon] [Today:Thu or 25]
Jul 26 02:49:15 dev01 vbbu: -- [UB-18.01.02-base] Start backup [State:poweroff] [Days:Thu] [Type:ova] [Shutdown:savestate]
Jul 26 02:49:15 dev01 vbbu:     Begin Clone : [UB-18.01.02-base-20190726-024915-vboxbu]
Jul 26 02:49:15 dev01 vbbu:      Disk free before clonevm /mnt/lv001-r0/backup/vms : 1710904MB
Jul 26 02:49:52 dev01 vbbu:       Exportdir size: 3920MB
Jul 26 02:49:52 dev01 vbbu:      Disk free after clonevm /mnt/lv001-r0/backup/vms : 1706984MB
Jul 26 02:49:52 dev01 vbbu:     End Clone export. 00:00:37
Jul 26 02:49:52 dev01 vbbu:     Begin VM register for OVA export : [UB-18.01.02-base-20190726-024915-vboxbu] [poweroff]
Jul 26 02:49:52 dev01 vbbu:     End VM register for OVA export. 00:00:00
Jul 26 02:49:52 dev01 vbbu:     Begin OVA export: [UB-18.01.02-base-20190726-024915.ova]
Jul 26 02:49:52 dev01 vbbu:      Disk free before OVA export /mnt/lv001-r0/backup/vms : 1706984MB
Jul 26 02:51:32 dev01 vbbu:       OVA size: 1462MB
Jul 26 02:51:32 dev01 vbbu:      Disk free after OVA export /mnt/lv001-r0/backup/vms : 1705521MB
Jul 26 02:51:32 dev01 vbbu:     End OVA export. 00:01:40
Jul 26 02:51:32 dev01 vbbu:     Begin VM unregister from OVA export : [UB-18.01.02-base-20190726-024915-vboxbu] [poweroff]
Jul 26 02:51:32 dev01 vbbu:     End VM unregister from OVA export. 00:00:00
Jul 26 02:51:32 dev01 vbbu:     Removing folder (+7 days old) /mnt/usb1/backup/vms/UB-18.01.02-base/UB-18.01.02-base.20190712-015009
Jul 26 02:51:32 dev01 vbbu:     Begin VM move from export to backup : [UB-18.01.02-base]
Jul 26 02:51:32 dev01 vbbu:      Disk free before move to backup /mnt/usb1/backup/vms/UB-18.01.02-base/UB-18.01.02-base.20190726-024915 : 4851301MB
Jul 26 02:51:34 dev01 vbbu:       Backup folder size: 1462MB
Jul 26 02:51:34 dev01 vbbu:      Disk free after move to backup /mnt/usb1/backup/vms/UB-18.01.02-base/UB-18.01.02-base.20190726-024915 : 4849839MB
Jul 26 02:51:34 dev01 vbbu:     End VM move.  00:00:02
Jul 26 02:51:34 dev01 vbbu: -- [UB-18.01.02-base] End backup [State:poweroff] 00:02:19
Jul 26 02:51:34 dev01 vbbu: -- [vl009] cannot backup. VM day mismatch. [VM days:Wed] [Today:Thu or 25]
Jul 26 02:51:34 dev01 vbbu: -- [di1] cannot backup. VM day mismatch. [VM days:Tue] [Today:Thu or 25]
Jul 26 02:51:34 dev01 vbbu: -- [di2] cannot backup. VM day mismatch. [VM days:Wed] [Today:Thu or 25]
Jul 26 02:51:35 dev01 vbbu: -- [di3] Start backup [State:running] [Days:Thu] [Type:ova] [Shutdown:savestate]
Jul 26 02:51:35 dev01 vbbu:     Begin VM savestate
Jul 26 02:51:39 dev01 vbbu:     End VM savestate. 00:00:04
Jul 26 02:51:39 dev01 vbbu:     Begin Clone : [di3-20190726-025135-vboxbu]
Jul 26 02:51:39 dev01 vbbu:      Disk free before clonevm /mnt/lv001-r0/backup/vms : 1710904MB
Jul 26 02:52:51 dev01 vbbu:       Exportdir size: 7625MB
Jul 26 02:52:51 dev01 vbbu:      Disk free after clonevm /mnt/lv001-r0/backup/vms : 1703279MB
Jul 26 02:52:51 dev01 vbbu:     End Clone export. 00:01:12
Jul 26 02:52:51 dev01 vbbu:     Begin VM restore state
Jul 26 02:52:53 dev01 vbbu:     End VM restore state. 00:00:02
Jul 26 02:52:53 dev01 vbbu:     Begin VM register for OVA export : [di3-20190726-025135-vboxbu] [running]
Jul 26 02:52:53 dev01 vbbu:     End VM register for OVA export. 00:00:00
Jul 26 02:52:53 dev01 vbbu:     Begin OVA export: [di3-20190726-025135.ova]
Jul 26 02:52:53 dev01 vbbu:      Disk free before OVA export /mnt/lv001-r0/backup/vms : 1703279MB
Jul 26 02:56:16 dev01 vbbu:       OVA size: 2672MB
Jul 26 02:56:16 dev01 vbbu:      Disk free after OVA export /mnt/lv001-r0/backup/vms : 1700607MB
Jul 26 02:56:16 dev01 vbbu:     End OVA export. 00:03:23
Jul 26 02:56:16 dev01 vbbu:     Begin VM unregister from OVA export : [di3-20190726-025135-vboxbu] [running]
Jul 26 02:56:16 dev01 vbbu:     End VM unregister from OVA export. 00:00:00
Jul 26 02:56:16 dev01 vbbu:     Begin VM move from export to backup : [di3]
Jul 26 02:56:16 dev01 vbbu:      Disk free before move to backup /mnt/usb1/backup/vms/di3/di3.20190726-025135 : 4849839MB
Jul 26 02:56:18 dev01 vbbu:       Backup folder size: 2672MB
Jul 26 02:56:18 dev01 vbbu:      Disk free after move to backup /mnt/usb1/backup/vms/di3/di3.20190726-025135 : 4847166MB
Jul 26 02:56:18 dev01 vbbu:     End VM move.  00:00:02
Jul 26 02:56:18 dev01 vbbu: -- [di3] End backup [State:running] 00:04:43
Jul 26 02:56:18 dev01 vbbu: -- [dvl001] Start backup [State:running] [Days:Thu] [Type:ova] [Shutdown:savestate]
Jul 26 02:56:18 dev01 vbbu:     Begin VM savestate
Jul 26 02:56:21 dev01 vbbu:     End VM savestate. 00:00:03
Jul 26 02:56:21 dev01 vbbu:     Begin Clone : [dvl001-20190726-025618-vboxbu]
Jul 26 02:56:21 dev01 vbbu:      Disk free before clonevm /mnt/lv001-r0/backup/vms : 1710904MB
Jul 26 02:58:09 dev01 vbbu:       Exportdir size: 11368MB
Jul 26 02:58:09 dev01 vbbu:      Disk free after clonevm /mnt/lv001-r0/backup/vms : 1699536MB
Jul 26 02:58:09 dev01 vbbu:     End Clone export. 00:01:48
Jul 26 02:58:09 dev01 vbbu:     Begin VM restore state
Jul 26 02:58:11 dev01 vbbu:     End VM restore state. 00:00:02
Jul 26 02:58:11 dev01 vbbu:     Begin VM register for OVA export : [dvl001-20190726-025618-vboxbu] [running]
Jul 26 02:58:11 dev01 vbbu:     End VM register for OVA export. 00:00:00
Jul 26 02:58:11 dev01 vbbu:     Begin OVA export: [dvl001-20190726-025618.ova]
Jul 26 02:58:11 dev01 vbbu:      Disk free before OVA export /mnt/lv001-r0/backup/vms : 1699536MB
Jul 26 03:03:52 dev01 vbbu:       OVA size: 4499MB
Jul 26 03:03:52 dev01 vbbu:      Disk free after OVA export /mnt/lv001-r0/backup/vms : 1695036MB
Jul 26 03:03:52 dev01 vbbu:     End OVA export. 00:05:41
Jul 26 03:03:52 dev01 vbbu:     Begin VM unregister from OVA export : [dvl001-20190726-025618-vboxbu] [running]
Jul 26 03:03:52 dev01 vbbu:     End VM unregister from OVA export. 00:00:00
Jul 26 03:03:52 dev01 vbbu:     Begin VM move from export to backup : [dvl001]
Jul 26 03:03:52 dev01 vbbu:      Disk free before move to backup /mnt/usb1/backup/vms/dvl001/dvl001.20190726-025618 : 4847166MB
Jul 26 03:03:56 dev01 vbbu:       Backup folder size: 4499MB
Jul 26 03:03:56 dev01 vbbu:      Disk free after move to backup /mnt/usb1/backup/vms/dvl001/dvl001.20190726-025618 : 4842667MB
Jul 26 03:03:56 dev01 vbbu:     End VM move.  00:00:04
Jul 26 03:03:56 dev01 vbbu: -- [dvl001] End backup [State:running] 00:07:38
Jul 26 03:03:56 dev01 vbbu: -- [scbase] cannot backup. VM day mismatch. [VM days:Sun] [Today:Thu or 25]
Jul 26 03:03:56 dev01 vbbu: -- [di4] Start backup [State:running] [Days:] [Type:ova] [Shutdown:savestate]
Jul 26 03:03:56 dev01 vbbu:     Begin VM savestate
Jul 26 03:04:00 dev01 vbbu:     End VM savestate. 00:00:04
Jul 26 03:04:00 dev01 vbbu:     Begin Clone : [di4-20190726-030356-vboxbu]
Jul 26 03:04:00 dev01 vbbu:      Disk free before clonevm /mnt/lv001-r0/backup/vms : 1710904MB
Jul 26 03:05:03 dev01 vbbu:       Exportdir size: 6289MB
Jul 26 03:05:03 dev01 vbbu:      Disk free after clonevm /mnt/lv001-r0/backup/vms : 1704615MB
Jul 26 03:05:03 dev01 vbbu:     End Clone export. 00:01:03
Jul 26 03:05:03 dev01 vbbu:     Begin VM restore state
Jul 26 03:05:05 dev01 vbbu:     End VM restore state. 00:00:02
Jul 26 03:05:05 dev01 vbbu:     Begin VM register for OVA export : [di4-20190726-030356-vboxbu] [running]
Jul 26 03:05:05 dev01 vbbu:     End VM register for OVA export. 00:00:00
Jul 26 03:05:05 dev01 vbbu:     Begin OVA export: [di4-20190726-030356.ova]
Jul 26 03:05:05 dev01 vbbu:      Disk free before OVA export /mnt/lv001-r0/backup/vms : 1704615MB
Jul 26 03:08:10 dev01 vbbu:       OVA size: 2514MB
Jul 26 03:08:10 dev01 vbbu:      Disk free after OVA export /mnt/lv001-r0/backup/vms : 1702101MB
Jul 26 03:08:10 dev01 vbbu:     End OVA export. 00:03:05
Jul 26 03:08:10 dev01 vbbu:     Begin VM unregister from OVA export : [di4-20190726-030356-vboxbu] [running]
Jul 26 03:08:10 dev01 vbbu:     End VM unregister from OVA export. 00:00:00
Jul 26 03:08:10 dev01 vbbu:     Begin VM move from export to backup : [di4]
Jul 26 03:08:10 dev01 vbbu:      Disk free before move to backup /mnt/usb1/backup/vms/di4/di4.20190726-030356 : 4842667MB
Jul 26 03:08:12 dev01 vbbu:       Backup folder size: 2514MB
Jul 26 03:08:12 dev01 vbbu:      Disk free after move to backup /mnt/usb1/backup/vms/di4/di4.20190726-030356 : 4840152MB
Jul 26 03:08:12 dev01 vbbu:     End VM move.  00:00:02
Jul 26 03:08:12 dev01 vbbu: -- [di4] End backup [State:running] 00:04:16
Jul 26 03:08:12 dev01 vbbu: -- [dvl002] cannot backup. VM day mismatch. [VM days:Fri] [Today:Thu or 25]

Guideloom · Post by **Guideloom** » 5. Aug 2019, 23:02

v2.27 released.
- update backuptag to match command (-vbbu)
- update readme to include the following warning:

NOTE: with Virtualbox 6.0.10 (all versions of 6 earlier then 6.0.10?) there is a bug in which if the VM is on an NFS (and possibly SMB) share, AND you use "savestate", the clone of the VM will fail with error VBOX_E_IPRT_ERROR (0x80BB0005). This is a known issue. (https://www.virtualbox.org/ticket/18811). The workaround is to either run Virtualbox 5.2.X, or move your VMs to a local ext4 filesystem, or use the acpi option outlined below. Will update this if there are changes.

virtualbox.org

vbbu backup script

vbbu backup script

Re: vbbu backup script

Re: vbbu backup script

Re: vbbu backup script

Re: vbbu backup script