ZFS + VirtualBox

Discussions related to using VirtualBox on Solaris hosts.

ZFS + VirtualBox

Postby sgt_hartman » 27. Nov 2009, 13:29

Hi everybody,

I've done a Perl program that make clones of VirtualBox machines in a fraction of a second.
How can this script run so fast? The reason behind this speed is the teaming of ZFS clone + snapshot commands, and VirtualBox createvm/modifyvm commands.

The benefits are:
- Virtual machine clones are made in less than a second.
- Virtual machine clones take approximately 200 kilobytes (yes, kilobytes :mrgreen: ): who knows ZFS can verify my claims...
- You don't have to copy gigabytes of virtual disks back and forth.
- You can make 20 clones in 10 seconds on a mainstream notebook or workstation with OpenSolaris.
- This system is well suited for testing purposes: every virtual machine take only the disk space necessary for the application you have to test.
- ZFS datasets that contains virtual disks can have the compression flag activated: you can spare more near 50% of disk space with no or zero cpu overhead

The program is about 250 lines long and with a reasonable number of comments.
I've done also a remake of Joerg Moellenkamp ZFSviz, to ease the display of parent-child relationships of ZFS dataset.

I'm not a professional developer, only a sysadmin: I ask the forum how is the best way to release these two programs to the public free to use and modify.

Here is the command output (no, it's not a joke)

marco@cypher.csf.it:~/Software/VirtualBox$ /opt/bin/perl zfs_clonevm.pl -s mint7
[mint7] -> [mint7_C000004]
Checking snapshot(s)...
All checks OK, cloning [mint7] to [mint7_C000004]...
Snapshot not specified or non existent:
I'll check if filesystem [rpool/export/home/marco/virtual_disks/mint7] has one.
Snapshot found.
Using snapshot [rpool/export/home/marco/virtual_disks/mint7@20091126].
Clone [rpool/export/home/marco/virtual_disks/mint7_C000004] created.
Attachments
gviz.jpg
virtual_disks ZFS dataset
gviz.jpg (99.29 KiB) Viewed 12104 times
sgt_hartman
 
Posts: 5
Joined: 27. Nov 2009, 13:11
Primary OS: OpenSolaris 11
VBox Version: PUEL
Guest OSses: Any

Re: ZFS + VirtualBox

Postby rdhalstead » 30. Nov 2009, 20:10

Where's the script. I would love to try it.

--ron
rdhalstead
 
Posts: 52
Joined: 26. Feb 2008, 18:58
Location: Las Vegas

Re: ZFS + VirtualBox

Postby kebabbert » 1. Dec 2009, 16:55

How did you draw that picture? With MS powerpoint?

PS. And where is the script?
kebabbert
Volunteer
 
Posts: 320
Joined: 31. May 2008, 10:00
Primary OS: OpenSolaris 11
VBox Version: OSE other
Guest OSses: WinXP, RedHat, Ubuntu

Re: ZFS + VirtualBox

Postby sgt_hartman » 2. Dec 2009, 13:00

First of all, the picture is drawn with a remake of ZFSviz.pl (from an idea of Joerg Moellenkamp).

I've attached two programs to this post: zfsviz.pl and zfs_clonevm.pl.

You have to install the SUNWgviz package to see the graphs (zfsviz.pl), and remember to add /opt/sfw/bin to your $PATH, because the program doesn't use an absolute path to call it.
The default behavior of zfsviz.pl is to draw a picture of a pool named rpool, but you can specify another pool or a children dataset as starting point.

Im the following example I draw the ZFS dataset virtual_disks inside a RAID1 pool called raid1:

Code: Select all   Expand viewCollapse view
$ /usr/bin/perl zfsviz10.pl -d raid1/virtual_disks


Just another note: zfsviz.pl has a kind of verbose output. If you add the -v switch on the command line, you'll see a slightly different graph. In each shape you'll see the space used by each dataset (real disk space), and a very important value that is compressratio: this is a value that starts from 1.00x and can be higher than it's start value if ZFS can compress data inside datasets.
To get an idea of how much space your dataset would have taken without compression, just multiply the used value by the compressratio value. You'll smile at the result :wink:

I mean this:
gviz-verbose-crop.jpg
zfsviz verbose output
gviz-verbose-crop.jpg (34.14 KiB) Viewed 11870 times



Now something about zfs_clonevm.pl...

Note that zfs_clonevm.pl has a configuration file, a very simple one: you have to change the settings according to your system setup...
It's all rather intuitive, but I'll post documentation as soon as I can write it.

IMPORTANT: zfs_clonevm, doesn't use pfexec calling zfs commands, so your user has to have delegated ZFS administration.
To get an example, look at program comments, but to summarize you have to write this:

Code: Select all   Expand viewCollapse view
$ pfexec zfs allow user_to_delegate_to create,destroy,mount,clone,snapshot rpool/export/home/user_to_delegate_to/virtual_disks


zfs_clonevm.pl requires the module XML::LibXML, while zfsviz requires the Perl Graphviz module.

There are some limitations about zfs_clonevm.pl for now:

1 - You have to put .vdi files in dedicated ZFS datasets under a "root" dataset.
2 - .vdi files have to have the same name as the ZFS dataset AND the virtual machine in VirtualBox.
Example:
virtual machine name: win_xp
dataset name: rpool/export/home/user_to_delegate_to/virtual_disks/win_xp
virtual disk name: /export/home/user_to_delegate_to/virtual_disks/win_xp/win_xp.vdi
3 - For the time being, zfs_clonevm.pl can clone oly machines with one SATA or IDE controller (SCSI is coming...) and only one virtual disk.
4 - I don't know if zfs_clonevm.pl works with the fresh new VB 3.1.0: I have to see the XML yet...

I'll check the forum from time to time, to see if help is needed. Comments and suggestions are welcome.

Enjoy :mrgreen:
Attachments
zfs_clonevm_2010.zip
Updated!
(5.07 KiB) Downloaded 48 times
zfsviz10.pl.zip
With this program you can draw graphs of ZFS file systems like the one I posted. Njoy!
(3.22 KiB) Downloaded 139 times
Last edited by sgt_hartman on 19. Mar 2013, 12:50, edited 1 time in total.
sgt_hartman
 
Posts: 5
Joined: 27. Nov 2009, 13:11
Primary OS: OpenSolaris 11
VBox Version: PUEL
Guest OSses: Any

ZFS + VirtualBox - 3.2.0 refactor

Postby sgt_hartman » 5. Jun 2010, 00:11

I wrote zfs_clonevm.pl in a hurry when VirtualBox version was 3.1.2, and I didn't knew a lot of things...

Time has passed, and little changes to the XML describing each virtual machine, turned the program in a useless pile of code.

So I read the new VirtualBox 3.2.0 User Manual searching for inspiration and maybe I found it.
I am going to use existing .xml files in a clever and more efficiend way, leaving apart the heavy use of XPath/DOM.
Here is the how (I'll post new code ASAP).

The command

Code: Select all   Expand viewCollapse view
VBoxManage createvm --name test_vm


creates a new configuration file for a new virtual machine, ~/.VirtualBox/Machines/test_vm.xml, but doesn't register the machine.
Inside this file there is a fresh new uuid: we "write down" this new uuid because we'll use it soon.

We'll not see test_vm inside VirtualBox GUI, until we give the command

Code: Select all   Expand viewCollapse view
VBoxManage registervm test_vm.xml


so what we get is a generic virtual machine XML definition file, pretty useless excep for his uuid. We are going to use a copy of an existing XML definition file: a quick change of uuid and presto, we have a new machine to register. :mrgreen:
When we try to register the modified XML definition, there are two problems that prevent us to succeed.
First and foremost, we are trying to register a new virtual machine that will use a .vdi image already attached to the original virtual machine. VirtualBox will catch this error and will not comply.
Then, as aptly noted in the User Manual, the subcommand registervm does not register XML definition files that have .vdi images attached, even if in an exclusive manner.
So we need to remove any .vdi image from the XML definition, register the new virtual machine, clone .vdi image files (all of them), and attach them at the right controller.

It can seem tricky, but in Perl code is about half lines of the previous version.
sgt_hartman
 
Posts: 5
Joined: 27. Nov 2009, 13:11
Primary OS: OpenSolaris 11
VBox Version: PUEL
Guest OSses: Any

Re: ZFS + VirtualBox

Postby larstobi » 19. Mar 2013, 13:08

Thanks for re-adding the zfs_clonevm.pl script!
larstobi
 
Posts: 1
Joined: 19. Jan 2013, 14:11


Return to VirtualBox on Solaris Hosts

Who is online

Users browsing this forum: No registered users and 2 guests