Create a VMDK that is split into fragments smaller than 2GB?

[SidS]

I have an existing 60GB VMDK disk image that's split into 2GB chunks. I want to clone the disk image into another VMDK but with an even smaller chunk/fragment size. I'm contemplating a chunk size between 200MB and 8MB to make the VMDK more friendly to backup software. Example: If I boot and only 100MB has changed, I want to avoid recording the entire 60GB VMDK to be 'modified' and requiring a full backup. (Details @ superuser(dot)com/questions/364585/time-machine-friendly-virtual-machine-virtualbox-image-files/ - I can't post URLs yet).

I looked at the VBoxManage clonehd command line option but it seems that the Split2G option hardcodes the split size at 2GB (!?). Is there any workaround to this so I can specify a user defined chunk size?

Thanks
Sid

Code: Select all

Usage:

VBoxManage clonehd <uuid>|<filename> <uuid>|<outputfile>
                   [--format VDI|VMDK|VHD|RAW|<other>]
                   [--variant Standard,Fixed,Split2G,Stream,ESX]
                   [--existing]

[mpack]

VirtualBox provides no tools to do that. "Split2G" corresponds to a standard VMDK variant in the VMDK spec, 2GB is not some arbitrary number that the devs dreamed up.

[SidS]

So then the 2GB is not only hardcoded in the code, it's actually 'hardcoded' in the spec itself ? That seems even more weird. I understand the heritage from FAT32 filesystems, but it seems like a very brittle assumption when designing a flexible spec/system that can be deployed in scenarios that have nothing to do with FAT32.

[mpack]

Incidentally: moving this topic to the "Suggestions" forum.

You may dream up alternative uses for the split file scenario, but that doesn't change what the feature was designed for, which was avoid file size limits in various flavours of FAT filesystem and hence provide for maximum portability (*). (2GB-1) bytes is the largest file size that can be expressed in 31 bits, i.e. in a signed 32 bit number. I don't see much point in having metaphysical discussions here about what the VMWare folks shoulda done...

However, the source code is available to you, you can make it do what you want. Your suggestion is now recorded in the proper forum, but I doubt the devteam will be interested unless it suddenly becomes a feature that lots of people want.

(*) The number of open file handles required could also be barrier to maximum portability. A 150GB fixed size VMDK split into (say) 1GB chunks would have 150 chunks. That's a lot of file handles to have open when the drive is in use. And of course the virtual disk will have fragmentation problems by definition.

[SidS]

While I appreciate the switch to the right forum and got bringing this to your dev/product managers, to be frank, I need to give you some feedback. First, I'm not "dreaming up alternatives" - this is rooted in reality on all our Mac dev machines deployed as multi-platform dev platforms. Second, by saying "arbitrary number that the devs dreamed up" in your original message, you allude that that is my implied reasoning. It's not and that's also a bit insulting to the VBox devs. I let it pass by on your first 'dreaming' reference but had to give you feedback on your second reference and drifting references to "metaphysical discussions". I'm sure you already know but as a forum mod, you're also perceived at the technical PR for Oracle in some ways. So being friendly and professional in your operations should rank somewhere.

Back to technical matters, I agree that physical fragmentation is a possibility. Fortunately it isn't a concern on SSDs. The # of file handles open IS a real issue which is why having the chunk size knob to turn would help in picking a decent tradeoff. That concern may diminish as system with 4GB+ RAM become mainstream (already in progress). One of my engineers also mentioned that the expressiveness of the stuff inside the .VMDK may also limit straightforward code changes. Anyway, I'm sure the VBox dev have more experience on this, so lets watch.

[Martin]

VMDK is the VMware virtual disk format. The specs are defined by VMware, not VirtualBox/Innotek/Sun/Oracle, so there is nothing Oracle can do about it.
Also your suggestion is only really useful for a very special environment like your setup. I don't think there are enough Vbox users out there with a similar environment to benefit from such a change.
Nothing keeps you from developing your own virtual disk format, but I don't see a valid business case for Oracle to do it.

[mpack]

To be fair, you could vary the size of VMDK extents within the current VMDK spec (each extent in the split2g variants is 2GB max, and each extent line in the descriptor has its own actual size). I just don't think there is really a good reason to actually do it: keep the virtual image one big file. Let the backup / archiving tool find efficient ways to back that up, or move onto FAT32 media. The latter are occasional requirements: you don't optimize a system for its occasional requirements, you optimize it for its main role!

[LucasHartmann]

I had a similar problem with the same solution as this one. I use sync tools (rsync, unison) for file synchronization between my work computers, and large HDD image files are a problem for those. I coult not find any ready solution for this online, but reading the previous posts allowed me te create one of my own.

As mentioned by mpack VMDK extents may be smaller than 2GB, but there are no tools for creating them in such a way. I got it working by doing the following:

1 - Creating a flat/fixed hard disck as big as a single fragment (128MB, for example).

Code: Select all

VBoxManage createhd --filename disk.vmdk --format VMDK --variant FIXED --size 128MB

2 - Replicating the slice as many times as needed (8 times for 1GB)

Code: Select all

for i in $(seq -w 0001 8); do
   cp disk-flat.vmdk disk-s$i.vmdk
done
rm disk-flat.vmdk

3 - Edit disk.vmdk to use the slices instead of the flat image:
3a - change createtype to "twoGbMaxExtentSparse"
3b - repeat RW line for each slice file.
3c - remove all ddb.geometry lines.

In order to make the process easier i wrote a script (linux, bash, requires gawk).
pastebin dot com/8G4BSgt7

To create a 1GB disk named disk.vmdk made of 128MB fragments use:
./create_split_vmdk.sh --filename disk --fragment 128 --size 1024

Hope this proves useful.

[mpack]

Why do you say "remove all ddb.geometry lines"? I would think they would be required, particularly in the case of disks smaller than 8GB.

[LucasHartmann]

Why do you say "remove all ddb.geometry lines"? I would think they would be required, particularly in the case of disks smaller than 8GB.

The original vmdk file is created together the first slice of the image, so ddb.geometry contains information for just that slice. The comlpete image is created by concatenating the first image several times, resulting in a geometry that is most likely different. I decided it was best to provide no information that to provide bad information, so i removed the ddb.geometry lines and left for virtualbox to figure it out.

On VirtualBox 4.1 this worked tor FIXED expentents, but not for SPARSE ones.
Not tested on VMWAREs.

Please note that I am not a VirtualBox or VMWARE developer and know nothing of how they work inside... This was merely a hunch that worked.

[mpack]

Funny that I seem to be having essentially this same conversation in two different threads. Yes, your hunch is probably ok, though I think technically it may be a deviation from the VMDK spec. It may cause failures if the drive is smaller than 8GB, but above it the BIOS disk CHS addressing maxes out and is effectively switched off.