Create a VMDK that is split into fragments smaller than 2GB?

Here you can provide suggestions on how to improve the product, website, etc.

Create a VMDK that is split into fragments smaller than 2GB?

Postby SidS » 8. Mar 2012, 11:24

I have an existing 60GB VMDK disk image that's split into 2GB chunks. I want to clone the disk image into another VMDK but with an even smaller chunk/fragment size. I'm contemplating a chunk size between 200MB and 8MB to make the VMDK more friendly to backup software. Example: If I boot and only 100MB has changed, I want to avoid recording the entire 60GB VMDK to be 'modified' and requiring a full backup. (Details @ superuser(dot)com/questions/364585/time-machine-friendly-virtual-machine-virtualbox-image-files/ - I can't post URLs yet).

I looked at the VBoxManage clonehd command line option but it seems that the Split2G option hardcodes the split size at 2GB (!?). Is there any workaround to this so I can specify a user defined chunk size?

Thanks
Sid

Code: Select all   Expand viewCollapse view
Usage:

VBoxManage clonehd <uuid>|<filename> <uuid>|<outputfile>
                   [--format VDI|VMDK|VHD|RAW|<other>]
                   [--variant Standard,Fixed,Split2G,Stream,ESX]
                   [--existing]
SidS
 
Posts: 4
Joined: 8. Mar 2012, 11:14
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win7 x64

Re: Create a VMDK that is split into fragments smaller than

Postby mpack » 8. Mar 2012, 11:32

VirtualBox provides no tools to do that. "Split2G" corresponds to a standard VMDK variant in the VMDK spec, 2GB is not some arbitrary number that the devs dreamed up.
mpack
Site Moderator
 
Posts: 29307
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Create a VMDK that is split into fragments smaller than

Postby SidS » 8. Mar 2012, 15:17

So then the 2GB is not only hardcoded in the code, it's actually 'hardcoded' in the spec itself ? That seems even more weird. I understand the heritage from FAT32 filesystems, but it seems like a very brittle assumption when designing a flexible spec/system that can be deployed in scenarios that have nothing to do with FAT32.
SidS
 
Posts: 4
Joined: 8. Mar 2012, 11:14
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win7 x64

Re: Create a VMDK that is split into fragments smaller than

Postby mpack » 8. Mar 2012, 15:31

Incidentally: moving this topic to the "Suggestions" forum.

You may dream up alternative uses for the split file scenario, but that doesn't change what the feature was designed for, which was avoid file size limits in various flavours of FAT filesystem and hence provide for maximum portability (*). (2GB-1) bytes is the largest file size that can be expressed in 31 bits, i.e. in a signed 32 bit number. I don't see much point in having metaphysical discussions here about what the VMWare folks shoulda done...

However, the source code is available to you, you can make it do what you want. Your suggestion is now recorded in the proper forum, but I doubt the devteam will be interested unless it suddenly becomes a feature that lots of people want.

(*) The number of open file handles required could also be barrier to maximum portability. A 150GB fixed size VMDK split into (say) 1GB chunks would have 150 chunks. That's a lot of file handles to have open when the drive is in use. And of course the virtual disk will have fragmentation problems by definition.
mpack
Site Moderator
 
Posts: 29307
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Create a VMDK that is split into fragments smaller than

Postby SidS » 9. Mar 2012, 00:50

While I appreciate the switch to the right forum and got bringing this to your dev/product managers, to be frank, I need to give you some feedback. First, I'm not "dreaming up alternatives" - this is rooted in reality on all our Mac dev machines deployed as multi-platform dev platforms. Second, by saying "arbitrary number that the devs dreamed up" in your original message, you allude that that is my implied reasoning. It's not and that's also a bit insulting to the VBox devs. I let it pass by on your first 'dreaming' reference but had to give you feedback on your second reference and drifting references to "metaphysical discussions". I'm sure you already know but as a forum mod, you're also perceived at the technical PR for Oracle in some ways. So being friendly and professional in your operations should rank somewhere.

Back to technical matters, I agree that physical fragmentation is a possibility. Fortunately it isn't a concern on SSDs. The # of file handles open IS a real issue which is why having the chunk size knob to turn would help in picking a decent tradeoff. That concern may diminish as system with 4GB+ RAM become mainstream (already in progress). One of my engineers also mentioned that the expressiveness of the stuff inside the .VMDK may also limit straightforward code changes. Anyway, I'm sure the VBox dev have more experience on this, so lets watch.
SidS
 
Posts: 4
Joined: 8. Mar 2012, 11:14
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win7 x64

Re: Create a VMDK that is split into fragments smaller than

Postby Martin » 9. Mar 2012, 22:32

VMDK is the VMware virtual disk format. The specs are defined by VMware, not VirtualBox/Innotek/Sun/Oracle, so there is nothing Oracle can do about it.
Also your suggestion is only really useful for a very special environment like your setup. I don't think there are enough Vbox users out there with a similar environment to benefit from such a change.
Nothing keeps you from developing your own virtual disk format, but I don't see a valid business case for Oracle to do it.
Martin
Volunteer
 
Posts: 2190
Joined: 30. May 2007, 18:05
Primary OS: Fedora other
VBox Version: PUEL
Guest OSses: XP, Win7, Linux, OS/2

Re: Create a VMDK that is split into fragments smaller than

Postby mpack » 10. Mar 2012, 11:49

To be fair, you could vary the size of VMDK extents within the current VMDK spec (each extent in the split2g variants is 2GB max, and each extent line in the descriptor has its own actual size). I just don't think there is really a good reason to actually do it: keep the virtual image one big file. Let the backup / archiving tool find efficient ways to back that up, or move onto FAT32 media. The latter are occasional requirements: you don't optimize a system for its occasional requirements, you optimize it for its main role!
mpack
Site Moderator
 
Posts: 29307
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Create a VMDK that is split into fragments smaller than

Postby LucasHartmann » 12. Dec 2012, 17:51

I had a similar problem with the same solution as this one. I use sync tools (rsync, unison) for file synchronization between my work computers, and large HDD image files are a problem for those. I coult not find any ready solution for this online, but reading the previous posts allowed me te create one of my own.

As mentioned by mpack VMDK extents may be smaller than 2GB, but there are no tools for creating them in such a way. I got it working by doing the following:

1 - Creating a flat/fixed hard disck as big as a single fragment (128MB, for example).
Code: Select all   Expand viewCollapse view
VBoxManage createhd --filename disk.vmdk --format VMDK --variant FIXED --size 128MB


2 - Replicating the slice as many times as needed (8 times for 1GB)
Code: Select all   Expand viewCollapse view
for i in $(seq -w 0001 8); do
   cp disk-flat.vmdk disk-s$i.vmdk
done
rm disk-flat.vmdk


3 - Edit disk.vmdk to use the slices instead of the flat image:
3a - change createtype to "twoGbMaxExtentSparse"
3b - repeat RW line for each slice file.
3c - remove all ddb.geometry lines.

In order to make the process easier i wrote a script (linux, bash, requires gawk).
pastebin dot com/8G4BSgt7

To create a 1GB disk named disk.vmdk made of 128MB fragments use:
./create_split_vmdk.sh --filename disk --fragment 128 --size 1024

Hope this proves useful. :-)
LucasHartmann
 
Posts: 2
Joined: 12. Dec 2012, 17:29
Primary OS: Ubuntu 12.12
VBox Version: PUEL
Guest OSses: WinXP

Re: Create a VMDK that is split into fragments smaller than

Postby mpack » 12. Dec 2012, 18:46

Why do you say "remove all ddb.geometry lines"? I would think they would be required, particularly in the case of disks smaller than 8GB.
mpack
Site Moderator
 
Posts: 29307
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Create a VMDK that is split into fragments smaller than

Postby LucasHartmann » 12. Dec 2012, 19:07

mpack wrote:Why do you say "remove all ddb.geometry lines"? I would think they would be required, particularly in the case of disks smaller than 8GB.


The original vmdk file is created together the first slice of the image, so ddb.geometry contains information for just that slice. The comlpete image is created by concatenating the first image several times, resulting in a geometry that is most likely different. I decided it was best to provide no information that to provide bad information, so i removed the ddb.geometry lines and left for virtualbox to figure it out.

On VirtualBox 4.1 this worked tor FIXED expentents, but not for SPARSE ones.
Not tested on VMWAREs.

Please note that I am not a VirtualBox or VMWARE developer and know nothing of how they work inside... This was merely a hunch that worked.
LucasHartmann
 
Posts: 2
Joined: 12. Dec 2012, 17:29
Primary OS: Ubuntu 12.12
VBox Version: PUEL
Guest OSses: WinXP

Re: Create a VMDK that is split into fragments smaller than

Postby mpack » 13. Dec 2012, 12:22

Funny that I seem to be having essentially this same conversation in two different threads. Yes, your hunch is probably ok, though I think technically it may be a deviation from the VMDK spec. It may cause failures if the drive is smaller than 8GB, but above it the BIOS disk CHS addressing maxes out and is effectively switched off.
mpack
Site Moderator
 
Posts: 29307
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP


Return to Suggestions

Who is online

Users browsing this forum: No registered users and 3 guests