Discuss All about VDIs

This is for discussing general topics about how to use VirtualBox.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Discuss All about VDIs

Post by TerryE »

This topic is to provide a discussion and feedback for the Tutorial: All about VDIs currently covers the following questions:
  • How are VDI files structured?
  • What's in the Header Descriptor?
  • So how are virtual bytes in my Virtual HDD mapped into physical bytes in my VDI?
  • So what about snapshots. How do they work?
  • So what are Immutable and Writethrough drives? How do they work?
  • Why do dynamic VDIs grow far more than the used space?
  • How can I reduce the size of a dynamic VDI on disk?
  • How do I get the best performance out of my VDIs?
  • Where is the best place to keep my VDIs?
  • How can I resize the partitions inside my VDI?
  • How do I backup my VDIs?
The topic is locked so only the moderators can update it, but by all means comment below. Tell me where I am wrong. Suggest new Qs that need to be added.

If you want to discuss the issue of tutorials in general; whether you think this format works and which other ones cry out to be written see A discussion on New Tutorials
ghr
Volunteer
Posts: 376
Joined: 25. May 2007, 22:46
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: DOS, Win3x, Win95, WinXP, Ubuntu, OS/2

Post by ghr »

TerryE: my compliments !!!
chronoboy
Posts: 89
Joined: 21. Jul 2008, 07:11
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Linux, Windows
Location: Canada

A great tutorial, we need more of them.

Post by chronoboy »

We need more of this type of content on these forums, or even a dedicated site for them.

A small question, if you change the type of VDI from normal to writethrough, does this cause any complications within the actual VDI format and image data?

When the VDI is created from scratch in either normal or writethrough, are the internals and how data is stored different?

Is there a way to convert a flat file disk image into a VDI and vise versa? A flat file in this case would be dd dumped hard disc.

Are there plans for VirtualBox to support gzipped or even ZIPed immutable VDIs?

Do VDIs contain a checksum of the data image? If not, is there plans to add this in the future? The checksum would only generate when powering off the VM or saving a state.

Are there plans for sparse VDI images which take full advantage of ext3 and NTFS sparse files?

Are there any plans for a tool which makes VDI splitting easy and simple? VMWare has something similar, which can split vmdks into 2GB chunks. A splitting tool would make it easier to back-up VDIs on fat32 partitions or external fat32 drives.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Re: A great tutorial, we need more of them.

Post by TerryE »

I will left these thoughts stew and update the Tutorial in a few weeks. In the meantime I will answer your specific Qs
chronoboy wrote:A small question, if you change the type of VDI from normal to writethrough, does this cause any complications within the actual VDI format and image data? When the VDI is created from scratch in either normal or writethrough, are the internals and how data is stored different?
Clearly to change the type you need to unregister then reregister the VDI. This means that you have first to discard / merge in your snapshots. The writethrough has a different value in the type field in the header, that's all. However once flagged as writethrough, the VDI are treated differently w.r.t. snapshotting as described in the tutorial.
chronoboy wrote:Is there a way to convert a flat file disk image into a VDI and vise versa? A flat file in this case would be dd dumped hard disc.
You can use the createrawvmdk command as described in the manual to map raw partitions. Booting from a clonezilla liveCD allows you to do partition copies: raw->VDI, VDI->raw, VMDK->VDI, etc.
chronoboy wrote:Are there plans for VirtualBox to support gzipped or even ZIPed immutable VDIs?
I don't think so, but writing this driver is less than a week's development. Maybe I'll get around to it one day. How are your C++ skills? :wink:
chronoboy wrote:Do VDIs contain a checksum of the data image? If not, is there plans to add this in the future? The checksum would only generate when powering off the VM or saving a state.
No. This a "render until Caesar" sort of issue. Decent file systems contain a level of jounalling. The H/W embeds CRCs. Adding this at this layer would have a major performance impact for little real functional benefit.
chronoboy wrote:Are there plans for sparse VDI images which take full advantage of ext3 and NTFS sparse files?
I am not part of the VBox project team, just a supporter like yourself so I am guessing, but I doubt it. The VDI page algo gives a level of sparse mapping that is portable across file systems. In practice the NTFS and ext3 sparse formats add little and the implementation is file system specific.
chronoboy wrote:Are there any plans for a tool which makes VDI splitting easy and simple? VMWare has something similar, which can split vmdks into 2GB chunks. A splitting tool would make it easier to back-up VDIs on fat32 partitions or external fat32 drives.
Most of the zip utilities include an option to split the compressed file into 2Gb chunks, so this one isn't worth it, but writing a standalone disk utility for format conversion, etc. is.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
chronoboy
Posts: 89
Joined: 21. Jul 2008, 07:11
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Linux, Windows
Location: Canada

Thank you for the info

Post by chronoboy »

Here is an interesting question:

What happens if we defrag and use zero out a snapshot?

Can we run compact on a snapshot?
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

I've just added some more examples to help understandability. I've also reproofed the article and cleaned up hopefully all but realistically most of the grammar and type issues.

Now to your Q. (1) If you haven't run zerofree before the snapshot, then running zerofree on a snapshotted drive can increase the snapshot, because zerofree will attempt to zero out any free space. If this is in a block in the underyling VDI, then is will actually create a new copy in the snapshot with these free-space blocks zeroed out.

You can run compact against a snapshotted drive but all compact does is to compact the base VDI in the VDI directory. It doesn't touch the deltas.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
chronoboy
Posts: 89
Joined: 21. Jul 2008, 07:11
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Linux, Windows
Location: Canada

Post by chronoboy »

Then, in other words, it would be best practice to defrag and zero out the guest before snapshotting. This would then make the snapshots smaller in the end?

Here is an issue I bumped into with a XP guest, it happens on other guest OSes as well:

After I snaphot the guest, the delta VDIs seems to grow very very fast. 200M after the first XP boot in the snapshot. I could not imagine XP using up this much space during a boot. At this point, with the information gather here, this would XP freeing up space, or even deleting and re-creating the swap file. I will be trying this step again after I finish up the compact process.

I took your idea on defraging, zeroing out, and compacting before burning to a disc. So far, the results are quite good. A much smaller VDI.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Remember that pagefile.sys and hiber.sys take up a lot of memory, and since these are allocated, sdelete won't touch these. Turn off hibernate, and in fact why have it when it is easier to pause your VM? Turn off your pagefile.sys and reboot into safemode; clean out all of the $NT crap dirs from \windows; run sdelete again; then set a fixed pagefile size and shutdown your XP system. Snapshot this and you should have just about the tightest XP config.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
jerome.hode
Posts: 4
Joined: 29. Jul 2008, 10:33

Post by jerome.hode »

my 2 cents:
- the image type is 1 for dynamic , 2 for fixed
- the sector size is 512 (typo in tutorial, 0x0200 != 8192)
- Fposition = DataOffset + ( Fblock * BlockSize ) + ZOffset (typo in tutorial)
- the "garbage/unused" footer is not so unused: from my experiments, if you dont have 0x00000200 at offset 0x01D4, VBox rejects the VDI.
- the data offset is very probably aligned to the next sector (from the "512+4*N" value)

by the way, you can forge yourself a VDI file with the data offset rounded to the next 4K page, if you like. It should be accepted and properly processed by vbox.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

jerome wrote: - the image type is 1 for dynamic , 2 for fixed
- the sector size is 512 (typo in tutorial, 0x0200 != 8192)
- Fposition = DataOffset + ( Fblock * BlockSize ) + ZOffset (typo in tutorial)
- the "garbage/unused" footer is not so unused: from my experiments, if you dont have 0x00000200 at offset 0x01D4, VBox rejects the VDI.
Thanks. I'll make these changes tomorrow.
jerome wrote:- the data offset is very probably aligned to the next sector (from the "512+4*N" value)
Yes, the VDI driver uses the DataOffset. It even includes the suggestion that the offset should be block aligned. And s**t, s**t, b****cks I rechecked the VboxManage utility with a few test cases a few moments ago and the 1.6.2 version does block align. I just didn't spot this on my walk through the 1.6.0 code, though I did suggest to Klaus E that he make this change since it was only 2 lines. Maybe he slipped the change into 1.6.2 or maybe I'm just brain-dead and it was there all along but I missed it. :lol:

Whatever, this detail is history and needs stripped out of the guide. Thanks for your review and comments.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
chronoboy
Posts: 89
Joined: 21. Jul 2008, 07:11
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Linux, Windows
Location: Canada

Post by chronoboy »

Shouldn't this most interesting option be listed in your tutorial

Code: Select all

VBoxManage internalcommands setvdiuuid <VIDFile>
It is suppose to allow the user to change the UUID of the VDI, for example if you use cp instead of CloneVDI, or if you use ZFS to manage clones.

Sun should definitely add some sort of options for ZFS users to safely change the UUID when cloning a VDI. This command has a somewhat scary warning in the help:
WARNING: This is a development tool and shall only be used to analyse problems. It is completely unsupported and will change in incompatible ways without warning.
Other than this tool, changing the UUID manually is not so user friendly:

Code: Select all

dd if=/dev/random of=<VDIFile> bs=1 count=6 seek=402 conv=notrunc
Besides, I can see the location of the UUID changing without warning.
jerome.hode
Posts: 4
Joined: 29. Jul 2008, 10:33

MakeSparseVDI

Post by jerome.hode »

The Windows users could perhaps have some interest in a small piece of code I wrote for my own fun to convert a VDI file into a fixed one backed by a NTFS sparse file:One advantage of a fixed sparse VDI over a dynamic one is that the result contains a flat volume image, still retaining the space advantage of a dynamic one. So you can possibly use the VDI file with tools that work with flat images, for example mounting the VDI file in the host with the (excellent) imdisk tool.

MakeSparseVDI can:
  • create a empty fixed sparse VDI file.
  • copy a VDI to a fixed sparse one
  • convert (inplace) a VDI into a fixed sparse one
  • display some info from the VDI header and partion table (useful for mounting with imdisk)
  • set a (random) new uuid
(Type "MakeSparseVDI" for a short usage description, and/or look at the source)

Additionally, I have also:
  • MakeSparseFile : create a NTFS sparse file, copy a (possibly sparse) file to a sparse one, or update a file to a sparse one (recovering the zero blocks).
  • zerofreeXP : writes zero in the free, non-zero sectors of a NTFS volume (requires at least XP, see source)
a typical usage (on a XP host) would be:
  • copy or convert the VDI with MakeSparseVDI
  • mount the sparse VDI with imdisk, and run zerofreeXP on the mounted image (supposing it's a NTFS filesystem)
  • unmount and run "MakeSparseFile update ..." to recover all the zeroed space
Please note that all this is in a early (dirty) stage (although I successfully converted some 60GB/6GB dynamic VDI files)

Have fun / use at YOUR OWN RISK... (especially "MakeSparseVDI convert" has the theorical capability of destroying your VDI file).

(feel free to move or delete this post if it is irrelevant)
Last edited by jerome.hode on 7. Aug 2008, 17:21, edited 1 time in total.
chronoboy
Posts: 89
Joined: 21. Jul 2008, 07:11
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Linux, Windows
Location: Canada

Post by chronoboy »

You do have an interesting idea with using sparse files. I am not exactly sure why dynamic VDI are not sparse files in the first place. Most modern file systems support sparse files, I guess using dynamic VDIs does have some advantages. For example, using it on file systems which do not support sparse files, such as FAT32. Also burning a sparse file to a Disc will not work as intended, same goes if you try copying a sparse file to a file system without sparse file support.

Have you done a benchmark on Sparse files -vs- Dynamic VDI? If sparse files show a high enough impact, perhaps Sun should consider adding support for creating sparse VDI images within VBox itself. And hey, the code is already made, so Sun can barely complain.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Sparse files could be a mixed blessing and curse. Some of the up-sides are:
  • Since the sparse block size under typical NTFS for example is 64K and the image block on dynamic VDIs is 1Mb, then using sparse file format will achieve higher compression levels as more non-blank areas will be eliminated.
  • The handling of sparse files is built into the OS / file-system and is thus relatively efficient.
Possible disadvantages of a sparse file approach could include:
  • The implementation of sparse files is OS specific. This means that you cannot move sparse VDIs around systems without their "exploding" back to full size.
  • The sparse attribute is not sticky on copy so cloning (or even copying) a sparse VDI will result in it exploding to full size.
  • A properly groomed dynamic VDI (e.g. one that has been copied using Gparted, then defragged using the SysInternals Contig tool) will actually run faster than a static VDI. The sparse allocation algorithm is essentially a framgmenting one so that sparse files will be noticeably slower than than static VDIs.
  • many compression / backup utilities are not sparse file aware and will logically read the entire file, again resulting in worse performance than dynamic VDIs
Clearly we need to do more experimentation here. And the other thing that we shouldn't discount is the use of straight compressed (at a filesystem level) VDI images, especially compression of base VDIs that we are going to snap against.

BTW jerome, please don't take this as negative response. I think what you are doing here is fantastic. The user community should bounce around such ideas and debate them robustly. Some very good improvement to VB could be built out of such debate.

PS. I tidied up your BBcode. It's a bit of a dog doing this but it hellps on readability. If you object then just shout :-)
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

chronoboy, you are right about my adding the setvdiuuid command. The disclaimer also amuses me because the internalcommands are no less a "development tool" than the rest of VBox and no less supported. Of course its use is only safe on unregistered VDIs without snapshots. Whilst the concept of "sparse files" is implemented in many FS, as I said the details are different and most utilities are not sparse file aware. The 1Mbyte image block size is a pragmatic compromise allowing you to optimise out the major blocks of whitespace without compromising performance too much.

Out interest I did a compact on a 1920Mb dynamic VDI that is the root partition for one of my Ubuntu JeOS LAMP systems. Note that this is fairly static since /var and /home are mounted on another VDI:

Code: Select all

D:\VirtualBox\VDI>compact /c ub804jeosSys.vdi

 Compressing files in D:\VirtualBox\VDI\

ub804jeosSys.vdi    685776896 : 354590720 = 1.9 to 1 [OK]

1 files within 1 directories were compressed.
685,776,896 total bytes of data are stored in 354,590,720 bytes.
The compression ratio is 1.9 to 1.

D:\VirtualBox\VDI>Contig ub804jeosSys.vdi

Contig v1.54 - Makes files contiguous
Copyright (C) 1998-2007 Mark Russinovich
Sysinternals - www.sysinternals.com

Summary:
     Number of files processed   : 1
     Number of files defragmented: 1
     Average fragmentation before: 8605 frags/file
     Average fragmentation after : 1 frags/file

D:\VirtualBox\VDI>compact /q ub804jeosSys.vdi

 Listing D:\VirtualBox\VDI\
 New files added to this directory will not be compressed.

Of 1 files within 1 directories
1 are compressed and 0 are not compressed.
685,776,896 total bytes of data are stored in 354,590,720 bytes.
The compression ratio is 1.9 to 1.
Note how fragmented the compressed image became before being defragged. It will be interesting to see if this is noticeably any slower.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
Post Reply