Discuss All about VDIs

[TerryE]

What the VBox team could really do with is some simple step-by-step descriptions of how to tidy up and resize a Linux (e.g. Ubuntu) configuration and a similar one for XP (and eventually Vista/Win7, I suppose) including commands to type and screen snapshots in the case of GUI based applications such as Gparted. Though the technical issues are pretty common, the architecture of Linux and Windows are just so different that they demand a completely different approach in detail, especially when explaining to non-specialist readers.

At 20,000ft, you really want to use this type of clone as a death-and-rebirth to clean out all the crap and establish a clean new baseline.

I must admit that if I were you I would go out and buy a USB HDD and use that when in a desktop / development mode. Lots of cheap storage for temporary VDIs etc.

In terms of Windows, the logical registry is actually made up of a number of hive files, some in Windows and two per user (typically called ntuser.dat and UsrClass.dat in the user's Documents and Settings directory hierarchy (let's abbreviate this to DnS for ease of this post). These last two are specific to the user context and their location is read when the user is authenticated (AFAIR in LSASS.exe) from the HKLM\SYSTEM\ControlSet001\Control\hivelist. The "proper" way to relocate DnS is to udpate the Unattend.txt file by reauthoring your media, but I've found that a variant of the KB236621 Article is the easiest way.

For a new user you must do the following from an administrator account:

• Run regedit and go to HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList
• Use Control Panel->User Accounts to create your new user (let's say newUser). Specify a password.
• Now Notepad runAs newUser. (This a right click option). Close notepad.
• If you now go back to regedit and refresh (F5). You will see a new entry for newUser. It's actually keyed by the SID of the new user, but it's easy to recognise as the one with newUser in the ProfileImagePath field. Now edit this and change %SystemDrive% to D:. Close regedit.
• Now in explorer move C:\DnS\newUser to D:\DnS\newUser.
• Now explore the Windows directory, find regedit and runAs newUser. You are now seeing the registry as newUser. Search for \DnS\newUser and make sure these refer to D:\DnS\newUser. (There should only be a dozen or so if you've never logged on interactively. Close regedit.
• If you don't want a password on the account go back into User Accounts and click on Remove the password.
• You can now log on as newUser and your account is homed on the D drive.

The same basic approach also works for an existing account but you will find that there are hundreds of references to D:\DnS\newUser, but there are freeware registry tools which can do such build search and replace exercises (I use a Perl script).

If you google something like XP hivelist ProfileList move account then no doubt you will find better explanations, not that this has anything to do with VBox

Contributing to this sort of forum is easy. All you need to do is to spend 30 years in the industry then get ill and suddenly discover that you can't walk for more than 50 yds without falling over and being knocked out for a week (CFS). When you're stuck in bed 22x7, it's amazing what you find interesting (as long as your brain works). Things like this forum keep me sane.

[baf]

Hi Terry thanks for a nice tutorial on VDI:s.
I'm using it now as a base for a fuse filesystem which will give read-only access to the contents of the disk. Both as one big file and as one file per partition. It will also give some "info" files about the vdi.
One thing I wonder about is why you don't use offsetData in your calculations?
I think this part of Q. So how are virtual bytes in my Virtual HDD mapped into physical bytes in my VDI?:

Hsize = 512 + 4N + (508 + 4N) mod 512

Can be reduced to only:

Code: Select all

offsetData*SectorSize

And that short form feels much safer and easier to understand for me.
I also wonder if that is the problem with your perl-script to defrag a VDI.

Keep up the good work.

[TerryE]

The abstraction interface is defined in include/Vbox/VBoxHDD-new.h. This inteface is implemented by the disk types (VDI, VHD, VMDK) in the directory src/VBox/Devices/Storage and in this directory VDICore.h lays out the header definitions for the VDI file structure which I summarised in the article. You are correct in that my formula explains how the offsets are calculated, but that this calculation is done once at VDI creation and then some further intermediates are calculated when the VDI is opened only the fields are used in computing the values in the data access routines.

So looking at the write operation, for example, the abstraction method pfnWrite is implemented by the VDI routine vdiWrite. The relevant lines (ignoring the special cases where a write fault occurs that requires the allocation of a new image block or the write spans a block boundary) are:

Code: Select all

uBlock   = (unsigned)(uOffset >> pImage->uShiftOffset2Index);
offWrite = (unsigned)uOffset & pImage->uBlockMask;
uint64_t u64Offset = (uint64_t)pImage->paBlocks[uBlock] * pImage->cbTotalBlockData
                   + (pImage->offStartData + pImage->offStartBlockData + offWrite);
rc       = RTFileWriteAt(pImage->File, u64Offset, pvBuf, cbToWrite, NULL);

where lines 1 and 2 split the start into a image block number and offset within block. Lines 3-4 compute the absolute position in the files and 5 does the write. Now in the offset computation the paBlocks[uBlock] * cbTotalBlockDataoffWrite + offWrite are the terms for the mapped block start + offset. The offStartData + offStartBlockData term is the offset to the start of the data; these are computed from the VDI header. For current (Type 1 VDIs) offStartData and and offStartBlockData are the same as offsetData and Block Extra Data in my header definition, and since the latter is zero for Version 1.1 VDIs, this all just collapses to offsetData (no SectorSize multiplier), which is what I use in the current verstion of that script.

One thing that I have thought about doing is setting the track size to a nice round multiple of the underlying cluster size (say 56 sectors rather than 63). That way my alignment trick actually works for all partitions.

[TerryE]

This is just a placeholder to myself that Windows VM users might prefer a freeware tool that is targeted at the Windows market rather than a live CD. Here is one referenced example: Partition Magic alternative = Partition Logic.

[jerome.hode]

Hello everyone,

Perhaps this could be of some interest for some readers:
Here is a small piece of C code to compress and reorganize dynamic VDI files in-place, that I was able to write using the informations of the tutorial:

• PackVDI

It aims to be a workaround for the "Shrink hard disk operation is not implemented!" problem (feature?) that I just experienced with "vboxmanage modifyvdi xxx compact" after a upgrade to vbox 2.1.
The code is actually written for win32, but should be easely ported to other OSes.
Type "PackVDI" for a short description of the command-line options, and/or look at the source (which is not nice, I know.. )

Please note that although I successfully packed some VDI of mines from 24GB to 13GB, I didnt test thoroughly, so as usual, USE AT YOUR OWN RISKS!!

Reminder: best compression is obtained by first defrag/pack+zerofree the VDI from the guest OS (using, for example and for a win32 guest, JkDefrag+sdelete), as every reader of this thread surely knows...

[carlos]

TerryE

In the site http://www.techthrob.com/tech/securedelete.php I found the following statement:

"Filesystems Starting with ext3, Linux filesystems overwrite files with zeros when you delete them, rather than just marking the file as "free space." Previous Linux filesystems, as well as the Windows filesystems (NTFS, FAT) use the latter method, which is why it is so easy to recover deleted files with software tools freely downloadable on the Internet. This is something to consider when deciding how much time you should spend wiping your drive."

If this is true, for Linux ext3 partitions zero free-space fill tools are not needed since the OS is writing zeroes after any file is deleted.

The question: How true is this statement?

Thanks

[TerryE]

This isn't consistent with my observation that compressing a Ext3 based VDI with and without running zerofree gives very different results. I had a quick poke around the Internet and came up with this Linux ext3 FAQ

Q: How can I recover (undelete) deleted files from my ext3 partition?
Actually, you can't! This is what one of the developers, Andreas Dilger, said about it:

• In order to ensure that ext3 can safely resume an unlink after a crash, it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.
  
  Your only hope is to "grep" for parts of your files that have been deleted and hope for the best.

Zeroing out the inode data is quick and efffectively renders the content inaccessible. However is leaves the dirty data blocks intact. If this statement was true then deleting a big file (like an Ext3 based VDI) would take a long time. It doesn't. I think the statement is wrong.

[reto2k9]

After I tried out "PackVDI.exe" on an ext3 formatted virtual disk the system was not able to boot anymore. Well, that was probably my fault and I understood the "Use it at your own risk" clause ;_) Maybe Sun had a good reason why they removed their built-in compact command as of release 2.1.4 ;_)

Another hint how I'm currently addressing the problem of fast blowing up VDIs:

I'm attaching another virtual 2 GB hard disk drive as a primary slave that can be easily mounted on the Linux guest system. The related mount point is dedicated for dynamic data (e.g. Database data or Scripts). So I have a base system and a pretty small "Update Disk" I can distribute.

--reto

[TerryE]

Reto, as I've posted elsewhere, the easiest workaround to packing a VDI is to clone it.

It's nice to see that you seem to have reached the same conclusion as me for Linux appliances. I spilt mine into an immutable system image which has partitions for /, /tmp and swap, with a separate VDI with a single partition for /var with /home symlinked to /var/home. For my current Ubuntu JeOS LAMP (which include Postgessql, Mysql, Php, Perl and Python support) the 7zipped system image is 90Mb unpacking to 550Mb, and the 7zipped blank var image is less than 10Mb.

I've set the cylinder size on both to 2Mb, so guest file clusters are aligned to host clusters, and I have a couple of /etc/init.d script one at 01 which executes a /var based script if it exits, so that I can personalise the immutable system image at start up and a guestproperty enumerator which runs at 99 (which I've discussed in another post). This latter allows me to drive the guest from the host though guestproperty settings. This enables me to provision and personalise a new guest VM from a single host script command in a few minutes. I might write this all up as a howto when I have a spare mo.

If you really want small VDIs, then compacting (even if you do the dd if=/dev/zero of=/dev/sdaN bs=4096 trick or use zerofree) is not the best way, because you have non-zero inodes all over the disk. Yes, ext3 zeroes the cluster pointers on freed nodes, but it does clear down entirely this meta data. As a result, by the end of the initial LAMP build process, my system VDI is around 1,500Mb. Doing zefofree gets this down to just over a Gb, but doing a cp -a of the contents of the image to a blank gives you a VDI which is only 550Mb.

[TerryE]

Could you please update your post: Tutorial: All about VDIs:

the CreateVDI function adds padding after the map to ensure that the first image block begins on a 512byte sector boundary??

In previous versions it was aligned to 512 bytes sectors, but now it's aligned to 4096 byte sectors.

Leolo, Thanks for this prompt. I must that since I've given up on using Windows, my need for working with Vbox has really dropped off. I still use it to run my LAMP VMs through. But you are correct to point out that the dev team have made a lot of changes since version 2.x when I first wrote this. It needs updating, but to do this I need to go through the (programmers documentation -- this is open-source after all) source code. I'll download the latest version of the source and have a browse, then update this

[mpickell]

TerryE,

How do you edit the header files of the VDI? I saw in the tutorial that you said "it is not for the faint of heart" but then i cannot find how to do it. i'm fine with digging into it, I need to fix a snapshot issue.

I had a snapshot merge that looks like it failed... and i may be missing an actual snapshot file from my chain (which i supposed is fatal), but i would like to check it out in the header and see if there is something i can do.

[mpack]

I doubt there is anything you can do, but the only VDI-header-editing tool I can think of is a hex editor, such as frHed. As Terry said, that option would not be for the faint of heart - you would have to understand in detail how VDI snapshot chains work.

[Anunes]

I am trying to "compact" a VDI file using the command "VBoxManage modifyvdi xxxx.vdi compact ", but I get "Syntax error : Invalid Command" error.
Instead of "modifyvdi" I also have tried "modifyhd" but same error.
Any idea of what I am doing wrong?

[mpack]

This isn't the VBoxManage support topic. It would be better if you started a new topic and gave precise details, including the exact command, your host OS, and the exact error message. But, I can tell you offhand that the command is "modifyhd <filename> --compact". Note the dashes.

[Anunes]

This isn't the VBoxManage support topic. It would be better if you started a new topic and gave precise details, including the exact command, your host OS, and the exact error message. But, I can tell you offhand that the command is "modifyhd <filename> --compact". Note the dashes.

Sorry for posting here.
I already open a new topic for this issue.

Any way the vboxmanage command example found hier viewtopic.php?f=35&t=8046 should be updated.
Thank you.