Alignment of blocks inside VDI file

[skoehler]

Hi,

what's the default alignment of the data inside VDI files?
I'm asking, since the alignment of the partitions on the virtual harddisk and the alignment inside the VDI file should somehow interact in such a way, such that 4k blocks inside the VM are mapped to 4k blocks inside the VDI file. This should ensure maximal performance.

Googling for this sort of stuff, somebody claimed that he had written a tool for aligning the data inside the VDI file such that sector 63 starts at a 4kB boundary. Which is probably a bad idea, since the second and third partitions may be badly aligned after using that tool.

VBoxManage showhdinfo didn't tell me anything about the internal alignment of the VDI file.

[mpack]

what's the default alignment of the data inside VDI files?

The drive image must be aligned on a 512byte boundary (1 sector), at minimum. That used to be the practice as well, however I think recent VBox versions now align the internal image of new drives on a 4K boundary - but it's been a while since I checked that. Of course neither alignment of the drive image ensures that the boot partition is optimally aligned. It's impossible to guarantee the perfect alignment at drive creation time, the partitions don't exist yet!

Googling for this sort of stuff, somebody claimed that he had written a tool for aligning the data inside the VDI file such that sector 63 starts at a 4kB boundary. Which is probably a bad idea, since the second and third partitions may be badly aligned after using that tool.

That "somebody" was me, see sticky in "Windows Hosts" for the CloneVDI tool download, and I'm open to better ideas if you have any.

Aligning sector 63 of an XP drive on a 4K boundary will typically place the boot A.K.A. OS partition on a 4K alignment, which is optimal for application loads, page swaps etc, i.e. most things that really affect performance. Also, XP drives typically only have one partition. If the virtual drive contains more than one partition then it's impossible to align them all on 4K boundaries unless the partitions happen to be spaced that way within the drive image - which is beyond the control of my tool, or indeed any tool which only tweaks the VDI header and not the image. In fact partitions on an XP drive will typically not be aligned on 4K boundaries, they will be aligned on cylinder boundaries, usually assumed to be 63 sectors for legacy reasons.

At the time I wrote the message which you referenced, the vast majority of VirtualBox VMs were, AFAIK, XP and Linux. The 63 sector alignment idea works well for those. True that the Linux swap partition remains randomly aligned, but that makes it no worse off than before. However Vista and Win7 VMs have become more popular since then, and I know those do use different default partition alignments, though I don't have the details to hand. From memory I think it's 128 sectors to the first partition, I don't recall how further partitions were aligned. I note that VirtualBox's new 4K image alignment should work well for Vista/Win7 if this is true. I'm not really a Linux user, so I couldn't tell you what alignments it typically uses: AFAIK the default was and still is 63 cylinders, per the PC legacy.

[skoehler]

what's the default alignment of the data inside VDI files?

The drive image must be aligned on a 512byte boundary (1 sector), at minimum.

I hope that they are more clever than that
For the asynchronous I/O there are using, it's crucial that the I/O requests (especially writes) are aligned to full clusters of the host filesystem, which would be 4k for most filesystems.

That used to be the practice as well, however I think recent VBox versions now align the internal image of new drives on a 4K boundary - but it's been a while since I checked that. Of course neither alignment of the drive image ensures that the boot partition is optimally aligned. It's impossible to guarantee the perfect alignment at drive creation time, the partitions don't exist yet!

Well, let's assume a reasonable installer (like the one of Windows 7 or recent Linux distributions) which align partitions to 1Megabyte boundaries.
Also, Windows XP can be installed in a "proper" way such that the partition starts at sector 2048 instead of 63.

However, when you create a virtual machine, you actually chose the operating system. So virtualbox knows about the OS when creating the virtual harddisk - well, if you use the wizard - and they could have chosen to align the VDI data in different way. Be that as it may be, I'd like a tool to verify the alignment of the data within a VDI file, to avoid any surprises.

Googling for this sort of stuff, somebody claimed that he had written a tool for aligning the data inside the VDI file such that sector 63 starts at a 4kB boundary. Which is probably a bad idea, since the second and third partitions may be badly aligned after using that tool.

That "somebody" was me, see sticky in "Windows Hosts" for the CloneVDI tool download, and I'm open to better ideas if you have any.

Aligning sector 63 of an XP drive on a 4K boundary will typically place the boot A.K.A. OS partition on a 4K alignment, which is optimal for application loads, page swaps etc, i.e. most things that really affect performance. Also, XP drives typically only have one partition. If the virtual drive contains more than one partition then it's impossible to align them all on 4K boundaries unless the partitions happen to be spaced that way within the drive image - which is beyond the control of my tool, or indeed any tool which only tweaks the VDI header and not the image. In fact partitions on an XP drive will typically not be aligned on 4K boundaries, they will be aligned on cylinder boundaries, usually assumed to be 63 sectors for legacy reasons.

Yes, this is all correct, as long as one keeps using windows xp's partitioning tool.
However, the existence of your tool, the possibility to align the data within the VDI file in some almost arbitrary way got me a bit worried, since there is no tool to tell which alignment is actually used by a given VDI file. And I need to know that, before I start aligning the partitions within the virtual machine (the gparted guys still suck at providing an UI which makes it easy to do so, unfortunately). In fact I have aligned the partitions in my WinXP machines to 1MB boundaries.

Also, I worry about that tool you wrote. Does it allow you to chose the alignment? Or does it always align sector 63 to a 4k boundary?

At the time I wrote the message which you referenced, the vast majority of VirtualBox VMs were, AFAIK, XP and Linux. The 63 sector alignment idea works well for those. True that the Linux swap partition remains randomly aligned, but that makes it no worse off than before. However Vista and Win7 VMs have become more popular since then, and I know those do use different default partition alignments, though I don't have the details to hand. From memory I think it's 128 sectors to the first partition, I don't recall how further partitions were aligned. I note that VirtualBox's new 4K image alignment should work well for Vista/Win7 if this is true. I'm not really a Linux user, so I couldn't tell you what alignments it typically uses: AFAIK the default was and still is 63 cylinders, per the PC legacy.

Recent versions of Linux tools (like fdisk, gparted, but not cfdisk I believe) and Windows 7 (and maybe Vista) align partitions in such a way, that partitions start at a sector number which is a multiple of 2048 (1MB alignment).
OSX seems to align partitions to multiple of 40 (20KB alignment, not chosing a power of two seems really dumb to me).

[mpack]

We seem to be talking at cross purposes at several points: CloneVDI is a cloning tool, not a partitioning tool. The partitions all already exist before CloneVDI ever sees a drive, and their alignments are what they are. Obviously VirtualBox can't control partition alignment either because there are no partitions when the drive is created. CloneVDI does in fact have a function which could slide partitions around, but with such a variety of partitioning schemes and guest OS out there, there is always the potential that being too fancy will corrupt the drive. I have to take that into account and not assume that everyone these days uses Win7.

It's been a while, I can't remember if the assumption that the first partition is at LBA 63 is still in there. I do remember planning to read the partition map to get the actual alignment, but I can't remember if I did it.... (Later) Just checked the code, and 63 is still in there. Yeah, maybe I should fix that.

If you are a programmer and/or know how to use a hex editor then finding out the current alignment of a VDI image is quite easy. I don't know of any user tools that report that info tho. Possibly I could add this to the "Show Header" info shown by CloneVDI.

[skoehler]

We seem to be talking at cross purposes at several points: CloneVDI is a cloning tool, not a partitioning tool.

No I'm fully aware of what CloneVDI is.

It's been a while, I can't remember if the assumption that the first partition is at LBA 63 is still in there. I do remember planning to read the partition map to get the actual alignment, but I can't remember if I did it.... (Later) Just checked the code, and 63 is still in there. Yeah, maybe I should fix that.

Certainly it would be nice if CloneVDI would be able to read the partition table of the virtual drive. However, it would be much easier and IMHO it is also mandatory to allow the user to chose whether CloneVDI changes the alignment, and if so by how much.

If you are a programmer and/or know how to use a hex editor then finding out the current alignment of a VDI image is quite easy. I don't know of any user tools that report that info tho. Possibly I could add this to the "Show Header" info shown by CloneVDI.

I am a programmer. But I have no documentation of the VDI file format at hand.
Actually, I was hoping for some VirtualBox developer to add this kind of information to the the output of "showhdinfo".

[mpack]

I have no documentation of the VDI file format at hand.

TerryE's All About VDIs tutorial is a good place to start. After that, the VirtualBox source code is the definitive structure reference, there being no other published spec.

I do not (and will not) allow CloneVDI users to select an image alignment - I do not believe in complicating a user interface with obscure features.

[skoehler]

I have no documentation of the VDI file format at hand.

TerryE's All About VDIs tutorial is a good place to start. After that, the VirtualBox source code is the definitive structure reference, there being no other published spec.

Thanks for the link.

I do not (and will not) allow CloneVDI users to select an image alignment - I do not believe in complicating a user interface with obscure features.

A tool, which in some cases does the wrong thing by default, does indeed not have any obscure features. However, that tool is a nightmare for those who know that the tool is wrong.

[mpack]

However, that tool is a nightmare for those who know that the tool is wrong.

Those who are unhappy with what a tool does are free to choose a different tool, or make their own.

[skoehler]

However, that tool is a nightmare for those who know that the tool is wrong.

Those who are unhappy with what a tool does are free to choose a different tool, or make their own.

True. The source of your tool is closed?

[mpack]

True. The source of your tool is closed?

Open. The zipped source download can be found in the second message of the sticky. CloneVDI is a vanilla Win32 project, and can be built with Visual C++ Express (2008 edition is the last one I tested).

[ChipMcK]

two cents:
Paragon used to have an alignment tool for XP systems - used it for my vms.
Western Digital is rumored to have an alignment tool for XP that is based on Acronis.

Best

[mpack]

Paragon used to have an alignment tool for XP systems - used it for my vms.

They still do I think. I happened to visit an HP Compaq support site yesterday, the Paragon tool was available as a download. It's a disk tool however, not a VM tool, meaning it will work by physically copying a lot of sectors, not just a VDI header tweak. The main purpose of the tool is to align partitions on 4K boundaries for the benefit of hdds whose sector size is >512 bytes.

Western Digital is rumored to have an alignment tool for XP that is based on Acronis.

If true, then best guess is that it uses the VHD format imaging feature of more recent Acronis versions (why not just use Win7 backup?), the VHD image can then be manipulated offline and later "restored". Has the advantage that everything can be done live, while still running the Windows OS being modified.

[mpack]

It's been a while, I can't remember if the assumption that the first partition is at LBA 63 is still in there. I do remember planning to read the partition map to get the actual alignment, but I can't remember if I did it.... (Later) Just checked the code, and 63 is still in there. Yeah, maybe I should fix that.

Just a quick correction here. I had another look at the code and discovered that the current version of CloneVDI (v2.05 as I write this) does indeed align the first partition on 4K boundary, just as I vaguely remembered it did. Yes, the 63 assumption is still in there, but only as a default to be used if CloneVDI is unable to find an identifiable partition map. At the time of writing only MBR style partition maps are supported.

So, the only thing I would do differently now is preferentially align the largest partition instead of the first partition. I say this because I've seen that Vista and later, and maybe Ubuntu as well, may create a smaller boot partition prior to the larger main partition. Obviously it's the latter which should have priority for optimization.

[Leolo]

skoehler,

Be careful, the post "Tutorial: All about VDIs" is outdated. And it contains wrong/outdated info. I warned about this some time ago:
viewtopic.php?f=1&t=8048&start=30#p173041

Newer versions of Virtualbox create .VDI files properly aligned for SSDs. You don't have to worry (I researched this because I own an intel SSD myself and wanted to be completely sure

PS: The following post is also outdated. Donut recommended a technique that was useful in its day, but it's no longer needed with new VirtualBox versions:
viewtopic.php?f=1&t=25336

Regards.