Mounting .vdi file on host

Discussions related to using VirtualBox on Linux hosts.
QBikal
Posts: 1
Joined: 13. May 2007, 20:47

Mounting .vdi file on host

Post by QBikal »

I recently had some success mounting a .vdi image within a Linux filesystem, and I thought others might benefit from my experience. First off, I am no expert. I've been tinkering with Linux for a few years now but only about a month ago did I start using it as my primary OS. Just for the record, Vista's price tag and lack of features pushed me to convert. A $300 OS that consumes something like 8GB HD and still can't burn iso files, that unthinkable. Even DSL at 50MB can do that. But that's a rant for a different forum. Currently I'm using openSuse 10.2. I wanted to use a Window$ guest OS for tasks that I haven't yet found easy alternatives for in Linux; like encoding videos for example. Transferring files via a virtual network worked well but speeds were limited and required some configuring. I briefly looked into “shared folders” and it looks like they would have been a great solution but I still wanted a little more control. Whether or not my solution has any great benefits I am still unsure but I figured it can't hurt to post.
First I created a virtual disk and configured the file size to be fixed. I doubt there are any simple ways of doing what I plan to with a dynamic file. Using the VM I formated the drive with one full size FAT32 partition. I also added some dummy files just for testing purposes. The rest is done in the host OS.
Before messing with the .vdi file in any way, make sure the guest OS is shutdown. Not suspended or saved state.
I then opened the vdi file in a gui hex editor that came with my distribution called KHexEdit. My virtual disk was only 300 MB and I have 1GB of ram. Using KHexEdit with multi GB files may not work so well. An alternative may be using the console command hexdump with appropriate options and piping output into more. Or you could dd the first 1 MB of your vdi file to another file and just open that. It helps to get an ascii view. The vdi file starts with “0000000000 <<< InnoTek VirtualBox Disk Image >>>...........................”
I then located the start of the FAT32 partition. Being the only partition on the drive, it's beginning was near the start of the file.

Code: Select all

0000034176 ................................................................
0000034240 ................................................................
0000034304 ëX.MSDOS5.0..."......ø..?.ÿ.?...gP..S...........................
0000034368 ..)2úÏøNO NAME    FAT32   3É.Ѽô{.Á.Ù½.|.N..V@´.Í.s.¹ÿÿ.ñf.¶Æ@f.
0000034432 ¶Ñ.â?÷â.ÍÀí.Af.·Éf÷áf.Fø.~..u8.~*.w2f.F.f.À.»..¹..è+.éH. ú}´}.ð¬
0000034496 .Àt.<ÿt.´.»..Í.ëî û}ëå ù}ëà.Í.Í.f`f;Fø..J.fj.fP.Sfh.....~.... .´
0000034560 A»ªU.V@Í......ûUª....öÁ.....þF.´B.V@.ôÍ.°ùfXfXfXfXë*f3Òf.·N.f÷ñþ
0000034624 Â.Êf.ÐfÁê.÷v..Ö.V@.èÀä..̸..Í.fa..Tÿ.Ã..f@I..qÿÃNTLDR      .....
0000034688 ..............................................Remove disks or ot
0000034752 her media.ÿ..Disk errorÿ..Press any key to restart.......¬ËØ..Uª
0000034816 RRaA............................................................
0000034880 ................................................................
0000034944 ................................................................
The start is somewhat obvious; or at least it is to me. The “MSDOS” is a big hint. If you are having trouble finding it, here is a tip. Label the partition with a unique name. The volume's name is stored near the very beginning of the filesystem so by searching for its first occurrence, you will find yourself within a few lines of the beginning. I don't recall labeling this volume anything which would explain the “NO NAME”. There also seems to be a lot of blank spaces in the beginning of the file which aids greatly when searching by eye. By the looks of this output, I would say the FAT32 partition starts with an offset of 34304. Note: that is in decimal, not hexadecimal. By default KHexEdit displays addresses in hex, and so I had to change that. I have done this process twice now and the addresses where not the same so I doubt my number will work for you. Both where in the 30000's though. Now we have the information we need.
To mount, I first created a directory /mnt/vdifs. All that's left is the mount command. After much trial and error I finally had success with:

Code: Select all

sudo mount -o loop,offset=34304,umask=000 /foo/bar.vdi /mnt/vdifs
Because only root has mounting privileges on my system I used sudo. For those familiar with mount it should be obvious that /foo/bar.vdi is the image I want to mount and /mnt/vdifs is where I want to mount it. The magic is in the options. -o is used to signify the start of mounting options. “loop” specifies that the source is a file already mounted within the system. Loop can be used like “loop=/dev/loop1” which would specify which loop device to use, but when entered alone mount decides which one to use. “offset=34304” is where within the file the filesystem starts. I am uncertain whether or not most filesystems specify their length or if it is determined by the partition table alone. If it is specified only by the partition table then there could be trouble if the partition doesn't extend to the very last byte of the drive. Mount may assume the filesystem's length is to the end of the vdi file and if this is not true data may be written out of the partition's boundaries. Even if the partition extends to the last byte of the virtual drive, it may not end on the last byte of the vdi file. I haven't examined the vdi file structure but I have been assuming that all VirtualBox related data is written at the start, leaving only raw hard drive data to be written to the end. This of cause only works with fixed size images because dynamic files will probably not be written in a linear fashion. Despite all the possible hang ups, I have not yet had a problem. One last option deals with file permissions. Because dos filesystems don't implement permissions like Linux filesystems do, the filesystem will be mounted as belonging to root. If regularly users need to be able to write to the drive, adding umask=000 will set all files writable by all users. One more point worth mentioning, if the offset was set correctly, mount should be able to determine the filesystem type by itself. If the offset was incorrect, mount will fail and you will probably receive an error message saying you need to indicate the filesystem type because mount doesn't know what the heck it's looking at. Don't be mistaken and assume that all you need to do is add a -t option.
After all this effort what has been accomplished? Using my custom mount command, I can now mount my vdi file into my Linux filesystem with full read and write capabilities and all operations are performed at full speed. I can even run fsck on the loop device if I know which one was used. Perhaps even use dd to copy a real partition into my vdi image file. Some cautions need to be taken though. Obviously, you should always make backups if the data is important. Also, be sure to unmount before starting your VM and never mount unless the VM has been shutdown or at least unmounted the partition. Just for fun I tried writing to the vdi file while the VM was using it and just like expected, I broke the filesystem. Data was written by the host, but the guest overwrote some the the TOC creating a mess. The files were not visible but the space they occupied was marked used, so the space was unrecoverable without a scandisk or fsck.
Maybe you learned something, maybe you didn't. If this helps you in some way then great, otherwise hopefully you at least enjoyed the read. If nothing else, this does prove plausibility of mounting vdi images in a useful way. With any luck, future versions of VirtualBox will incorporate this feature with only point and click needed to use. Doing this through VirtualBox could also ensure that vdi files are not mounted on the host and guest at the same time.
akageekster
Posts: 5
Joined: 27. May 2007, 07:41

Post by akageekster »

Hi QBikal,

I like you're pioneering efforts so far.........and if you don't mind, I'll try to take it to the next level by manually creating a VDI image from an existing raw image made with dd...... :mrgreen:

Until the next version comes out, I'll have to do it this way to be able to use my existing installations.........I'll start off easy and try it with a Linux os installed on a 1G pendrive, and see what happens........

And I want to thank you for pointing the way to make this possible....... :D

---akageekster
clarksonator
Posts: 1
Joined: 28. May 2007, 18:34

Awesome

Post by clarksonator »

I have been having random crashing on my Windows 2K guest machine whenever I would copy large files over the network to my Ubuntu host. After seeing this post I was able to find the starting offset for my NTFS drive and then I was able to mount the drive. Now my files transfers will be faster and less error prone.

Thanks!!!
mAx
jza
Posts: 50
Joined: 7. Jun 2007, 19:05
Location: Villahermosa, Tabasco
Contact:

Post by jza »

Does this also applies to NTFS type filesystem since I doubt that there might be any MSDOS tagline.

Also could you put the piping options you did to your hexdump?

This is really useful, I wonder why there is not a tutorial about this on the VirtualBox wiki.

I use the recomendation on dd the first mb:

Code: Select all

dd if=MyImage.vdi of=dump bs=1M count=1
This gave me the following hexvalue:

Code: Select all

0000000000 <<< InnoTek VirtualBox Disk Image >>>...........................
0000000064 ..............................................................
0000000128 ................................................................
0000000192 ................................................................
0000000256 ................................................................
0000000320 .........................2..a.......?..............?............
0000000384 ....?...w.??..?L.~?93.(?E?????:C.2...???........................
0000000448 ................................................................
0000000512 ....m...n...o...p...q...r...s...t...u...v...w...x...y...z...{...
I select 320 as the offset (that might have been my mistake) and then plug:

Code: Select all

$ mount -o loop,offset=320,umask=000 PerotSystem.vdi /mnt/winblows/
mount: you must specify the filesystem type
akageekster
Posts: 5
Joined: 27. May 2007, 07:41

NTFS partitions

Post by akageekster »

NTFS partitions start off like so:

Code: Select all

00007e00  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |ëR.NTFS    .....|
:)
---akageekster
stranger10
Posts: 3
Joined: 19. Jul 2007, 18:47

Post by stranger10 »

Has anyone successfully mounted an ext2 filesystem in this way?

I have had no problem determining the offset for FAT formatted partitions but had no success with an ext2 partition.

How do you determine the offset for ext2 partitions?

Thanks
stranger10
Posts: 3
Joined: 19. Jul 2007, 18:47

Post by stranger10 »

Still no help on this? Anyone?
Ingo
Volunteer
Posts: 731
Joined: 22. Aug 2007, 10:13
Location: Germany

Re: NTFS partitions

Post by Ingo »

akageekster wrote:NTFS partitions start off like so:

Code: Select all

00007e00  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |ëR.NTFS    .....|
Thank you for the useful hint. I do it now automagically the ***x-way (on bash):

Code: Select all

sudo mount -t ntfs-3g -o loop,\
uid=user,gid=group,umask=0007,fmask=0117,offset=\
0x$(hd -n 1000000 ~/.VirtualBox/VDI/image.vdi | \
grep "eb 52 90 4e 54 46 53" | cut -c 1-8) \
.VirtualBox/VDI/image.vdi ~/mnt/
afanofosc
Posts: 4
Joined: 12. Dec 2007, 04:42

Post by afanofosc »

I think that the offset in a .vdi file to the beginning of the file system can be found at offset 456. The files start with a pre header that has this structure (forgive my Object Pascal syntax):

VDIPREHEADER = record
szFileInfo : array[0..63] of Char;
u32Signature : cardinal;
u32Version : cardinal;
end;

Following this structure is the header of type 1:

VDIHEADER1 = record
cbHeader : cardinal;
u32Type : cardinal;
fFlags : cardinal;
szComment : array[0..255] of Char;
offBlocks : cardinal;
offData : cardinal;
Geometry : VDIDISKGEOMETRY;
u32Translation : cardinal;
cbDisk : int64;
cbBlock : cardinal;
cbBlockExtra : cardinal;
cBlocks : cardinal;
cBlocksAllocated : cardinal;
uuidCreate : RTUUID;
uuidModify : RTUUID;
uuidLinkage : RTUUID;
uuidParentModify : RTUUID;
end;

I think that offData is the offset to the filesystem but I could be wrong. Those of you who have successfully mounted your vdi files using manually calculated offsets could you check to see if the value you determined matches the 4-byte value stored at offset 456? In the .vdi files at my disposal the headers look like this:

x.vdi
File Info: <<< innotek VirtualBox Disk Image >>>\n
Signature: BEDA107F
Version: 65537
cbHeader: 384
u32Type: 2
fFlags: 0
Comment:
offBlocks: 512
offData: 1024
Geometry.cCylinders: 0
Geometry.cHeads: 0
Geometry.cSectors: 0
Geometry.cbSector: 512
u32Translation: 3
cbDisk: 8388608
cbBlock: 1048576
cbBlockExtra: 0
cBlocks: 8
cBlocksAllocated: 8

ubuntu.vdi
File Info: <<< innotek VirtualBox Disk Image >>>\n
Signature: BEDA107F
Version: 65537
cbHeader: 384
u32Type: 1
fFlags: 0
Comment:
offBlocks: 512
offData: 40448
Geometry.cCylinders: 20083
Geometry.cHeads: 16
Geometry.cSectors: 63
Geometry.cbSector: 512
u32Translation: 3
cbDisk: 10365173760
cbBlock: 1048576
cbBlockExtra: 0
cBlocks: 9885
cBlocksAllocated: 6158

freeBSD.vdi
File Info: <<< innotek VirtualBox Disk Image >>>\n
Signature: BEDA107F
Version: 65537
cbHeader: 384
u32Type: 1
fFlags: 0
Comment:
offBlocks: 512
offData: 40448
Geometry.cCylinders: 20083
Geometry.cHeads: 16
Geometry.cSectors: 63
Geometry.cbSector: 512
u32Translation: 3
cbDisk: 10365173760
cbBlock: 1048576
cbBlockExtra: 0
cBlocks: 9885
cBlocksAllocated: 63

test2.vdi
File Info: <<< innotek VirtualBox Disk Image >>>\n
Signature: BEDA107F
Version: 65537
cbHeader: 384
u32Type: 2
fFlags: 0
Comment: Converted image from test.vdi
offBlocks: 512
offData: 25600
Geometry.cCylinders: 0
Geometry.cHeads: 0
Geometry.cSectors: 0
Geometry.cbSector: 512
u32Translation: 3
cbDisk: 6457171456
cbBlock: 1048576
cbBlockExtra: 0
cBlocks: 6159
cBlocksAllocated: 6159

John Hansen
Ingo
Volunteer
Posts: 731
Joined: 22. Aug 2007, 10:13
Location: Germany

Post by Ingo »

Hi John,
very interesting. Have you found this in the source?
I've checked it on a fix and dynamic sized VDI but couldn't find any offData on offset 456 (0x1C8).

Code: Select all

$ hd -vn 512 dyn.vdi
00000000  3c 3c 3c 20 69 6e 6e 6f  74 65 6b 20 56 69 72 74  |<<< innotek Virt|
00000010  75 61 6c 42 6f 78 20 44  69 73 6b 20 49 6d 61 67  |ualBox Disk Imag|
00000020  65 20 3e 3e 3e 0a 00 00  00 00 00 00 00 00 00 00  |e >>>...........|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  7f 10 da be 01 00 01 00  80 01 00 00 01 00 00 00  |................|
--- snip ---
00000180  10 27 00 00 86 0a 00 00  09 ed a2 aa c2 82 15 49  |.'.............I|
00000190  20 ab 09 e7 e1 39 44 bd  a8 f3 39 de 22 e3 c6 43  | ....9D...9."..C|
000001a0  ce 8a ec 20 dc 01 cf a4  00 00 00 00 00 00 00 00  |... ............|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200
$ hd -vn 512 fix.vdi 
00000000  3c 3c 3c 20 69 6e 6e 6f  74 65 6b 20 56 69 72 74  |<<< innotek Virt|
00000010  75 61 6c 42 6f 78 20 44  69 73 6b 20 49 6d 61 67  |ualBox Disk Imag|
00000020  65 20 3e 3e 3e 0a 00 00  00 00 00 00 00 00 00 00  |e >>>...........|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  7f 10 da be 01 00 01 00  80 01 00 00 02 00 00 00  |................|
--- snip ---
00000180  00 08 00 00 00 08 00 00  d8 4d e6 dc d0 76 7a 42  |.........M...vzB|
00000190  f3 aa 84 1e a3 48 e2 a7  40 9c 37 17 ea f9 7b 4d  |.....H..@.7...{M|
000001a0  d8 9f 3a d1 24 5c 84 6b  00 00 00 00 00 00 00 00  |..:.$\.k........|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200
$
b.t.w. offset 76 (0x4C) shows if it is dynamic=1 or fix=2. That was me told by innotek.
.
Last edited by Ingo on 13. Dec 2007, 09:58, edited 2 times in total.
afanofosc
Posts: 4
Joined: 12. Dec 2007, 04:42

Post by afanofosc »

The structures I mentioned are Object Pascal translations (Delphi or FreePascal) of VDI header structures found in the virtual box source code. But I checked further last night and the value stored in the offData structure member is not the offset to the file system in the .vdi file. And I calculated the offset incorrectly. The actual offset to the offData value is sizeof(VDIPREHEADER)+sizeof(cardinal)+sizeof(cardinal)+sizeof(cardinal)+256+sizeof(cardinal).

72+4+4+4+256+4=344

My apologies for the wrong offset value and for thinking that offData was the right place to look to find the start of the file system within the .vdi file. I am still inclined to believe that the values stored in the VDIHEADER1 structure are the place to look to find the file system offset. It may require a calculation involving more than one of the header values.

The 76 byte offset for the vdi file type (dynamic vs fixed) is the u32Type field in the VDIHEADER1 structure. It is at sizeof(VDIPREHEADER)+sizeof(cardinal) = 72+4 = 76.

John Hansen
afanofosc
Posts: 4
Joined: 12. Dec 2007, 04:42

Post by afanofosc »

The value 456, by the way, is the sizeof(VDIPREHEADER)+sizeof(VDIHEADER1), which is obviously not the right value for where the offData member of VDIHEADER1 is located in the file. Doh!

John Hansen
afanofosc
Posts: 4
Joined: 12. Dec 2007, 04:42

Post by afanofosc »

These two structures are defined in src\VBox\Devices\Storage\VDICore.h at line 78 (VDIPREHEADER) and 131 (VDIHEADER1). This is from the VirtualBox-1.5.2_OSE source code distribution. The comments in this header file do seem to suggest that offData is the right value.


/** Offset of image data from the begining of image file.
* Should be sector-aligned for HDD access optimization. */
uint32_t offData;


DECLINLINE(unsigned) getImageDataOffset(PVDIHEADER ph)
{
switch (GET_MAJOR_HEADER_VERSION(ph))
{
case 0: return sizeof(VDIPREHEADER) + sizeof(VDIHEADER0) + \
(ph->u.v0.cBlocks * sizeof(VDIIMAGEBLOCKPOINTER));
case 1: return ph->u.v1.offData;
}
AssertFailed();
return 0;
}

In VDICore.cpp the vdiInitHeader function initializes offData using this code (line 160):

/* Init offsets. */
pHeader->u.v1.offBlocks = RT_ALIGN_32(sizeof(VDIPREHEADER) + sizeof(VDIHEADER1), VDI_GEOMETRY_SECTOR_SIZE);
pHeader->u.v1.offData = RT_ALIGN_32(pHeader->u.v1.offBlocks + (pHeader->u.v1.cBlocks * sizeof(VDIIMAGEBLOCKPOINTER)), VDI_GEOMETRY_SECTOR_SIZE);


At line 319 the vdiSetupImageDesc function sets pImage->offStartData using the getImageDataOffset shown above.

Maybe the tests I ran last night were incorrect and this is the right value for the start of the file system. I will try a few more tests to see if I can figure out if I was doing something wrong last night. I was trying to mount an ext2fs disk from a .vdi file on a FAT32 partition mounted on /media/sda1 while running ubuntu on my laptop (with sda2 mounted at /) The offset that I thought was right (25600) resulted in a "specify the filesystem type" message so I dumped the first megabyte and at 25600 I saw stuff that did not look like it was correct.

John Hansen
ghr
Volunteer
Posts: 376
Joined: 25. May 2007, 22:46
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: DOS, Win3x, Win95, WinXP, Ubuntu, OS/2

Post by ghr »

Nice tool, ImDisk !

For MS DOS & Win9X the magic strings are:
- MSDOS5.0 (DOS5, DOS6)
- MSWIN4.1 (WIN95, WIN98SE)

The harddisk starts 3 bytes before those strings.

More partitions ? Just keep on scanning :-)
Ingo
Volunteer
Posts: 731
Joined: 22. Aug 2007, 10:13
Location: Germany

Post by Ingo »

How do I find an ext2/ext3 partition?
Is there a magic string?
The simplest way to a complex system is to start with a simple system.
Post Reply