Differencing VMDK Drives Created from Parent Physical Drive? How to Merge?

[Alamantus]

Hi there, I'm in a situation that no web searching has brought me an appropriate result for.

Context:
I set up my web server to connect to a physical drive by following a tutorial that directed me to run

Code: Select all

VBoxManage internalcommands createrawvmdk -filename "C:\Users\<user_name>\VirtualBox VMs\<VM_folder_name>file_name.vmdk" -rawdisk \\.\PhysicalDrive#

. This worked great after I found my drive number using

diskpart

and then

list disk

and offlined the disk, and I used it for about 6 months in my guest Ubuntu 20.04 VM without any trouble.

Little did I know that the files I started adding to the physical drive were not in fact being added to the drive itself, but to a new vmdk file in the server's Snapshots folder instead! This vmdk file was created only a month after I created the raw physical drive vmdk, but since it seamlessly overlays within the guest OS, I had no idea. Flash forward to a power failure screwing up my guest OS, and I want to access the files from the physical drive from a different computer, where I find that I don't have any of the new files I had added over the past 6 months.

I searched around and that's when I noticed the snapshot files in the Virtual Media Manager labeled Type (Format) "Differencing (VMDK)" when I hover over them:

Snapshots VMDK files of my physical drive

physicaldrivesnapshots.png (34.61 KiB) Viewed 4111 times

Sure enough, my plan of using the external hard drive to store the files so my main hard drive didn't get filled up did not work as I had intended, and all my files and changes over the last 6 months are trapped in those UUID-labeled VMDK files instead.

Question:
How can I merge these snapshot files back onto the original physical drive? And how can I prevent Virtualbox from using snapshot files instead of the physical drive in the future?

I found this tutorial that uses a VMWare tool (www [dot] ubackup [dot] com/enterprise-backup/merge-vmdk-files [dot] html), but it appears to just merge into a new VMDK file instead of actually overwriting an existing one. And my issue goes further because I want that merged result back on the actual hard drive. Also, when I ran this

Code: Select all

.\vmware-vdiskmanager.exe -r "D:\Robbie\Virtual Machines\nuevo2\Snapshots - Copy\PhysicalDrive3.vmdk" -t 0 "D:\Robbie\Virtual Machines\nuevo2\Snapshots - Copy\merge\PhysicalDrive3.vmdk"

I got this:

Creating disk 'D:\Robbie\Virtual Machines\nuevo2\Snapshots - Copy\merge\PhysicalDrive3.vmdk'
Failed to convert disk: The specified file is not a virtual disk (0x3ebf).

Further context: I don't know what I'm doing, so I copied the raw vmdk file PhysicalDrive3.vmdk to the Snapshots folder (a copy, as you can see above) and named the snapshot vmdk files as PhysicalDrive3-0001.vmdk, PhysicalDrive3-0002.vmdk, and PhysicalDrive3-0003.vmdk because it seemed like incremental numbers were necessary or something, but I really don't know.

Any help with this would be very much appreciated!

[mpack]

The command you used at the top of your post does not create snapshots or differencing disks, so something is missing from this story.

VirtualBox provides no tools that let you modify a host drive, and while it's easy enough to do using Macrium Reflect (backup from inside the VM, restore to the physical drive), I would think very carefully before doing so, because the VM is a different PC, and the virtual drive contents are not necessarily correct for the physical PC.

Frankly, accessing the physical drive at all was IMO a mistake. What on earth for? A cloned virtual drive would have been better.

[fth0]

I don't know how to solve the issue, but I can give some hints:

The virtual disk might have a read-only property. This could be defined within the VMDK descriptor file, which is relatively easy to recognize. Additionally, AFAIR, administrator privileges are required on a Windows host for raw disk access, but I'm not sure if there is a distinction between read-only and read-write access.

If you provide a zip file containing the PhysicalDrive3.vmdk file and a VBox.log file from a VM run, we can have a look.

[Alamantus]

The command you used at the top of your post does not create snapshots or differencing disks, so something is missing from this story.

That's all I did with it. I added it as a drive to the VM, and I see that it's classified as "normal" rather than immutable or anything like that in the virtual media manager. The write-through option in section 5.4 of the manual is not currently an option for me, which may have been my original mistake, but at the time I had no idea about it.

VirtualBox provides no tools that let you modify a host drive, and while it's easy enough to do using Macrium Reflect (backup from inside the VM, restore to the physical drive), I would think very carefully before doing so, because the VM is a different PC, and the virtual drive contents are not necessarily correct for the physical PC.

That's an interesting idea that could potentially work! Maybe if I attach another large drive as a shared device I can save the backup to that so it can be accessed by the host to restore the correct files or something.

Frankly, accessing the physical drive at all was IMO a mistake. What on earth for? A cloned virtual drive would have been better.

To be clear, this physical drive is an external drive used only for storage. It's an ext4 formatted hard drive that I chose specifically to be an external drive for this VM, and for some reason I had to use the raw disk to connect it because Nextcloud would not let me use a shared drive as the storage directory. I have no idea why not, but this is the only thing that worked when I was setting it up.

Maybe I could have done it differently, but this was the only way I knew how to have my files on the external drive and safe if I ever needed to grab it and go or move to a different computer. (Obviously that's not how it worked out, but saving the files to a virtual drive on Virtualbox also does not accomplish my goal.)

Additionally, AFAIR, administrator privileges are required on a Windows host for raw disk access, but I'm not sure if there is a distinction between read-only and read-write access.

You are correct about administrator privileges for raw disk access. Every time I launch Virtualbox, I must manually run 3 executables as administrator, so I created a folder with shortcuts to run first VBoxSVC.exe, then VBoxSDS.exe, and finally VirtualBox.exe. (I was looking at what executables were running when I was setting it up, and running these as administrator are what allowed me to access my physical drive—only running VirtualBox.exe as admin never worked, it had to include the first 2 executables first. I don't know enough about VirtualBox to know why.)

The virtual disk might have a read-only property. This could be defined within the VMDK descriptor file, which is relatively easy to recognize. ... If you provide a zip file containing the PhysicalDrive3.vmdk file and a VBox.log file from a VM run, we can have a look.

It's not set to Immutable or anything, but maybe you can see something! Like I said, I don't know enough about virtualbox to really know what's going on, so I've attached the PhysicalDrive3.vmdk file and just included all of the log files I have for the VM.

Be aware, the date created for all of them are from yesterday, so I don't know if older logs get cleared automatically, but my VM has been having other issues I can't figure out, so they may be clogged with issues related to the broken OS—but I have no real idea, I'm sorry to say.

If it helps more, here are the contents of PhysicalDrive3.vmdk when I open it in a text editor:

Code: Select all

# Disk DescriptorFile
version=1
CID=5ba50917
parentCID=ffffffff
createType="fullDevice"

# Extent description
RW 5860466688 FLAT "\\.\PhysicalDrive3" 0

# The disk Data Base 
#DDB

ddb.virtualHWVersion = "4"
ddb.adapterType="ide"
ddb.geometry.cylinders="16383"
ddb.geometry.heads="16"
ddb.geometry.sectors="63"
ddb.uuid.image="8a517ff6-cf9e-4b5e-a246-bbb7c4cb0699"
ddb.uuid.parent="00000000-0000-0000-0000-000000000000"
ddb.uuid.modification="c5ac3515-e855-4326-b596-ec77036ab489"
ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"
ddb.geometry.biosCylinders="1024"
ddb.geometry.biosHeads="255"
ddb.geometry.biosSectors="63"

[fth0]

Ok, it'll get a bit complicated now, so please read everything carefully before taking any action:

1. The read-only vs. read-write topic: In PhysicalDrive3.vmdk, in the Extent description, we can see that the physical disk is supposed to be attached to the VM as read-write, so far, so good. I'll make an educated guess that this physical disk is still attached to the Windows host OS, and it will therefore deny VirtualBox the read-write access, unless the physical disk is detached from within the Windows Disk Management. Important: Do not try to change this right now, or you might loose data!

2. The VirtualBox differencing disk images: The VM nuevo2 has two virtual disks, nuevo2.vdi being the first. The second virtual disk currently (VBox.log) consists of the base disk \\.\PhysicalDrive3 (1.36 TB) plus the differencing image {e35f326c-a1fd-473c-8cb3-3dccdb2949bb}.vmdk (~350 MB). Previously (VBox.log.3), the second virtual disk consisted of the base disk \\.\PhysicalDrive3 (1.36 TB) plus the differencing image {2e00ef58-8b6a-455d-b494-adfea9b237d7}.vmdk (~115 GB). Additionally, there is a third differencing image {7ce2...}.vmdk (~350 MB). Important: If you'd change the physical disk to read-write right now and start the VM, the physical disk might get modified, which could damage the (probably important) blue colored combination.

3. The Windows host OS is running under the Hyper-V hypervisor, providing you with additional problems. I don't know if it also plays a role in the topics above.

In the following, I'll suggest one possible strategy to solve the situation:

a) First, the pink and green colored differencing images above are quite small and probably contain only content that is written during the guest OS startup. We'll throw them away in this strategy. Please tell if you think that my assessment is wrong.

b) I'll guess that you're interested in (some of) the ~115 GB of data in the blue colored differencing image above. If you have additional 2.73 TB of disk space available somewhere, clone the disk image using VBoxManage clonemedium disk {2e00ef58-8b6a-455d-b494-adfea9b237d7}.vmdk new.vmdk.

c) Make a backup of the pink, blue and green colored differencing images above, then try to remove them inside the Virtual Media Manager.

d) In the Windows host OS, detach \\.\PhysicalDrive3, perhaps using DiskPart. Google how to do that.

e) Start the VM and shut it down from within the guest OS. If a new differencing image has been created, remove it and go back to step d). If no differencing image has been created, you've succeeded in making the physical disk writable.

f) Add new.vmdk as a third virtual disk image to the VM, start the VM and copy the important data to the second virtual disk image, which now should equal the physical disk.

g) Get rid of Hyper-V on the Windows host: HMR3Init: Attempting fall back to NEM (Hyper-V is active).

Please report back what worked and what not.

[Alamantus]

Thanks so much for looking into this! It looks like you are correct—the changes I want are actually just in one of the differencing disks, the largest one. I switched which differencing disk was connected to the VM, and the smaller ones were just copies of the old drive state while the large one had the changes I needed. The other ones seem to have been created very recently though I'm not 100% sure when or why. I also didn't realize the bit about Hyper-V! I did install WSL a little while ago, which does use Hyper-V (in WSL2 as far as I can tell), so that makes some sense why there are problems.

I don't have the extra ~3TB to spare, but your post did help me realize that I could just take a backup of the files and essentially start over. I am able to copy the files from the VM out to a shared folder, which may be all I need since the drive was just for file storage anyway. The scary thing for me is realizing that if I want the current state of the drive (i.e. including file deletions and moves I made), I essentially will need to wipe the drive and copy the files back onto it again because just copying/overwriting will leave the deleted/moved files in places. Maybe I can use robocopy in Windows with the /mir option? I'll have to see what will work best for my storage space situation.

Thank you again! If I make any significant discoveries as I try to recover this, I'll update here, but for now this should be sufficient for me.