Raw host (normal) disk and differencing snapshot - merge??

[whytek]

1st, apologies if this has been covered, i'm not having much luck with the forum search..

Summary: I'm looking for a manual merge differencing command, and some clarification on what would happen if I let VB merge after the Original Disk has changed. This is a question for tekkies..


Setup:
* Virtual Machine on Linux host, guest is Windows XP
* Guest Primary master is a Normal VDI
* Secondary Master is a Raw Host VMDK built as in Section 9.5.1.2 of the manual with the -partitions 1,2 parameter.
* One of those partitions is my XP partition that I directly boot into.

This is more or less the series of events that lead me to a current situation.
* A snapshot was taken on the XP Virtual Box, - VirtualBox made differencing disk for the Primary and Secondary Master
* I didn't boot up the Virtual Machine in some months.
* On booting up the virtual machine today, Windows XP went right into scandisk mode on the Raw host HD, the disk looked severely corrupted. (i stopped it after I realised what was going on)

This is what I assume is happening:
The raw host disk has changed dramatically since the differencing disk was last written to, so as it says in the manual, if a sector is not available in the differencing disk, it goes to the original, but if the original has changed, then well... anything can happen, right? OK So the lesson here is don't make a raw host disk in "normal" mode if you are going to write to it outside virtualbox, use write-through mode, correct?

Now, the thing is, I would like to delete the original snapshot on this XP virtual machine and have the Primary Master original and differencing disk merged, yet if I do this, common sense is telling me that VB is going to try and merge the differencing disk for the secondary master too!! arghh. I certainly don't want that to happen, if it's going to take the sectors in the differencing disk and write them to the Raw Host disk.. that would be disaster for my XP partition! Obviously I don't want the differencing disk for the raw host at all, I would like to just delete them.

I tried renaming the differencing disk, but now VB won't let me delete the snapshot and merge, it complains about file not found on the differencing disk files. I tried removing the differencing disk in Virtual Media Manager, but I can't (presumably because it's in use by the snapshot, I would have to delete the snapshot, but then I would loose the changes to the primary master which I want to keep)

To sum up:
Am I correct? If i let vb merge the differencing disk to the raw host disk, will it trash my XP partition?
How can I merge the primary master?

I hope I have explained, thanks!

[mpack]

If you are worried about corrupting your raw partition then back it up to removable storage before you start, using Acronis, Ghost, Clonezilla or even (shudder) dd.

While all snapshots are differencing disks, not all differencing disks are snapshots. AFAIK the "merge" action only happens when you delete a snapshot, not when you delete any old difference disk. However, I wouldn't want to bet my data on that, so take a backup like I said.

And you ought to stop messing about with individual filenames etc - let the GUI take care of that, that's what it's there for. Thrashing around blindly will only add to the confusion.

[whytek]

um, thanks mpack for that.. although I have to admit I can't see what you are making reference to there in your last sentence.

To update, since last night, I've pretty much discovered that there is no way to manually do a merge down of original and differencing disks.
The code to do it is obviously there, as it happens when the Delete snapshot functions correctly, so maybe at some point we could have access to it via VBoxManage internalcommands or something? thanks.

I also could not get the GUI to allow me to delete the snapshots in this VM, not any of them.. I can create news ones and then delete them, but I just can't delete the existing ones. I tried all kinds of combinations of manual edits on the xml config files, removing all other storage apart from the disks I want to merge, but nothing. I also found a lot of references to posts by some fairly unhappy people who found themselves in a similar situation, or worse, lost work by losing access to a differencial disk for one reason or another. In short.. the general opinion seems to be don't make snapshots on a system where you care much about data written to disk while running the snapshot. i hope this gets fixed and cleaned up at some point.. (can't wait to try out MacOS too under linux host!!)

k.

[mpack]

um, thanks mpack for that.. although I have to admit I can't see what you are making reference to there in your last sentence.

My last message was referring to where you said you had tried renaming a difference disk, I assumed that meant renaming the file outside the GUI, since I don't know what else it could refer to. In your most recent message you also make general references to manual xml edits. Basically, if you start manipulating the VM in unsupported ways then some here (myself included) will be less inclined to support you.

To update, since last night, I've pretty much discovered that there is no way to manually do a merge down of original and differencing disks.

The aforementioned Acronis, Ghost, Clonezilla etc could be used for this. My CloneVDI tool should be able to do it as well come to think of it. In fact I would have thought that "VBoxManage clonehd <mostrecentchild.vdi>" would work as well, no need for an "internalcommands".

the general opinion seems to be don't make snapshots on a system where you care much about data written to disk

That is certainly my opinion.

[whytek]

Thanks again mpack, - I'm detecting the tone getting a little argumentative here.. so let's knock that on the head.. ok?
I moved out of the GUI because the GUI wouldn't let me do what I needed to do, this is a natural move for me as i cut my teeth "outside of the gui"

Basically, if you start manipulating the VM in unsupported ways then some here (myself included) will be less inclined to support you.

Am I in the wrong forum? My question didn't seem OS specific, neither host nor guest OS, and there isn't an "experimenting" forum.

I'm interested in discussion, experimentation, learning and ultimately improving virtualbox.
That's why i'm experimenting with renaming files and making analysis of the configuration files, and then posting the results here.
I consider it part of a learning process, in which I am now advancing. I maybe posted too early, I should have done some more experimentation myself first.

Now, - Ok I now understand your reference to Acronis etc.. If you mean mount the differential disk inside a virtual box and use one of those utilities to basically copy it to another disk - yes I can see that working, good. TrueImage would avoid those pesky windows activation problems when copying a windows system partition.

clonehd worked brilliantly! this is excellent. Thanks for that. It was not clear to me from the manual
that clonehd can be used for that. For the record, you can merge differential disks like this:

VBoxManage clonehd Machines/Windows/Snapshots/\{9985b154-50e3-4110-af17-3b76c597d583\}.vdi cloned.vdi --format VDI

where Machines/Windows/Snapshots/\{9985b154-50e3-4110-af17-3b76c597d583\}.vdi is the path on my system to the latest differential disk and cloned.vdi is the new disk I am creating.
I can now delete the old disk and all its differencial copies and replace the master in my VirtualBox with cloned.vdi, thus recovering the disk space used up by all the snapshots.

mpack, I was also unaware of your CloneVDI tool, next reboot in windows i'll check it out. thanks again.

[mpack]

Thanks again mpack, - I'm detecting the tone getting a little argumentative here.. so let's knock that on the head.. ok?

There was none of that on my part. Disagreement need not always descend into abuse - however I agree that the point is moot anyway.

My point about doing unsupported things is simply that if we have someone who may be a novice, and who doesn't "keep to the tourist trail" (i.e. makes peripheral changes which he refers to only vaguely) then it's hard for me to model in my head just what exactly he has done, which makes it difficult for me to see a solution. That's all. If you look back to my original comment on this then you'll see that I mentioned my concern: confusion. Others heads may work in a different way and be quite happy with this scenario.

Glad you found the tips helpful.

[gillianreynolds]

In a real PC, hard disks and CD/DVD drives are connected to a device called hard disk controller which drives hard disk operation and data transfers. VirtualBox can emulate the four most common types of hard disk controllers typically found in today's PCs: IDE, SATA (AHCI), SCSI and SAS.[18]

IDE (ATA) controllers have been in use since the 1980s. Initially, this type of interface worked only with hard disks, but was later extended to also support CD-ROM drives and other types of removable media. In physical PCs, this standard uses flat ribbon parallel cables with 40 or 80 wires. Each such cable can connect two devices to a controller, which have traditionally been called "master" and "slave". Typical hard disk controllers have two connectors for such cables; as a result, most PCs support up to four devices.

In VirtualBox, each virtual machine has one IDE controller enabled by default, which gives you up to four virtual storage devices that you can attach to the machine. (By default, one of these four -- the secondary master -- is preconfigured to be the machine's virtual CD/DVD drive, but this can be changed.[19])

So even if your guest operating system has no support for SCSI or SATA devices, it should always be able to see the default IDE controller that is enabled by default.

You can also select which exact type of IDE controller hardware VirtualBox should present to the virtual machine (PIIX3, PIIX4 or ICH6). This makes no difference in terms of performance, but if you import a virtual machine from another virtualization product, the operating system in that machine may expect a particular controller and crash if it isn't found.

After you have created a new virtual machine with the "New Virtual Machine" wizard of the graphical user interface, you will typically see one IDE controller in the machine's "Storage" settings where the virtual CD/DVD drive will be attached to one of the four ports of this controller.

Serial ATA (SATA) is a newer standard introduced in 2003. Compared to IDE, it supports both much higher speeds and more devices per hard disk controller. Also, with physical hardware, devices can be added and removed while the system is running. The standard interface for SATA controllers is called Advanced Host Controller Interface (AHCI).

For compatibility reasons, AHCI controllers by default operate the disks attached to it in a so-called "IDE compatibility mode", unless SATA support is explicitly requested. "IDE compatibility mode" only means that the drives can be seen and operated by the computer's BIOS. Still, disks assigned to those slots will operate in full-speed AHCI mode once the guest operating system has loaded its AHCI device driver.

Like a real SATA controller, VirtualBox's virtual SATA controller operates faster and also consumes less CPU resources than the virtual IDE controller. Also, this allows you to connect up to 30 virtual hard disks to one machine instead of just three, as with the VirtualBox IDE controller (with the DVD drive already attached). Of these, the first four (numbered 0-3 in the graphical user interface) are operated in IDE compatibility mode by default.

For this reason, starting with version 3.2 and depending on the selected guest operating system, VirtualBox uses SATA as the default for newly created virtual machines. One virtual SATA controller is created by default, and the default disk that is created with a new VM is attached to this controller.
Warning

The entire SATA controller and the virtual disks attached to it (including those in IDE compatibility mode) will not be seen by operating systems that do not have device support for AHCI. In particular, there is no support for AHCI in Windows before Windows Vista, so Windows XP (even SP2) will not see such disks unless you install additional drivers. It is possible to switch from IDE to SATA after installation by installing the SATA drivers and changing the controller type in the VM settings dialog.[20]

To add a SATA controller to a machine for which it has not been enabled by default (either because it was created by an earlier version of VirtualBox, or because SATA is not supported by default by the selected guest operating system), go to the "Storage" page of the machine's settings dialog, click on the "Add Controller" button under the "Storage Tree" box and then select "Add SATA Controller". After this, the additional controller will appear as a separate PCI device in the virtual machine, and you can add virtual disks to it.

To change the IDE compatibility mode settings for the SATA controller, please see the section called “VBoxManage storagectl”.

SCSI is another established industry standard, standing for "Small Computer System Interface". SCSI was standardized as early as 1986 as a generic interface for data transfer between all kinds of devices, including storage devices. Today SCSI is still used for connecting hard disks and tape devices, but it has mostly been displaced in commodity hardware. It is still in common use in high-performance workstations and servers.

Primarily for compatibility with other virtualization software, VirtualBox optionally supports LSI Logic and BusLogic SCSI controllers, to each of which up to 15 virtual hard disks can be attached.

To enable a SCSI controller, on the "Storage" page of a virtual machine's settings dialog, click on the "Add Controller" button under the "Storage Tree" box and then select "Add SCSI Controller". After this, the additional controller will appear as a separate PCI device in the virtual machine.
Warning

As with the other controller types, a SCSI controller will only be seen by operating systems with device support for it. Windows 2003 and later ships with drivers for the LSI Logic controller, while Windows NT 4.0 and Windows 2000 ships with drivers for the BusLogic controller. Windows XP ships with drivers for neither.

Serial Attached SCSI (SAS) is another bus standard which uses the SCSI command set. As opposed to SCSI, however, with physical devices, serial cables are used instead of parallel ones, which simplifies physical device connections. In some ways, therefore, SAS is to SCSI what SATA is to IDE: it allows for more reliable and faster connections.

To support high-end guests which require SAS controllers, VirtualBox emulates a LSI Logic SAS controller, which can be enabled much the same way as a SCSI controller. At this time, up to eight devices can be connected to the SAS controller.
Warning

As with SATA, the SAS controller will only be seen by operating systems with device support for it. In particular, there is no support for SAS in Windows before Windows Vista, so Windows XP (even SP2) will not see such disks unless you install additional drivers.

In summary, VirtualBox gives you the following categories of virtual storage slots:

four slots attached to the traditional IDE controller, which are always present (one of which typically is a virtual CD/DVD drive);

30 slots attached to the SATA controller, if enabled and provided that your guest operating system can see it; these slots can either be

in IDE compatibility mode (by default, slots 0-3) or

in SATA mode;

15 slots attached to the SCSI controller, if enabled and supported by the guest operating system;

eight slots attached to the SAS controller, if enabled and supported by the guest operating system.

Given this large choice of storage controllers, you may ask yourself which one to choose. In general, you should avoid IDE unless it is the only controller supported by your guest. Whether you use SATA, SCSI or SAS does not make any real difference. The variety of controllers is only supplied for VirtualBox for compatibility with existing hardware and other hypervisors.