Changing from immutable to normal disk loses changes and viceversa

Discussions related to using VirtualBox on Windows hosts.
bazinga
Posts: 3
Joined: 23. Oct 2019, 09:21

Changing from immutable to normal disk loses changes and viceversa

Post by bazinga »

Hello,
I'm using Virtualbox for some years and occasionally use immutable disks but I still can't get how to deal with them or if I have to realize that VB has some bugs in this regard. Consider this scenario: I have a VM with a normal disk and I want to convert it to invariable.

To do this i'm using the VB GUI > File > Virtual Media Manager > choose the disk > change from Normal to Immutable > confirm (OK)

When I need to modify or update the machine I do the same to revert the disk to Normal, start the VM, perform my updates, close (turn off) the VM and then convert the disk to invariable again with the same steps.

The problem is: once the disk is reverted from Normal to Immutable it looses all the update / changes.
I also tried (and this my usual way) to detach the disk from the VM before changing its type (normal or immutable), then reattaching it to the VM once I changed the disk type. But this way also has no effect, every change made inside the VM OS (when disk is set to Normal) is lost when I reconvert the disk.

What's wrong with this process? Does the immutable disk attribute have to be handled differently? Or is it buggy?

Also, when I make changes to the VM sub-versions of the disk are create, as in image attached.
Which version do I have to select when I re-attach the disk to the VM? The main item or the latest sub-version?
Attachments
VirtualBox_disk_versions.png
VirtualBox_disk_versions.png (10.3 KiB) Viewed 4545 times
Last edited by mpack on 6. Jul 2022, 16:26, edited 1 time in total.
Reason: Fix heinous spelling error "looses" in title. Didn't bother fixing it in body.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Changing from immutable to normal disk looses changes and viceversa

Post by scottgus1 »

bazinga wrote:I have a VM with a normal disk and I want to convert it to invariable immutable.
Just want to keep terms steady. "Immutable" is the term used in the manual. "Invariable" as a word does not appear in the manual. (We both know what we're talking about, but we don't want to confuse other readers.)

The manual says:
Immutable images only remember write accesses temporarily while the virtual machine is
running. All changes are lost when the virtual machine is powered on the next time.
This means that each time the VM is restarted, the changes should be gone.
bazinga wrote:The problem is: once the disk is reverted from Normal to Immutable it looses all the update / changes.
Are you also including snapshots in this scheme?

Maybe your setup is too complicated, and using a disk style that is designed to lose data intentionally is part of the problem. Backups of the normal disk could enable you to revert to a known state, and are tremendously more stable, without any intentional data loss design constraints.

What ultimate goal are you trying to achieve?
bazinga
Posts: 3
Joined: 23. Oct 2019, 09:21

Re: Changing from immutable to normal disk looses changes and viceversa

Post by bazinga »

I need a VM with a immutable disk to have a test environment where to try new programs, upgrades of software, different configurations, etc.
So an immutable disk is perfect for this use.
But I want to keep the OS and the basic software installed updated, so every now and then I need to convert the disk to Normal, apply updates and then revert to immutable again. That's pretty simple.
But once I convert the disk to normal, apply updates and convert it again to immutable all updates are lost. That's weird, there has to be something wrong with a step.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Changing from immutable to normal disk looses changes and viceversa

Post by fth0 »

FWIW, I'm using a similar strategy, but with VM snapshots instead of the immutable property:

After I've setup a new VM for the first time, I create a snapshot called "Base". For testing purposes, I run the VM, do some tests and close the VM by using the VM window's close button and selecting "Power off the machine" > "Restore current snapshot 'Base'". When updating the VM, I run the VM, update the guest, properly shut down the VM from within the guest OS, delete the previous snapshot and recreate it. Of course, more complex workflows are possible.

Regarding your problems with the immutable property, I also could imagine that you're doing something wrong, especially if you don't know the technical background. I'd suggest to read 5.5. Differencing Images for a better understanding of the underlying principles.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Changing from immutable to normal disk looses changes and viceversa

Post by scottgus1 »

I would handle this with basic bog-standard file-copy backups of the VM folder in the state I want it to be able to revert to.

A file copy backup (especially if integrity-verified with a file compare after the copy) to multiple locations not only serves as the "original" that you're now using 'immutable' on and glitching when updating, but also is a real backup, which can recover from the major disasters that could happen to any computer at any time.

As a replacement for the 'immutable-normal-immutable' process, once you get the VM exactly as you like it, shut it down completely from within the VM OS (not save-state). Then copy the VM's folder (and any disk files that are stored outside the VM's folder) to one or more backup media (preferably more than one for true backup purposes). Run a FC file compare on the copies and original to confirm that the backups are good.

Now, use the VM as normal (not immutable-ing the drive, leave it 'normal). When the time comes for you to revert to the original VM state and update, simply copy the backup disk file back over the original that has been changed. Now the VM disk has been reverted to the backed-up state. Then you can run updates and make a new set of backups. Added benefit: versioning, if you keep the original backups and not overwrite then with the new backups. Updates go south? Something you didn't notice two reverts ago? Older backups save the day!

Even regular Virtualbox snapshots can go bad if the host has a glitch while the snapshot is being restored or deleted. And we don't know where we are with the 'immutable-normal-immutable' process. Backups cover both possible failures.
Galactic
Posts: 6
Joined: 28. Apr 2016, 04:52

Re: Changing from immutable to normal disk loses changes and viceversa

Post by Galactic »

When there is a report of data loss that takes such a small amount of time to test, I think it pays to go ahead and do the test. So that's what I did. I was able to confirm this behavior.

The problem occurs during any immutable to normal transition. The final switch back to immutable, as performed by bazinga, plays no part.

Just to better illustrate the root of the problem, I'm actually going to give an example where I start with a normal disk, then switch to immutable and back.

When switching from NORMAL to IMMUTABLE, the following happens in that VM's configuration file:
- The existing HardDisk has its type changed from "Normal" to "Immutable".
- A new snapshot disk is created as a child of that HardDisk, with autoReset="True".
- The AttachedDevice uuid is switched from the main disk to that of the child snapshot.

This will cause all changes to be written to the snapshot, which is reset automatically every time the VM is started. So far, so good.

When switching from IMMUTABLE to NORMAL, the following happens in that VM's configuration file:
- The HardDisk has its type changed from "Immutable" to "Normal".

And that's all that happens. The type will now be displayed as "Normal" in the GUI, etc. But all changes are still written to the snapshot, which is reset automatically every time the VM is started. Whoops.

From this point on, you can change back and forth between immutable and normal to your heart's content, and the only change that will be made is to the type displayed.

When switching from immutable to normal, the best procedure (for now) is to additionally release and remove the child snapshot. In bazinga's case, this would mean removing the one beginning with "ea382483-". Since that snapshot is what was technically attached to the VM, the main disk file will have to be reattached to the VM afterward. Future changes will then be written to the main disk file as expected.

Due to the data-loss issue, I would consider a fix to be important and necessary. One possibility is to change the behavior so that the snapshot is auto-deleted. Another more cautious approach would be to detect a condition that is not going to act as expected (i.e. the disk has just been changed from immutable to normal and has a child with autoReset="True") and display an explanatory dialog box that gives the option to perform or not perform the same steps in my workaround above.
Last edited by mpack on 7. Jul 2022, 14:48, edited 1 time in total.
Reason: Flip terms accidentally swapped by poster.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Changing from immutable to normal disk loses changes and viceversa

Post by fth0 »

Galactic wrote:When switching from immutable to normal, the best procedure (for now) is [...]
I think you're correct (*). The VirtualBox User Manual does not explain how to get back from the immutable to the normal state, and executing the manual steps is necessary. It's conceivable that no automatic procedure is implemented on purpose. Since the Virtualbox developers usually do not read the user-to-user forums here, you could create a enhancement request in the Bugtracker.

(*) with the minor exception that you confused the states in your "has its type changed from [...] to [...]" statements. You might want to correct them in your post, or explain why you think it's correct. ;)

bazinga wrote:What's wrong with this process? Does the immutable disk attribute have to be handled differently? Or is it buggy?
Have those questions been sufficiently addressed with the posts so far, or do you still have questions?
Galactic
Posts: 6
Joined: 28. Apr 2016, 04:52

Re: Changing from immutable to normal disk loses changes and viceversa

Post by Galactic »

fth0 wrote:(*) with the minor exception that you confused the states in your "has its type changed from [...] to [...]" statements. You might want to correct them in your post, or explain why you think it's correct. ;)
You are absolutely right. Those should both be flipped. If I haven't spotted and fixed a typo before submitting, there must be one somewhere. That's some kind of universal law. Don't see an edit button though.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Changing from immutable to normal disk loses changes and viceversa

Post by mpack »

I have done the edits for you.

One pedantic note: an immutable drive involves a differencing disk, it does not involve a snapshot. In this case the distinction is important because there is no snapshot info in the .vbox file, so if the VM were to be converted from immutable to a snapshot type then snapshot info would have to be synthesized. Or simply delete the differencing disk, though that involves data loss. I suspect this will be the complication that leads to the bug.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Changing from immutable to normal disk loses changes and viceversa

Post by fth0 »

mpack wrote:if the VM were to be converted from immutable to a snapshot type
I like being pedantic, too: ;)

A snapshot is a state of a VM, while immutable is a type of a virtual disk image. So there is no such thing as an immutable VM. You can even have snapshots of VMs when the VM already contains an immutable image, making things even more complicated.
mpack wrote:I suspect this will be the complication that leads to the bug.
BTW, I don't see any bug ... I think I can explain everything the OP wrote as intended and even documented behavior.
Galactic
Posts: 6
Joined: 28. Apr 2016, 04:52

Re: Changing from immutable to normal disk loses changes and viceversa

Post by Galactic »

mpack wrote:One pedantic note: an immutable drive involves a differencing disk, it does not involve a snapshot.
Yeah, I agree with you here. I should use that term. In the moment, I was trying to describe it in vbox config file terms as "not the parent HardDisk element, but the child ones listed as being in the Snapshot folder". Differencing disks/images are often referred to as snapshots by various software, so it's not like wrong wrong generally. But it is too easy to confuse it with the VM snapshot feature in this case.
fth0 wrote:BTW, I don't see any bug ... I think I can explain everything the OP wrote as intended and even documented behavior.
I would make the case that this is, indeed, a bug (i.e. a problem that needs fixing). The reason is that the Normal/Immutable distinction is entirely a vbox config file construction. It is not a property of the disk images themselves.

We can say to ourselves "Well of course changing the type to Normal doesn't do anything. That disk isn't attached to the VM anymore. The differencing disk is attached instead." But then, I ask you, what does it mean for that disk to be labelled "Immutable" in the first place?

A disk isn't immutable simply because it is left unchanged. Every disk image that is part of a VM snapshot ought to have the "Immutable" label in that case. No, a disk is immutable because a differencing disk was created and attached with autoReset="true". It is the existence of this configuration in its entirety that makes a disk immutable (the fact that you can manually edit the config file later to mess it all up notwithstanding).

It stands to reason that changing a disk from Immutable to Normal must remove its immutableness. If the aforementioned configuration changes are not reversed, then it simply cannot claim to be "Normal". Therefore, the current behavior is not the appropriate one. This is especially true since it risks unintentional data loss.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Changing from immutable to normal disk loses changes and viceversa

Post by mpack »

Galactic wrote: A disk isn't immutable simply because it is left unchanged. Every disk image that is part of a VM snapshot ought to have the "Immutable" label in that case. No, a disk is immutable because a differencing disk was created and attached with autoReset="true".
Don't agree. The disk is immutable because the user asked for it to be tagged and treated that way. The stuff you highlight are just details of the current implementation, not part of the definition. And treatment of snapshot parents do not become a comparable case just because some details of the implementation are similar.

But I do agree that it's clearly a bug. If the documented API allows you to change a drive from immutable to normal then either that should work or you should be explicitly prevented. Under no circumstances should it result in a broken config that requires manual edits to fix.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Changing from immutable to normal disk loses changes and viceversa

Post by fth0 »

I'll agree that the current behavior doesn't match the expectations and also isn't user-friendly. Let me describe a few more details that may not be obvious:

1. At the time you change the type of a disk image from Normal to Immutable, the disk image is not attached to any VM (if necessary, you're forced to detach it during the type change). No differencing image is created during this type change.

2. Afterwards, you can choose to attach the immutable disk image to one or more VMs (just like Multiattach mode images, but with the additional autoReset property). Although it looks like it, VirtualBox does not attach the immutable disk image to the VM, but creates a differencing image and attaches that to the VM. (In the VirtualBox Manager in the Storage tab of the VM configuration, hover over the disk image to see the 'true' technical details.)

3. When using a VM with an immutable disk image, you can create (an arbitrary tree of) VM snapshots, and the autoReset property then resets the VM to the current snapshot instead of the immutable image. Note that a VM can have multiple immutable and normal disk images at the same time, complicating the setup even further.

When the user changes the type of one disk image from Immutable to Normal, what should VirtualBox do? There are a lot of possibilities, like automatically handling the simplest case discussed in this thread, or interactively handling some cases, or denying the type change in complex setups.

I'm not saying that there is no good answer to what VirtualBox is supposed to do, just that it's not a simple change, and therefore competes with other VirtualBox enhancements for the available development resources.

mpack wrote:But I do agree that it's clearly a bug. If the documented API allows you to change a drive from immutable to normal then either that should work or you should be explicitly prevented. Under no circumstances should it result in a broken config that requires manual edits to fix.
After you've changed the type of a disk image from Immutable to Normal, (nobody tells you that) you have to decide if you want to keep the differencing image with the autoReset property or to remove it yourself in the Virtual Media Manager. That's bad usability, of course.

Let's see if we can get an assessment from klaus ...
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Changing from immutable to normal disk loses changes and viceversa

Post by fth0 »

... have had a chat with klaus and some experiments tonight, with the following results:

1. Immutable images are being used by very few people, and the typical use case does not involve persisting changes, so a bug in that area could have gone unnoticed for quite a long time.

2. Once upon a time, it was only possible to change the image type (e.g. from 'immutable' to 'normal') when there didn't exist a differencing image, which means you were forced to remove it manually before doing the type change.

3. Later on, the implementation of changing the image type was enhanced, but seems to allow the differencing image and the autoReset property to keep existing, as discussed in this thread. This can be considered a bug. But:

4. There is a better workflow: Starting the VM having an immutable image Base.vdi and a differencing image Diff.vdi, make changes you want to persist and shut down the VM from within its OS. Afterwards, use VBoxManage clonemedium disk Diff.vdi Temp.vdi and VBoxManage clonemedium disk Temp.vdi Base.vdi --existing, and you're done.

5. There's an even shorter alternative directly using VBoxManage clonemedium disk Diff.vdi Base.vdi --existing, but this doesn't work (and perhaps is another bug).

I've verified 4. myself. Please test if it also works for you, and report back. TIA.
Galactic
Posts: 6
Joined: 28. Apr 2016, 04:52

Re: Changing from immutable to normal disk loses changes and viceversa

Post by Galactic »

mpack wrote:Don't agree. The disk is immutable because the user asked for it to be tagged and treated that way. The stuff you highlight are just details of the current implementation, not part of the definition. And treatment of snapshot parents do not become a comparable case just because some details of the implementation are similar.
Actually, you are restating the same thing I was. Sorry if that wasn't clear. I'm saying disks labelled "Immutable" and ones part of snapshots and labelled "Normal" may incidentally share some qualities, but they aren't the same thing. I made that comparison to drive home the point that selecting Immutable for a disk has a special meaning. It is implemented in a way that respects that meaning. Likewise, switching back to Normal should make the proper changes to respect its own meaning.

---

I've had time to do more thorough tests now (even finding another bug I may post about later, but I digress). The documentation and "VBoxManage modifymedium --autoreset" command show that autoReset is intended as a way to override the default behavior of an attached differencing disk. But it just opens up a logical can of worms. This is the ultimate source of the bug. Let's see why...

For brevity:
- When I say "current state", I mean the disk from the AttachedDevice element of what is displayed in the GUI as "Current State".
- When I say "parent", I mean the highest parent in the tree (i.e. the base image and not a differencing disk).

For disks with children, the relevant details are as follows:
1. A type change always changes the type of the parent.
2. If children already exist, a type change does not modify them.
3. If a child must be created in the process, this occurs after the type change.
4. A new child receives the autoReset="true" attribute only if the parent is Immutable.
5. VirtualBox honors the autoReset attribute (except for snapshots) instead of the default behavior of the parent disk. The documentation for VBoxManage modifymedium says that the autoReset attribute can only be modified on Immutable disks, but that's not true. It can be freely enabled or disabled on other disk types as well.

Examples (starting with a newly-created Normal disk):
Normal -> Immutable -> Normal will end up leaving autoReset="true" on the current state.
Normal -> Multi-attach -> Immutable will lack autoReset on the current state.

Depending what type changes are made and whether the user has manually enabled or disabled autoReset on anything, autoReset may be present or absent on any number of children in the tree and in any combination.

I feel that being able to use the same parent disk with multiple VMs or linked clones, and have different behaviors on each, might be useful rarely. But I don't think it's logically workable in its current implementation.

Consider:
1. Snapshots always act like snapshots and ignore autoReset. So, autoReset only matters for the current state of each VM in question.
2. It is opaque to the user whether autoReset is enabled, and very easy for that setting to get overwritten. If a new current state gets created (e.g. by restoring a snapshot), then it will always have the parent's default for autoReset.
3. Since autoReset is intended to be a manually adjustable setting, independent of disk type, it's impossible to automatically handle this setting during a disk type change.

Given all of the above, I don't know how autoReset could ever be used reliably. Personally, I would remove this feature in its current form and simply honor the parent disk's type. If something like autoReset is present at all, it needs to be on a per-disk per-VM setting. So in the GUI, it could be a checkbox in the VM's storage settings for a particular disk.
Post Reply