[SOLVED] After Clone virtual guest gets into emergency mode

[abcuser]

EDIT: Jump down directly to the solution.
On Windows 10 host on VirtualBox 7.0.10 I have fresh downloaded ISO and installed Rocky Linux 9.2 and then updated it with "sudo dnf update" and rebooted computer. Computer boots successfully, I can login etc.

In VirtualBox GUI I have right clicked on virtual machine and selected Clone. Typed in new virtual machine in Name field all other settings left default (like I always do) and clicked Finish button. Machine is cloned. I start up cloned machine and when machine starts up Linux gets into emergency mode. It looks like something went wrong at cloning. I deleted cloned virtual machine and did another clone. The same problem.

I am attaching zipped VBox.log and print-screen after Linux boots.

Any idea what is wrong?

[mpack]

I can make a guess as to the problem, but haven't we discussed this before? You need to modify the grub boot so that a partition is identified by hardware slot rather than UUID, as the latter is changed by cloning.

[abcuser]

@mpack, I have been using VirtualBox 15+ years and I have cloned plenty of machines and I have never got into this problem before. I mostly used Ubuntu, maybe this is no issue on Ubuntu. But now I need "Rocky Linux 9" which I have first time got into this problem.

I have pressed "e" on grub menu.

grub.png (9.37 KiB) Viewed 5169 times

but now I don't know what to change? There are no UUIDs visible.

If it helps, some info from /etc/fstab

fstab.png (9.22 KiB) Viewed 5169 times

and info from "lsblk"

lsblk.png (4.21 KiB) Viewed 5169 times

[scottgus1]

From the log I see normally running VM "hardware". And the error messages are from inside the VM OS. So Virtualbox appears to have not actually done anything it's not programmed to do.

all other settings left default (like I always do)

We should know what defaults these settings are on your host, just to be sure.

Just a point of logic to give consideration to:

I have been using VirtualBox 15+ years and I have cloned plenty of machines and I have never got into this problem before. I mostly used Ubuntu, maybe this is no issue on Ubuntu.

How long one has used Virtualbox is not correlatable to what a particular OS allows for its boot instructions:

modify the grub boot so that a partition is identified by hardware slot rather than UUID

If Ubuntu IDs its partitions by hardware slot and Rocky does by something else, then a 15-year-old default that doesn't break Ubuntu could break Rocky. And FWIW the information about your VM's grub settings and your years of experience came after Mpack's response to your original 'I cloned a Linux VM, it won't boot, what do I do?' post sans any indication there was experience behind the issue or awareness of what could have gone wrong.

Curiously, and maybe this is simply some form of "variablizer", Rocky's boot listing mentions a "/dev/mapper/rl_etc" and the actual hardware listing says it is "/dev/sda/sda2/rl_etc". There is also a UUID related to booting in the /etc/fstab. I'm no Linux guru, but maybe these differences are important.

Can you please post both of the Rocky Linux .vbox files, the original and the clone:

Right-click each Rocky VM in the main Virtualbox window's VM list, choose Show in Explorer/Finder/File Manager. Copy the VM's .vbox file (not the .vbox-prev file) to the desktop (Configure your host OS to show all extensions if the folder that opens does not show a .vbox file). Zip both the .vbox files and post the zip file, using the forum's Attachments tab. Please label which is the original and which is the clone.

Also, please check whether the Clone command's "Keep Hardware UUIDs" was checked or unchecked when you made the clone.

And please post the output of

vboxmanage showmediuminfo "driveletter:\path\to\each\VM's\hard\drive.vdi"

for each VM?

[abcuser]

We should know what defaults these settings are on your host, just to be sure.

I did in red:

defaults.png (81.05 KiB) Viewed 5122 times

----------------------------------------------------------------------------------------------------------------------

Can you please post both of the Rocky Linux .vbox files, the original and the clone:

and

And please post the output of

vboxmanage showmediuminfo "driveletter:\path\to\each\VM's\hard\drive.vdi"

for each VM?

All files attached at:

vbox.zip

(3.34 KiB) Downloaded 19 times

----------------------------------------------------------------------------------------------------------------------

Also, please check whether the Clone command's "Keep Hardware UUIDs" was checked or unchecked when you made the clone.

It was unchecked. See image above.
----------------------------------------------------------------------------------------------------------------------
Everything at till now I have had both settings "Keep Disk Names" and "Keep Hardware UUIDs" unchecked (like I always do when I do a clone).

Now I tested all possibilities of those two settings:
1. "Keep Disk Names" checked and "Keep Hardware UUIDs" unchecked.
2. "Keep Disk Names" unchecked and "Keep Hardware UUIDs" checked.
3. Both "Keep Disk Names" and "Keep Hardware UUIDs" checked.
Starting each of the virtual machine one at the time (no other virtual machine running) and I get the same error after Linux boot "emergency mode" in all three cases.

[mpack]

AFAIK "Keep hardware UUIDs" would not help anyway. That effectively refers to the motherboard ID. Disk UUIDs are something else and will always change when you clone a VM, because VirtualBox requires that all registered media on a host has a unique UUID.

That is precisely the problem: if your boot code selects the boot drive based on its UUID then booting will be broken in any clone.

[scottgus1]

As my forum guru colleague Mpack has observed, I was curious about the status of "Keep hardware UUIDs". And as Mpack points out, the manual does say this checkbox only handles things like the "motherboard" UUID which might interfere with activation and licensing.

So I may have been barking up the wrong tree.

FWIW the clone VM uses a disk file called "clone.vdi". But the "showmediuminfo.txt only contains two copies of the original VM's "Rocky_Linux_9_2_samo_za_kloniranje.vdi" output.

And the info may have been not pertinent anyway, if disk UUIDs always change.

However, here is a possible test: Swap the VMs' hard drives. "Rocky_Linux_9_2_samo_za_kloniranje.vdi" to the clone, "clone.vdi" to the original. If the problem also swaps from the clone to the original, and the clone starts booting, then the problem is inside the grub loaders of the OS's, not in Virtualbox.

If so, you'll have to get on Rocky Linux's support channels to figure out how to rebuild the grub loader.

Another possibility: if you change the grub in the clone to say "/dev/sda/sda2/rl_etc" instead of "/dev/mapper/rl_etc", and the thing boots, it could be the "mapper" isn't mapping something correctly anymore. Find the "mapper" and see what it maps to.

[fth0]

I don't know the solution (yet), but I can give you an additional hint:

In the GRUB output, the definition of $root is not shown and its definition will probably be the missing link to be found and corrected. Note that it's not the value in the root= argument, but something else pointing to a combination of /dev/sda1 and /boot).

---

Regarding UUIDs, there's a common misunderstanding, because there are multiple (UU)IDs involved:

Code: Select all

a) c2011e67-835e-4335-9e60-a28c4df402f1  VirtualBox hard disk UUID
b) ata-VBOX_HARDDISK_VBc2011e67-f102f44d  VirtualBox/Linux hard drive ID
c) 0bb9c7c1-e6fc-4f1a-9adb-f463e77d1d37  Linux filesystem UUID

a) VirtualBox uses a UUID to manage hard disks in its Media Registry. This UUID gets changed when cloning, but it doesn't play a direct role in GRUB/Linux.

b) Linux uses a hard drive ID that is provided by the (virtual) hard drive. VirtualBox derives this hard drive ID from its hard disk UUID (look closely at the example above to see how). When this hard drive ID is used in GRUB to select a hard drive, it has to be corrected after cloning.

c) Linux can use a UUID to identify filesystems in /etc/fstab. This UUID can be safely used in GRUB to select a hard disk partition, because VirtualBox does not change it when cloning.

PS: Debian/Ubuntu/Mint search the boot partition using the Linux filesystem UUID, that's why VirtualBox's cloning doesn't damage anything.

[mpack]

a) VirtualBox uses a UUID to manage hard disks in its Media Registry. This UUID gets changed when cloning, but it doesn't play a direct role in GRUB/Linux.

It definitely does play a role. I have a lot of experience with drive cloning, I know whereof I speak.

When the guest OS asks for a drive UUID, this is what VirtualBox gives it - rather I believe it delivers a shorter code derived from the UUID, but the effect is the same.

The situation used to be the same for the motherboard UUID. Rather than reinvent the wheel VirtualBox used to give the guest OS the same UUID it already used to identify the VM. A few versions back it separated those two functions, but no such separation AFAIK has ever happened with drives.

[fth0]

When the guest OS asks for a drive UUID, this is what VirtualBox gives it - rather I believe it delivers a shorter code derived from the UUID, but the effect is the same.

This is exactly what I tried to describe in b) in my post. And you didn't follow my orders to look at the example, else you wouldn't have said "I believe".

It definitely does play a role. I have a lot of experience with drive cloning, I know whereof I speak.

I know! I'm just adding some details to your experience. You seem to have ignored the word "direct", which was meant to implicitly say that the role is indirect (as described in b)). Was my English phrasing not good enough to convey that?

PS: My attempt to describe the difference between Linux hard drive IDs and Linux filesystem UUIDs was motivated by your usage of the term "UUID" in places where only "ID" would have been correct.

[abcuser]

Problem solved.
TL;DR: jump to SOLUTION.

First of I did another test:
- Rocky Linux 8.8 clone with VirtualBox machine and cloned virtual machine works fine.
- Rocky Linux 9.2 clone with VirtualBox machine and cloned virtual machine goes in "emergency mode" when booting.

How I solved the problem?
- In cloned machine at "Emergency mode" I typed in root password.
- Then executed: "journal -x" to go through boot process. There I found the error related to VLM.
- I searched the web for error message:
warning: no autoactivation for /dev/sda2: sydstem.devices t10.ATA_VBOX_HARDDISK_Vb current t10.ATA_VBOX_HARDDISK".
- I have got the the VirtualBox topic where in the last post there was useful information for me.

SOLUTION:
1. Open /etc/lvm/lvm.conf file with text editor (I used "vi").
2. Searched "use_devicesfile" in file and I got: # use_devicesfile = 1
3. I removed the "#" character and changed 1 to 0, to get: use_devicesfile = 0
4. Rebooted Linux with "reboot" command and magic has happened, problem solved.

[scottgus1]

Glad you're up and running!

It would be very curious to see if Rocky 8.8 has the "use_devicesfile=0" #'d or not.

Also, if you remove the # in Rocky 9's file in the original, does it still boot OK? And if you clone the modified Rocky 9 original with the # removed, does the clone boot OK?

[abcuser]

It would be very curious to see if Rocky 8.8 has the "use_devicesfile=0" #'d or not.

In Rocky 8.8 I have checked the file /etc/lvm/lvm.conf and there is setting:
# use_devicesfile = 0
Line is commented out, but this is default. But obviously this has changed in Rocky Linux 9 to:
# use_devicesfile = 1

Also, if you remove the # in Rocky 9's file in the original, does it still boot OK? And if you clone the modified Rocky 9 original with the # removed, does the clone boot OK?

I did the test:
1. I deleted the clone virtual machine, just to make sure I haven't done anything else.
2. In original virtual machine in file /etc/lvm/lvm.conf I changed to "use_devicesfile = 0".
3. Cloned a virtual machine (I did "clone" just like described in my previous posts, see clone image above).
4. Started original and immediately after, I started clone machine. So both machines are booting at the same time.
5. Both machine successfully booted and I can login just normally and it appears everything is working fine.

EDIT: I have asked a question on Rocky Linux forum, just to link if something interesting appears. I got hint there may be LVM problem.

[fth0]

Thanks for the hints and explanations!

For readers interested in background information: Chapter 11. Limiting LVM device visibility and usage.

[scottgus1]

Thanks for doing the tests! Glad the new clone worked.