Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Discussions about using Linux guests in VirtualBox.
Post Reply
jkane13
Posts: 31
Joined: 1. Nov 2008, 04:31

Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Post by jkane13 »

I am getting to an EFI Shell prompt and when I try to look at FS0: there is nothing to see.

Config:
Windows 10 64 bit host
Ubuntu 17.04 64 bit guest
VirtualBox 5.1.26
Made sure the motherboard bios has virtualization enabled. I only saw 32 bit O/S options before doing that.

I installed Windows on a SSD card in UEFI mode. Then I rebooted and installed Ubuntu on another SSD card in UEFI mode. I can boot from the UEFI menu (F2) with no problems bouncing between the 2.

I created the guest and told it "Do not add a virtual hard disk".

Then I ran this:
PS C:\Program Files\Oracle\VirtualBox> .\VBoxManage.exe internalcommands listpartitions -rawdisk \\.\PhysicalDrive2
Number Type StartCHS EndCHS Size (MiB) Start (Sect)
1 0xef 0 /32 /33 7 /197/62 60 2048
5 0x83 7 /230/32 1023/254/63 238413 126976

PS C:\Program Files\Oracle\VirtualBox> .\VBoxManage internalcommands createrawvmdk -filename "C:\Users\Jeff\VirtualBox VMs\Ubuntu\Ubuntu.vmdk" -rawdisk \\.\PhysicalDrive2
RAW host disk access VMDK file C:\Users\Jeff\VirtualBox VMs\Ubuntu\Ubuntu.vmdk created successfully.

I then added the virtual VDMK to the VM as a SATA drive under Storage.

Under System I made sure Enable EFI is checked. If I do not, I get a grub recovery menu.

When I start the VM, I get to an EFI shell prompt. If I enter "FS0:" and then "dir", it says "File Not Found".

If under Storage I Use Host I/O Cache is enabled, I get "edit: Disk Error" when trying to create a startup.nsh file.

If not enabled, I can edit startup.nsh. I then see it in the folder, but it is only thing there. No EFI folder or anything else.. When I reset, it hangs. I suspect because there is not path to EFI\ubuntu\grubx64.efi.

I booted Ubuntu and added a startup.nsf file in the root of the /boot partition. It is still there if I boot to Ubuntu.

I am stuck now. I have been reading this forum for a couple of days now and it got me as far as I am. But still stuck at the EFI Shell prompt. I can enter FS0: but I see nothing there. It has to be that EFI is not seeing the partition correctly inside of VirtualBox. If this will help. here is the Shell Map.

FS0: Alias(s):HD16a0a1:;BLK2:
PciRoot (0x0)/Pci(0xD,0x0)/Sata(0x0,0x0,0x0)/HD(1,MBR,0x6C17A9B0,0x800,0x1E000)
BLK0: Alias(s):
PciRoot (0x0)/Pci(0x1,0x1)/Ata(0x0)
BLK1: Alias(s):
PciRoot (0x0)/Pci(0xD,0x0)/Sata(0x0,0x0,0x0)
BLK3: Alias(s):
PciRoot (0x0)/Pci(0xD,0x0)/Sata(0x0,0x0,0x0)/HD(2,MBR,0x6C17A9B0,0x1EFFE,0x1D1A6802)
BLK3: Alias(s):
PciRoot (0x0)/Pci(0xD,0x0)/Sata(0x0,0x0,0x0)/HD(2,MBR,0x6C17A9B0,0x1EFFE,0x1D1A6802)/HD(1,MBR,0x00000000,0x1F000,0x1D1A6800)


Windows Disk manager shows that disk 2 is the right disk for Ubuntu.
DISKPART> select disk 0
Disk 0 is now the selected disk.
DISKPART> list partition
Partition ### Type Size Offset
------------- ---------------- ------- -------
Partition 1 Primary 500 MB 1024 KB
Partition 2 Primary 1862 GB 501 MB

DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> list partition
Partition ### Type Size Offset
------------- ---------------- ------- -------
Partition 1 Recovery 450 MB 1024 KB
Partition 2 System 100 MB 451 MB
Partition 3 Reserved 16 MB 551 MB
Partition 4 Primary 232 GB 567 MB

DISKPART> select disk 2
Disk 2 is now the selected disk.
DISKPART> list partition
Partition ### Type Size Offset
------------- ---------------- ------- -------
Partition 1 Primary 60 MB 1024 KB
Partition 0 Extended 232 GB 61 MB
Partition 2 Logical 232 GB 62 MB


I would have to reboot and close this browser where I am tying right now to do that to show what the partitions look like in Ubuntu. I can do that in a follow up if it will help.

Any ideas on my next step?

- Jeff
jkane13
Posts: 31
Joined: 1. Nov 2008, 04:31

Re: Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Post by jkane13 »

Nothing from anyone? Not even a comment asking why I need to use UEFI? :roll:

I gave up trying to figure it out. I reinstalled windows and ubuntu. Actually had to do windows twice. And it is still wrong. It keeps putting the boot record on the wrong drive. Right now it is on the 2TB mechanical disk drive. Not at all where I want it. Grrr!

But ... I was able to get Ubuntu to boot in Virtual box last night. However, I am using bios now instead of UEFI.

This is my first new computer in 15 years. When I saw that EFI was the new way to go, I figured I should use it instead of being old fashioned. However, it appears that EFI is still not ready for prime time.

I get to rebuild from scratch again tonight. I will have to remove the partitions on the 2TB drive to keep windows from seeing it and putting ht boot record there without even asking me about it first. Grrrr!
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Post by mpack »

jkane13 wrote:However, it appears that EFI is still not ready for prime time.
Not in VirtualBox certainly, as it's only intended for OS X guests. Discussions on this subject already exist.
jkane13
Posts: 31
Joined: 1. Nov 2008, 04:31

Re: Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Post by jkane13 »

There are a lot of threads about making EFI work in Virtualbox. They all assume there is something in FS0: or at least in one of the BLK#: devices. None of them I can find discuss what to do when a directory listing shows no files in any of the devices at all.
Last edited by socratis on 18. Aug 2017, 23:31, edited 1 time in total.
Reason: Removed unnecessary verbatim quote of the whole previous message.
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Post by Perryg »

There is no problem with EFI under normal use, but add RAW access and that is where you have an issue. I can't tell for sure since I have never tried to do this with RAW, but I suspect it will ruin the partitions ability to boot as a dual boot device.

Also from what I know ( Dan will need to confirm this ) on a Windows host you would need to run the guest as administrator to be able to get read/write to the partition. Linux hosts do not have this requirement.
jkane13
Posts: 31
Joined: 1. Nov 2008, 04:31

Re: Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Post by jkane13 »

It is true that the host has to run as administrator. That I can confirm.

Accessing them as raw did not do anything to the dual boot ability. Then again, it never quit made it into a full boot though for the guest Ubuntu.
jkane13
Posts: 31
Joined: 1. Nov 2008, 04:31

Re: Win Host Ubuntu Guest EFI VDMK on SSD FS0: sees nothing

Post by jkane13 »

If someone reads this later and wonders how it all turned out ...

Hardware:
Ryzen 1700x
ASRock Fatal1ty X370 Motherboard
32GB DDR4 G.Skill - Flare X Memory
GForce GTX 750 Ti WindForce 4 GB Video Card
Seagate - Barracuda 2TB 3.5" 7200 RPM HDD w/64MB Cache
2 Western Digital - Blue 250GB M.2-2280 SSD cards

Windows 64 Bit Home Edition
Ubuntu 17.04
Virtualbox 5.1.26

After 5 from scratch re-installs of both O/S's, and 6 for Windows, I have a machine that can dual boot from the Bios (F11) and can run either Windows or Ubuntu as the host with the other O/S in a Virtualbox inside. I did not want to use some dual boot tool like Grub2. I want to be able to run either O/S stand alone on it's own SSD, or to be able to run it inside of a VM while using the other O/S. I want to be able to remove all drives except one SSD and still be able to use the machine and not have to do a backup and restore.

Quick review of the rest of this thread. I installed as Bios the first time without even thinking about it. Then read that UEFI was the way to go to hedge my bets against the future and make this computer last longer. VirtualBox would not work with raw disks and UEFI. It saw empty drives in the EFI Shell. I had to reinstall to get back to Bios Boot records.

Just an FYI: I installed VMWare Player and it had no apparent way to use Raw Disks at all. The Pro version had discussions about it but those features were not in the free version. VirtualBox does not fully support raw disks, but it can be done with a little tweaking at least.

One of the catches I hit on was that I left the 2TB hard disk formatted after the first install. When going to UEFI, that was OK. But, when going back to Bios Boot, the Windows boot record kept getting installed to that HD instead of to an SSD. Windows was not letting me pick the location. I deleted that partition and left it like that until I had both O/S's installed on their own SSD's. That forced the install of the boot record to the correct SSD. The sequence was delete all partitions on all drives and then install Windows first, then Ubuntu, and then formatting the 2TB shared drive. Ubuntu did see the Windows install and added it as a boot option in Grub. I planned to remove it, but when I let Ubuntu do an update while in a Guest VM, grub removed it for me. Good enough. That was easier than I thought it would be. :-) As for deleting all partitions, that was not simple. You can not, of course, remove the partition that is currently booted from. And, the Windows install CD did not seem to follow my instructions in the custom set up to remove the UFI partition. I ended up doing it from Ubuntu and then telling Windows custom install to remove the Ubuntu partition which it was willing to do.

Getting Ubuntu to boot in a Virtualbox from Windows requires running Virtualbox as Administrator. Make sure you do this or it will fail. Follow the instructions you can find easily for how to manually set up raw disks for Virtualbox. The part they do not explain is sequence of events. Don't create the raw VDMK until after you create the VirtualBox VM. When creating it, tell it to not use any hard disk and you will add one later. If you try to put the VDMK in the folder before then, the create of the VM will fail because the path is already there and it won't reuse it. Once the VM is created, go ahead and run the command line for creating a VDMK pointing to the raw disk. Feel free to open that file in wordpad. It is XML, or it looks like xml. It is just some settings that tell the VM how to find the raw partition. Of course, only edit and save if you know what you are doing! Edit the Settings in the VM and add an existing hard disk. Pick the VMDK and save the VM configuration.

I found this site http://www.serverwatch.com/server-tutor ... ox-vm.html to have very simple directions for the command needed to create raw VDMK.

You may want to go through all of the other configuration settings too. Be sure to add the extensions pack before you do. Things like USB 3.0 support are added. I am not going to go over all the details, but make sure your motherboard has virtualization enabled. Also in the VM make the virtualization settings as you need them. Google searches can tell you a lot about how to set things up. Being dual boot is not special to those things. You can do things like that later too when you need them. Such as adding a shared drive. Sometimes it's easier to get things working first and then add all the features you want later.

If you are repeating the create VM step, do not just delete the folder with file explorer. Only remove it in the VirtualBox tool. Tell it to remove all data too. There is a config file in another location, you can search about this and find more details, that will store the UUID of the VDMK. When you try to reinstall it later, it will complain that it does not match. Those are .xml files and can be manually edited if you like. Or, create new VM with the same disk install location, and then go ahead and remove it right away. That will remove the old UUID from the config file and you can create it one more time clean.

Getting VirtualBox to work in Ubuntu was a bit more challenging. First I tried to use the Software Install GUI. There are 2 VirtualBox packages that do not say what version. I went to Virtualbox and downloaded the .deb file instead. When I tried to just open it, the Install Tool GUI opened, but did nothing. No matter how many times I clicked the install button, nothing seemed to happen. I tried to install from the command line. Got errors. There are dependencies that are not there by default and the package does not know how to find them. Time to go back and actually read the package download page from VirtualBox. ;-) Yeah, it's all explained there with step by step install instructions. Well, not all that clear, but the commands are mixed in with things that are not needed.

Installing the extensions needs to be done on the command line also. There is no file type linking like in Windows of course. I found the command somewhere. I forget where. Search for it. :-P

Next it gets messy! You want to create the VM first without a hard disk like in Windows. That creates the right XML files and the path structure. Then you need to run the create VMDK file on the path that VirtualBox wants to see it. You must do this as root. The same kind of command as with Windows, but for Linux. The commands are found in the same post listed above.

Unless you are logged in as root, it still will not work. You could run it from the command line with sudo if you wanted to. If you try to run as yourself and not root, you will get an error about not being able to open up the raw disk. This is because that disk device is owned by root and the group is disk. By default users are not in the group disk. There are no world permissions so you can not access the raw device /dev/sdb (or what ever the last letter is of your other SSD card). A search shows a few ways to skin this cat. Most are one time shots. You can change the permissions on /dev/sdb, but the next time you boot Ubuntu that gets changed back to defaults. The most long term way with the least complexity is to add yourself to the group disk. That sounds easy. sudo usermod -a -G disk <userName>. You should then just have to log out and back in. For some reason with Ubuntu 17.04 that did not work. I ended up having to reboot to get the new group membership to show up. I have no idea why, but I did it twice to make sure I was seeing what I thought I was. That is a gotcha!

Once you can see "disk" as a group when entering the groups command for yourself, you should be ready. Edit the VM to add the VDMK disk the same as it was done in Windows. It should then start up as expected.

If you want to add the application to your launcher, you run it the first time from the command line and then right click the icon to pin to the launcher. The command should be `virtualbox` to start the GUI. It should be on the path. After you pin it, you can then start VirtualBox from the launcher. If you want to directly start a specific VM, the command is `VBoxManage startvm "Windows"` where Windows is the name of the VM.

I didn't like that grub2 was presenting me a menu with just Ubuntu (and memory tests) in it. I wanted it to just boot into Ubuntu fast. Editing /etc/default/grub and making GRUB_TIMEOUT=0 skips the menu. Run update-grub and on the next boot no grub splash menu, just booting directly into Ubuntu. Of course that means if there is a boot problem after a new kernel compile, you will not get a chance to pick an older kernel from grub. For me this is worth it. I know how fix that problem.

Both Windows and Ubuntu are very fast to boot. Less than 15 seconds for Ubuntu and maybe 20 seconds for Windows. I am able to run Windows as the host and start up Ubuntu as a guest with no problems. Then I can reboot into Ubuntu as the host and run Windows as a guest. I was going to use Windows as the host since it tends to be more picky about the hardware, but after playing with both and seeing how fast I can reboot, I am using Ubnutu as the host and keeping Windows running on a second monitor since I use it less often. I want to keep Windows running because in the past I found that I ended up waiting 20 minutes for Windows to finish installing updates every time I started it up. If I run into a direct hardware issue, I can quickly make Windows the host and Ubuntu the guest again.

Of course, the disclaimer of "Do NOT try to read the host SSD card as a raw device from the guest" is something to heed. I did not try to do this to see what would happen. It just makes sense to not do it. Use the VirtualBox shared drives concept. I am sharing the 2TB "D" drive on both. I am not mounting the host drive on either guest. Any file sharing will be done on D.

I am kind of disappointed with the way Ubuntu mounts the 2TB "D" drive. It puts it down a path of /media/<userName>/D. I don't want to do a lot of customization like I did to my old Ubuntu machine. I always had problems on major version upgrades because I customized things like mount paths. I will learn to live with that path. I used to call it /d. And I mount a NAS drive as /backup. I guess I can keep using /backup as a short path, but the D drive will have a longer path name now. Hmm ... while typing this it occurs to me that I can probably create a link to just /d. Have to play with it and see if it works.

So ... that is my story. I hope this helps someone else some day.
Post Reply