MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Discussions related to using VirtualBox on Mac OS X hosts.
Post Reply
da_667
Posts: 4
Joined: 22. May 2021, 03:37

MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by da_667 »

Good Evening,

I've been trying to troubleshoot a problem I'm having with Virtualbox on MacOS Big Sur 11.3.1 My environment is a little bit interesting. I'm running MacOS Big Sur in a VMware Fusion VM, and inside of that VM, I'm running Virtualbox. Before it gets said, I know that nested virtualization is, in general, pretty awful. This is for testing purposes only. I would also like to confirm that the Big Sur VM has been given ample disk space (80GB) RAM (8GB), and CPU cores (4). Additionally, I have the settings "Enable hypervisor applications in this virtual machine" and "Enable IOMMU in this virtual machine" enabled to do nested virtualization.

In fact, running virtual machines inside the VM isn't the problem I'm having, I'm having a problem with the "VirtualboxVM" application, and obtaining a virtual console session to my VM. I am attempting to run a virtualbox VM with the following settings:

ISO: pfSense 2.5.1
RAM: 1GB
Disk: 16GB, dynamically allocated
Type: FreeBSD, 64-bit)
vm name: walls
what happens when I try to start a virtual console session to my pfSense VM.
what happens when I try to start a virtual console session to my pfSense VM.
Capture2.PNG (90.07 KiB) Viewed 5519 times
Here is a snippet of the VBoxSVC[.]log file:

Code: Select all

nspr-6    ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={85632c68-b5bb-4316-1900-5eb28d3413df} aComponent={SessionMachine} aText={Saved screenshot data is not available (VERR_NOT_Supported)}, preserve=false aResultDetail=-37
I've tried google, and searching the forums to see if there was any information I can fall back on, and I didn't find much. As I've stated above, I can run the VM in headless mode, and the little preview window in the virtualbox manager even shows what is going on, but attempting to connect a session instantly crashes, the console session and the virtual machine itself. Out of curiousity, I tried running:

Code: Select all

VirtualBoxVM --startvm walls 
to start my VM from the command line, and I get a little bit more information, but not much:

Code: Select all

VirtualBoxVM[1049:30603] *** WARNING: Textured winow <QNSWindow: 0x7ffa2b7485c0> is getting an implicitly transparent titlebar. This will break when linking against newer SDKs. Use NSWindow's -titlebarAppearsTransparent=YES instead.
Unable to open Fosl library at path /usr/lib/libFosl_dynamic.dylib
zsh: segmentation fault VirtualboxVM --startvm walls
Trying to find libFosl_dynamic.dylib, failing, and segfaulting.
Trying to find libFosl_dynamic.dylib, failing, and segfaulting.
Capture.JPG (58.22 KiB) Viewed 5519 times
Curious, I googled for that error message "Unable to open Fosl library at path /usr/lib/libFosl_dynamic.dylib" and found an old post that implied that virtualbox doesn't need that library, and that the system that had this problem was heavily modified.

forums [.] virtualbox [.] org/viewtopic [.] php?f=8&t=87547


I can assure you that this is a stock OSX Big Sur virtual machine. I have since installed XCode since I had heard that Xcode includes "libFosl_dynamic.dylib", but they appear to be specific dylibs use to build Apple TV, iOS, or WatchOS applications. Even so, its not possible to to copy the libFosl_dynamic[.]dylib file to /usr/lib, because the directory is protected by SIP. So I tried putting it in /usr/local/lib, and VirtualBox doesn't look for the library there.

SO THEN, I tried running DYLD_INSERT_LIBRARIES=/usr/local/lib/libFosl_dynamic.dylib VirtualboxVM --startvm

and it still segfaulted, refusing to acknowledge that the library was even there. At this point, I'm at a loss and I would appreciate any insight. Is this something I should submit a bug for? Is there more information I could provide?

Thanks in advance for your help.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by mpack »

The only supported nested VM scenario is VirtualBox inside VirtualBox. And even that requires specific CPU features, and I wouldn't recommend it.
da_667
Posts: 4
Joined: 22. May 2021, 03:37

Re: MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by da_667 »

Hey there, I appreciate your time and response. I kind of figured that this isn't a normal scenario by any means. Has anyone experienced this type of problem running VirtualBox on Big Sur bare-metal? I would imagine not, or there would probably have been a lot more search results for the problem I posted.

I looked a little bit more into this problem, and I've discovered that Big Sur has changed how dynamic libraries are accessed. The dynamic linker cache requests applications access dylibs in a specific way. I have no doubt you know about this, but I'm far from a MacOS expert, so it was news to me. I'm more curious as to why its requesting it in the first place.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by mpack »

da_667 wrote:I have no doubt you know about this, but I'm far from a MacOS expert
Nope, I'm not a MacOS user. I've never even seen it running.

You can certainly wait for a MacOS user to come along, but I suspect the answer will be the same: too much can go wrong when you run one VM inside another, and it will be slow even if it works (5 layers of heavy abstraction from the hardware will not lead to ideal performance (OS1+hypervisor1+OS2+hypervisor2+OS3).
granada29
Volunteer
Posts: 687
Joined: 3. Mar 2015, 07:27
Primary OS: Mac OS X other
VBox Version: OSE other
Guest OSses: Linux, macOS, Windows

Re: MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by granada29 »

@da_667

What hardware are you using to run the VMWare Fusion VM? Please bear in mind that if it is not Apple manufactured we are unable to provide any support.
da_667
Posts: 4
Joined: 22. May 2021, 03:37

Re: MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by da_667 »

Hey there,

Just a heads up, I was able to get a Big Sur virtual machine running on Virtualbox, and the exact same issue occurs. I get the exact same errors.
Capture2.PNG
Capture2.PNG (85.31 KiB) Viewed 5337 times
and then the exact same error running

Code: Select all

VirtualBoxVM --startvm walls
Capture.JPG
Capture.JPG (56.68 KiB) Viewed 5337 times
The Big Sur VM has: 4 virtual CPUs allocated, Nested AMD-V/VT-x is enabled, Nested Paging is enabled, 6GB of RAM is allocated, and 80GB of disk space is allocated. What's really fun is that I can't even run virtual machines in headless mode when trying to launch the VM headless. Soon as I start the VM in headless mode, the entire Big Sur VM locks up and kernel panics.

If virtualbox inside of virtualbox is considered "supported" what do I do to acquire support for this problem or submit a bug report? What other information do you recommend I gather for this?

Thanks in advance for your help and your responses so far.
da_667
Posts: 4
Joined: 22. May 2021, 03:37

Re: MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by da_667 »

granada29 wrote:@da_667

What hardware are you using to run the VMWare Fusion VM? Please bear in mind that if it is not Apple manufactured we are unable to provide any support.

does this provide you the information you require?
stats.jpg
stats.jpg (31.26 KiB) Viewed 5337 times
cox290
Posts: 1
Joined: 4. Mar 2023, 11:02

Re: MacOS Big Sur 11.3.1: Failed to open a session for the virtual machine [name]

Post by cox290 »

Hey, I meet the same problem.
Has anyone experienced this type of problem running VirtualBox on Big Sur bare-metal?
Yes I'm exactly the one running VB on Big Sur...Have you cracked this?
Post Reply