So many issues: MacOSX host, Server 2022 and Win10 Guests

[bdonnachie]

I'm trying to run 1 x Kali Linux (which is fine), 1 x Server 2022 (Which is finally fine) and 5 x Win10 Guests (which seem to enjoy random crashes) on MacOSX 14.1.1 w/64GB RAM, 3.2GHz 6-core intel i7 with 1TB SSD onboard, 4TB external SSD where the VMs live and a 24TB zfs array.

Tried all sorts including:

• Downgraded to virtual box v6 including all the pain with SIP
  Upgraded to v7.0.12 r159484 (Qt5.15.2) (the one I'm now running)
  2GB RAM w/2-cores
  Changed to VBoxVGA driver
  Disabled audio
  Disabled clipboard
  Moved to dedicated external SSD and increased guest disk space to 100GB
  And so much more....

I thought I'd finally found a 'winning incantation' but this morning, VBoxIntNetSwitch crashed with bad instruction / 'api misuse':

Code: Select all

Crashed Thread:        6  Dispatch queue: com.apple.root.default-qos.overcommit

Exception Type:        EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes:       0x0000000000000001, 0x0000000000000000

Termination Reason:    Namespace SIGNAL, Code 4 Illegal instruction: 4
Terminating Process:   exc handler [4079]

Application Specific Signatures:
API Misuse

Which then took four of my guest VMs out with bad access:

Code: Select all

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       UNKNOWN_0xD at 0x0000000000000000
Exception Codes:       0x000000000000000d, 0x0000000000000000

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [60561]

PastedGraphic-1.png (60.24 KiB) Viewed 5580 times

I'm very grateful for all the work that has gone into VirtualBox but I desperately need a solution that works. I need these simulations to run consistently for several months for my research and the ability to set log in credentials from the host with VirtualBox really helps with my simulation.

There's not much of interest in the virtual box logs but I've attached them along with the crash reports.

CrashLogs.zip

(49.06 KiB) Downloaded 259 times

VirtualBox Logs.zip

(117.33 KiB) Downloaded 270 times

While I have a script which monitors for crashed VMs and restarts them - this crash took out the internal network breaking my simulations.

I really need these to work so I can move onto my next phase of research... please help!

[bdonnachie]

My last remaining Win10 guest has crashed:

Screenshot 2023-11-23 at 12.20.13 copy.jpg (39.38 KiB) Viewed 5575 times

VirtualBox log attached....

Code: Select all

03:12:44.588174 VMMDev: Guest Log: DllCanUnloadNow (refs=2)
03:13:22.622532 AssertLogRel /Users/vbox/tinderbox/7.0-mac-rel/src/VBox/VMM/VMMAll/PGMAllPhys.cpp(2772) VBOXSTRICTRC pgmPhysWriteHandler(PVMCC, PPGMPAGE, RTGCPHYS, const void *, size_t, PGMACCESSORIGIN): PGM_HANDLER_PHYS_IS_VALID_STATUS(rcStrict, true)
03:13:22.622579 rcStrict=VERR_NO_MEMORY GCPhys=00000000f0003818 pPage=Za:BAD:0000ffffe000:0000000:-0000 
03:13:22.622608 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
03:13:22.622628 Console: Machine state changed to 'Stuck'
03:13:22.622906 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
03:13:22.622906 !!
03:13:22.622907 !!         VCPU0: Guru Meditation -8 (VERR_NO_MEMORY)
03:13:22.622908 !!
03:13:22.622913 !!

Win10-Ghosts-4_VBox.log.zip

(83.32 KiB) Downloaded 261 times

Going to try the latest test build... it's been over a month of trying to get these working and I'll have to cut my losses soon.

[scottgus1]

It appears you have room to run these VMs memory-wise.

However, you're on Sonoma, which requires Virtualbox 7.0, and where Apple forces Virtualbox to run using Apple's hypervisor and no kexts. So some of Virtualbox's capabilities are limited. For example, Internal networking wasn't working at the beginning of 7.0 on Mac Ventura. Might still be limited.

Also:

4TB external SSD where the VMs live

If this means all the VMs, then while an SSD can handle more VMs than a spinning platter drive can, I'm not sure how well an SSD could handle seven VMs.

The VMs are flat-lining, meaning they stop responding inside the VM container.

How well does the setup work when only one Windows 10 VM is running, or only two, along with the Server and Kali VMs?

Another possibility, since you have an Intel Mac, try booting it to Windows 10 (no need to activate just for a test) get this host Windows 10 to not use Hyper-V (HMR3Init: Attempting fall back to NEM (Hyper-V is active)) and see how things perform under the native Virtualbox hypervisor.

[bdonnachie]

Thank you for the quick reply - very much appreciated.

The Win10 VMs aren't doing a great deal - staggered boot and then CMU Ghosts simulates web browsing and writes documents to a network share on the server. The focus of the simulation is the server so I can 'cope' with the Win10 VMs crashing as long as I can bring them back up.

Great suggestion on trying Windows - although would Linux be better perhaps? (Shame the SSD is formatted in APFS but that's quick enough to fix!).

[scottgus1]

Staggered boot is good. Make sure they aren't all updating at once.

Windows vs Linux would be convenience and familiarity mostly, though Linux doesn't have Hyper-V to fight, just be sure it doesn't have KVM or Qemu started. If you'd feel more comfortable on a Linux host, go for it. Please let us know how things go!

[granada29]

Possibly the hardware is over-subscribed when you are running 7 VMs, each with 2 CPUs allocated.

Your machine has 6 physical cores (hyperthreads don't count) so you could try running your model(s) with only 1 core allocated to each 'ghost' VM. I'm not sure how Win 10 will react to that, so it may introduce other problems.

Good luck!

[bdonnachie]

Windows vs Linux would be convenience and familiarity mostly, though Linux doesn't have Hyper-V to fight, just be sure it doesn't have KVM or Qemu started. If you'd feel more comfortable on a Linux host, go for it. Please let us know how things go!

Thanks! Beginning to wish I'd built a Linux box... I'm quietly optimistic with the latest test build (Version 7.0.13 r160260). It seems happy but I've had this sense of optimism a few times now. If it so much as wobbles before the weekend is out, it's getting flattened and turned into a Fedora box.

Your machine has 6 physical cores (hyperthreads don't count) so you could try running your model(s) with only 1 core allocated to each 'ghost' VM. I'm not sure how Win 10 will react to that, so it may introduce other problems.

Believe it or not, they used to crash a lot more often with 1-core each! 2 was on the verge of semi-stable. Everything crossed that I'm finally there...

[bdonnachie]

Believe it or not, they used to crash a lot more often with 1-core each! 2 was on the verge of semi-stable. Everything crossed that I'm finally there...

It's been a couple of days and no crashes so far. I think v7.0.13 r160260 might actually be the winning combination! I'll give it another couple of days to be absolutely sure and then restart my simulations. But any sign of a wobble and I'm installing Fedora...

Thanks again.

[bdonnachie]

It was going so well but I recently had two kernel panics - one apparently Google Drive (I forgot it was still running - doh!) and another in kernel_task.

I gave it fair warning. It's being turned into a Fedora box!

[bdonnachie]

First impressions with Fedora 39 are extremely positive. I accept that it's a fresh install and doesn't have the 'baggage' my MacOS install had but so far all VMs are extremely responsive.

A few quirks with the Apple wifi and bluetooth but I can cope with those for the sake of completing the simulations. *touches a lot of wood* so far seems to be behaving very nicely!

Thanks again for all the help.