More RAM for VM = much slower Windows guest boot time ?

Discussions about using Windows guests in VirtualBox.
Post Reply
madbrain
Posts: 26
Joined: 26. Mar 2011, 00:52
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: CentOS OS/2

More RAM for VM = much slower Windows guest boot time ?

Post by madbrain »

I have a host Windows machine with 32 GB of RAM. Task manager shows there are still many gigs available - ie. it doesn't get into a paging situation.
I have experimented with different RAM amounts for a Windows 10 VM.

Here are the boot times.
2GB RAM : 42 seconds
6GB RAM : 55 seconds
16GB RAM : 4 minutes and 43 seconds ! And yes, I'm certain my host wasn't paging even in this case.
I am measuring boot time from the moment I press "start" in virtualbox to when I hear the Windows startup sound.

I thought perhaps Windows 10 memory compression was causing this, so I tried disabling it using "Disable-MMagent -mc" in Powershell.
Unfortunately, the boot time results were essentially unchanged. RAM actually used varied, due to compression.

I had similar results with Windows 11 - lower RAM amount is better, though I only tried 6GB and 16GB.

With an Ubuntu 20 guest, there was no difference in boot time whether I assigned 2GB or 20GB of RAM to the VM. It was 42 seconds in both cases. This is fortunate, since I really do need the high RAM amount for development (Chrome compilation).

What's the reason for this massive difference here for the Windows guest boot time as a function of RAM assigned to the VM ?

FYI, my host is an i7-5820k running Windows 10.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: More RAM for VM = much slower Windows guest boot time ?

Post by mpack »

Please provide a VM log file. Make sure the VM is fully shut down, then right click it in the manager UI. Select "Show Log" and save "VBox.log" (no other file) to a zip file. Attach the zip here.
madbrain
Posts: 26
Joined: 26. Mar 2011, 00:52
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: CentOS OS/2

Re: More RAM for VM = much slower Windows guest boot time ?

Post by madbrain »

mpack wrote:Please provide a VM log file. Make sure the VM is fully shut down, then right click it in the manager UI. Select "Show Log" and save "VBox.log" (no other file) to a zip file. Attach the zip here.
Here you go.

I have provided two logs for my Win11 guest VM.
One with 4GB. It took 32 seconds until the sound played back. I shut it down at 41 seconds.
One with 16GB. It took 2 minutes 59 seconds until the sound played back. I shut it down soon after at 3 mins 19 seconds.

I ran these back to back. The only thing I changed was the RAM assigned in the VM properties.

I had memory compression disabled in the Windows 10 host, but I don't believe that affects the result meaningfully. A massive boot time difference remains depending on RAM assigned, and it boots much faster with less RAM.
Attachments
ram.zip
(69.13 KiB) Downloaded 16 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: More RAM for VM = much slower Windows guest boot time ?

Post by fth0 »

VirtualBox only allocates the host memory when it is accessed by the guest OS. When starting up, a Linux OS usually only accesses the memory that it needs, so that the total host memory allocation gets distributed over the first hours of runtime. A Windows OS usually accesses all the available memory during startup, so the total host memory allocation only gets distributed over the startup phase.

Although the following may seem counter-intuitive, please disable the use of large memory pages and repeat your measurement:

Code: Select all

cd "C:\Program Files\Oracle\VirtualBox"
VBoxManage modifyvm "Win 11" --largepages off
I'm really curious about what you'll find out.
madbrain
Posts: 26
Joined: 26. Mar 2011, 00:52
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: CentOS OS/2

Re: More RAM for VM = much slower Windows guest boot time ?

Post by madbrain »

fth0 wrote:VirtualBox only allocates the host memory when it is accessed by the guest OS. When starting up, a Linux OS usually only accesses the memory that it needs, so that the total host memory allocation gets distributed over the first hours of runtime. A Windows OS usually accesses all the available memory during startup, so the total host memory allocation only gets distributed over the startup phase.

Although the following may seem counter-intuitive, please disable the use of large memory pages and repeat your measurement:

Code: Select all

cd "C:\Program Files\Oracle\VirtualBox"
VBoxManage modifyvm "Win 11" --largepages off
I'm really curious about what you'll find out.
Thanks ! That made a dramatic difference.

The startup time with 4GB dropped to 27 seconds.
The startup time with 16GB dropped to 49 seconds.

What's the downside of this setting ? And why isn't it the default for Windows 10/11 guests ?
Last edited by madbrain on 13. Oct 2021, 21:43, edited 1 time in total.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: More RAM for VM = much slower Windows guest boot time ?

Post by fth0 »

madbrain wrote:What's the downside of this setting ?
On x86 CPUs, memory is organized in pages (4 kB), large pages (2 MB) and huge pages (1 GB) (and even bigger pages (512 GB)). When VirtualBox needs to allocate 1 GB (in smaller memory blocks), it can either request 512 large pages or 512 * 512 = 262144 (small) pages from the host OS. Therefore, it seems natural to assume that using large pages is more efficient.

That being said, the memory management of a modern host OS is quite a complex beast, and although I know quite a bit about it, I cannot really answer your question in general. You should simply disable the use of large pages for your VirtualBox VMs, and if it suits you, all is fine. ;)
madbrain
Posts: 26
Joined: 26. Mar 2011, 00:52
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: CentOS OS/2

Re: More RAM for VM = much slower Windows guest boot time ?

Post by madbrain »

fth0 wrote:
madbrain wrote:What's the downside of this setting ?
On x86 CPUs, memory is organized in pages (4 kB), large pages (2 MB) and huge pages (1 GB) (and even bigger pages (512 GB)). When VirtualBox needs to allocate 1 GB (in smaller memory blocks), it can either request 512 large pages or 512 * 512 = 262144 (small) pages from the host OS. Therefore, it seems natural to assume that using large pages is more efficient.

That being said, the memory management of a modern host OS is quite a complex beast, and although I know quite a bit about it, I cannot really answer your question in general. You should simply disable the use of large pages for your VirtualBox VMs, and if it suits you, all is fine. ;)
Yes, I'm a developer as well, and it makes sense to me that large pages should be faster than small pages as well, but perhaps there is a bug somewhere, or an optimization that needs to be done. Surely the Virtualbox developers have done some benchmarking when they implemented the large pages feature and decided to make it the default.

I'm wondering if others are seeing this dramatic difference too ...
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: More RAM for VM = much slower Windows guest boot time ?

Post by fth0 »

madbrain wrote:I'm wondering if others are seeing this dramatic difference too ...
As far as can be guesstimated from VirtualBox users posting here in the forums, the majority of free VirtualBox users only provide a few GB of memory to their VMs, so they usually won't notice.

When you did your initial measurements, what was the uptime of the Windows host? Does it make a considerable difference, when the Windows host has just been started before the measurements?
madbrain
Posts: 26
Joined: 26. Mar 2011, 00:52
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: CentOS OS/2

Re: More RAM for VM = much slower Windows guest boot time ?

Post by madbrain »

fth0 wrote:
madbrain wrote:I'm wondering if others are seeing this dramatic difference too ...
When you did your initial measurements, what was the uptime of the Windows host? Does it make a considerable difference, when the Windows host has just been started before the measurements?
It was measured in hours. I know there are a lot of things that load on startup. That wasn't the issue. The problem is entirely repeatable by changing the RAM assigned to the VM back and forth. Assigning more RAM always equals slower boot time, but much more so with large pages on.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: More RAM for VM = much slower Windows guest boot time ?

Post by fth0 »

One possible theory behind the slowness is that although the Windows host OS still has plenty of free memory, it already has considerable additional work to do to provide large pages to VirtualBox. AFAIK, it hasn't been proven yet ... but there have been VIrtualBox users that didn't run into the issue when starting their VM right after starting the Windows host OS.
madbrain
Posts: 26
Joined: 26. Mar 2011, 00:52
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: CentOS OS/2

Re: More RAM for VM = much slower Windows guest boot time ?

Post by madbrain »

fth0 wrote:One possible theory behind the slowness is that although the Windows host OS still has plenty of free memory, it already has considerable additional work to do to provide large pages to VirtualBox. AFAIK, it hasn't been proven yet ... but there have been VIrtualBox users that didn't run into the issue when starting their VM right after starting the Windows host OS.
I set largepages back to on, rebooted my host Windows OS, and still saw the very slow startup of the VM. So, that definitely doesn't solve the issue here.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: More RAM for VM = much slower Windows guest boot time ?

Post by fth0 »

Thanks for the feedback. Can you perform another experiment for me?

Reboot the Windows host, start the VM with large pages enabled, wait 5 minutes, and shut down the VM from within the guest OS. Reboot the Windows host, start the VM with large pages disabled, wait 5 minutes, and shut down the VM from within the guest OS. Provide VBox.log and VBox.log.1.
Post Reply