Solaris + SMP: Performance decreases as you add processors to guest

Discussions related to using VirtualBox on Solaris hosts.
Post Reply
Forest4Trees
Posts: 9
Joined: 3. Dec 2013, 18:51

Solaris + SMP: Performance decreases as you add processors to guest

Post by Forest4Trees »

I found that with VirtualBox 5.0.x, when you have 1-2 processors assigned to a VM, performance is quite fast. But the more processors you add, the slower the guest VM becomes. I've gone up to 12 processors, which is very slow to boot and extremely laggy after logging in (even Task Manager stopped responding.)

Consider the following hardware:
  • Version: VirtualBox 5.0.22
    Host:
    • O/S: Solaris 11.2.15.4.0
      Hardware: Oracle X4-2L
      CPUs: Quantity 2 of E5-2690 v2, 10 cores, 20 threads (Total of 20 cores, 40 threads)
      RAM: 512GB
    Guest:
    • O/S: Windows 10 64-bit guest (upgraded from Win 7 64-bit)
      RAM: 32GB
      Processors: 12
      Operating System Type: Windows 10 (64-bit)
      Acceleration: VT-x/AMD-V, Nested Paging, PAE/NX, Hyper-V, Paravirtualization
While the guest VM was running with 12 processors, but the guest was otherwise idle, I ran the Solaris "pstack" and "truss" commands on the VBoxHeadless process at 5 second intervals, for 5 minutes, to analyze the stack traces of each thread. After weeding out all sleeping/waiting threads, I found the following stack traces doing a tremendous amount of ioctl() system calls against the /dev/vboxdrv device (file descriptor 0x13):
Stack Trace 1:
     ffff80ffbf51edba ioctl    (13, ffffffffc0185687, ffff80ffbe13ddc0)
     ffff80ffa385df0f SUPR3CallVMMR0Ex () + 30f
     ffff80ffa0accad3 _Z19vmR3HaltGlobal1HaltP6UVMCPUjm () + 1e3
     ffff80ffa0acd004 VMR3WaitHalted () + f4
     ffff80ffa0a62d66 EMR3ExecuteVM () + 656
     ffff80ffa0acbbcf _Z25vmR3EmulationThreadWithIdP11RTTHREADINTP6UVMCPUj () + 12f
     ffff80ffa37d8aac rtThreadMain () + 2c
     ffff80ffa38557c1 _Z18rtThreadNativeMainPv () + 51
     ffff80ffbf515ef5 _thrp_setup () + a5
     ffff80ffbf5161a0 _lwp_start ()
Stack Trace 2:
     ffff80ffbf51edba ioctl    (13, ffffffffc0185698, ffff80ffbcbceda0)
     ffff80ffa3862442 SUPSemEventWaitNoResume () + 82
     ffff80ff9fd3d440 _Z15ahciAsyncIOLoopP9PDMDEVINSP9PDMTHREAD () + 330
     ffff80ffa0a87aea _Z15pdmR3ThreadMainP11RTTHREADINTPv () + 6a
     ffff80ffa37d8aac rtThreadMain () + 2c
     ffff80ffa38557c1 _Z18rtThreadNativeMainPv () + 51
     ffff80ffbf515ef5 _thrp_setup () + a5
     ffff80ffbf5161a0 _lwp_start ()
Stack Trace 3:
     ffff80ffbf51edba ioctl    (13, ffffffffc0185698, ffff80ffbc2fcc60)
     ffff80ffa3862442 SUPSemEventWaitNoResume () + 82
     ffff80ffa0b45828 _Z29pdmR3R0CritSectEnterContendedP11PDMCRITSECTmPK15RTLOCKVALSRCPOS () + 98
     ffff80ffa0b45914 PDMCritSectEnter () + a4
     ffff80ffa0a98424 PGMR3PhysReadExternal () + 34
     ffff80ffa0a7a4e8 _Z20pdmR3DevHlp_PhysReadP9PDMDEVINSmPvm () + 78
     ffff80ff9fd0b4f4 _Z17ohciR3ThreadFrameP9PDMDEVINSP9PDMTHREAD () + 254
     ffff80ffa0a87aea _Z15pdmR3ThreadMainP11RTTHREADINTPv () + 6a
     ffff80ffa37d8aac rtThreadMain () + 2c
     ffff80ffa38557c1 _Z18rtThreadNativeMainPv () + 51
     ffff80ffbf515ef5 _thrp_setup () + a5
     ffff80ffbf5161a0 _lwp_start ()
As a side note, we have another identical Solaris 11.2 host running a Windows Server 2012 guest under VirtualBox 4.3.x, with 8 processors, and have had no performance problems at all. Last year, we tried to upgrade it to 5.0.x, but the guest VM would practially hang on startup. So, I suspect this SMP performance issue was introduced with the 5.0.x branch.
Attachments
VBox.log.gz
gzipped log of a Win 10 guest with 12 processors.
(28.66 KiB) Downloaded 110 times
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by mpack »

I'm failing to see the unexpected part of this. All guests behave this way, it has been discussed numerous times. If you add vCPUs that the guest isn't using, the guest will get slower because those vCPUs still need to have their context saved and restored, which is extra work to no benefit. The trick is to give the VM the number of cores it wants. No more, no less. Experiment.
Forest4Trees
Posts: 9
Joined: 3. Dec 2013, 18:51

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Forest4Trees »

The workloads for this type of server aren't consistent. At times, we may have 8 simultaneous CPU-intensive processes, and at other times, it's just your normal O/S services.

Why is this not a problem in VirtualBox 4.3.x, then?

So, the answer is "That's how 5.0.x works.", and "Just deal with it."?
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Perryg »

With version 5 the DEVs are working on a lot of IO issues and probably why you are seeing the difference between version 4* and 5*

I would raise a ticket at bugtracker

Be sure to provide the logs and any diagnostic information you have.
Ramshankar
Oracle Corporation
Posts: 793
Joined: 7. Jan 2008, 16:17

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Ramshankar »

Thanks for the report.

I can't, off the top of my head, remember any change that can cause severe SMP performance issues that happened between 4.3 and 5.0. Of course, there are thousands of changes being done so it's possible that something did indeed change that causes a regression in performance.

I would suggest comparing VirtualBox 4.3 with VirtualBox 5.0 on identical hardware, identical host OS versions and identical guest (use OVA import/export option to copy and use the exact same guest and guest config.) with using 6 to 7 virtual-CPUs (assuming you have two 10 core boxes, ignoring hyper-threading) and then re-post your results.

It would also help to run some long-running benchmark inside each of the guest, say compiling something for 5-10 minutes and comparing the result.
Oracle Corp.
Forest4Trees
Posts: 9
Joined: 3. Dec 2013, 18:51

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Forest4Trees »

The hardware and O/S is pretty much identical between the 4.3 and 5.0 hosts. However, the guest O/S is not the same in this case.

I'll work towards getting some more scientific benchmarks soon, but that effort will take a bit of time to put together. I'd like to have that information, before officially filing a bug.
Ramshankar
Oracle Corporation
Posts: 793
Joined: 7. Jan 2008, 16:17

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Ramshankar »

The main thing is to not over-commit the number of virtual CPUs to a guest to exceed your physical CPU core count (ignore hyper-threading for the time being). So on a 10 core box, it would be much more relevant for us to see how a single 6-7 VCPU'd guest performs rather than one with 12+ VCPUs.
Oracle Corp.
Forest4Trees
Posts: 9
Joined: 3. Dec 2013, 18:51

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Forest4Trees »

This system has 2 physical CPUs, and each CPU has 10 cores and 20 threads. That makes a total of 20 cores.
Forest4Trees
Posts: 9
Joined: 3. Dec 2013, 18:51

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Forest4Trees »

I didn't have a compiler, and the slowness is seen even in the booting of the guest. So, I used the same host (2 CPU, 10 cores each) to run the tests below.

To rule out the health of the guest, I used the developer VM (Windows 7 + IE 9) from Microsoft: https://developer.microsoft.com/en-us/m ... tools/vms/

The only changes to the VM were increasing memory to 4GB, and video memory to 64MB.

Below are the timings for how long it takes to boot the guest VM (complete when you see the start button):
  • VirtualBox 4.3.32:
    • 4 CPU: 0:24
      10 CPU: 0:30
      16 CPU: 1:06

    VirtualBox 4.3.38:
    • 4 CPU: 0:23
      10 CPU: 0:30
      16 CPU: 0:48
    VirtualBox 5.0.4:
    • 4 CPU: 0:22
      10 CPU: 1:44
      16 CPU: 3:36

    VirtualBox 5.0.22:
    • 4 CPU: 0:21
      10 CPU: 1:40
      16 CPU: 4:15
I'll post these results as well in a ticket on bugtracker.
Steffen M.
Posts: 17
Joined: 12. Sep 2013, 16:56

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Steffen M. »

I've just encountered this old thread as this problem currently seems to hit us on Solaris 11.4 SRU 27 and VirtualBox 6.1.16. Do you have the bug ID, please?
Forest4Trees
Posts: 9
Joined: 3. Dec 2013, 18:51

Re: Solaris + SMP: Performance decreases as you add processors to guest

Post by Forest4Trees »

The bug number is 15557.

https://www.virtualbox.org/ticket/15557

We gave up, and moved this server from a virtual machine to a physical host.
Post Reply