4.3.x Performance regression vs 4.2.18

[billsc26]

After installing first 4.3.0 and now 4.3.2, my virtual machines just *feel* significantly slower. This is most pronounced with XP guests. It is hard to quantify "slower" so I ran SuperPI just to have a data point. This is all with XP SP3 clean installs on a Core I7 machine. VT-x and nested paging are enabled.

VirtualBox 4.3.2: 2M run - 35 seconds
VirtualBox 4.2.18: 2M run - 29 seconds

While may not seem like much of a difference, it takes SuperPI about 30 seconds to actually start on 4.3.2, and about 5 seconds on 4.2.18. All the other programs I use in the XP guests just feel slower, in some cases more than others. I did not test 4.3.0 as the I/O APIC bug made my XP guests unusable.

[noteirak]

Minimum information needed for assistance.

[michaln]

Thanks for the report, and especially posting some numbers and how you arrived at them. Most of our users unfortunately don't seem to be capable of providing any information beyond "it feels slower".

Now could you please also attach VBox.log for your test VM from 4.3.x and if you have it, also from 4.2.18?

[michaln]

Actually... please also check the CPU utilization in the guest, to make sure that the slowdown isn't caused by something else eating up CPU cycles.

[billsc26]

After more testing to capture the log files, here is what I found - and it's interesting and not what I expected. I re-installed 4.3.2 and then 4.2.18 for testing. All tests were with v1.9 of SuperPI at the 2M test.

Native host: 27 sec
4.2.18: 30 sec
4.3.2: 35 sec

The first time I ran the 4.3.2 test, I also got the 30 second time. While checking to make sure I really was running 4.3.2, I noticed that I was still using the 4.2.18 guest additions. When I installed the 4.3.2 additions and rebooted, that's when I really noticed how much slower the VM was. Checking task manager in the guest, I see "svchost" eating a lot of CPU time (one CPU @ 100% load). I waited about 30 minutes and it eventually died down to 3-5 % on a single CPU, but the SuperPI performance never recovered (!).

So it looks like it may have something to do with the guest additions, maybe?

[Verne2k]

there are definately some major performance issues with 4.3.x .

some say it has to do with the emulated network stuff https://www.virtualbox.org/ticket/12300 but im not sure it's only that. guest additions seems to be involved tho.

[michaln]

there are definately some major performance issues with 4.3.x.

The only thing that's definite is a lack of good evidence. But if you have some actual numbers and all the details, please do post them -- we're interested.

some say it has to do with the emulated network stuff https://www.virtualbox.org/ticket/12300 but im not sure it's only that. guest additions seems to be involved tho.

"Some say" that such vague noise leads nowhere. That issue does not appear to be relevant at all...

[michaln]

The first time I ran the 4.3.2 test, I also got the 30 second time. While checking to make sure I really was running 4.3.2, I noticed that I was still using the 4.2.18 guest additions. When I installed the 4.3.2 additions and rebooted, that's when I really noticed how much slower the VM was. Checking task manager in the guest, I see "svchost" eating a lot of CPU time (one CPU @ 100% load). I waited about 30 minutes and it eventually died down to 3-5 % on a single CPU, but the SuperPI performance never recovered (!).

So it looks like it may have something to do with the guest additions, maybe?

Definitely maybe. How certain are you that the busy svchost wasn't Windows Update? Unfortunately XP doesn't make it easy to tell and you'd need something like Process Explorer to see what's really going on. All I can say with certainty is that Windows Update is known to cause high CPU utilization for unreasonably long time and it has nothing to do with virtualization (but VMs are affected just as much as physical systems). I notice in the logs that the 4.2 run was restored from a saved state, while the 4.3 run was not. That could easily cause Windows Update to behave differently.

And just to double check... you're saying that VBox 4.3 with 4.2 Guest Additions was just as fast as VBox 4.2, but the performance dropped once you installed 4.3 Additions?

[Verne2k]

there are definately some major performance issues with 4.3.x.

The only thing that's definite is a lack of good evidence. But if you have some actual numbers and all the details, please do post them -- we're interested.

im sorry but i had to downgrade asap because it was so bad that it was not usable anymore, on 2 completely different hosts and guests. i will try to get some evidence in my spare time

some say it has to do with the emulated network stuff https://www.virtualbox.org/ticket/12300 but im not sure it's only that. guest additions seems to be involved tho.

"Some say" that such vague noise leads nowhere. That issue does not appear to be relevant at all...

not relevant? im sorry? that says that something that has to do with NAT emulation hogs the cpu very badly, and tbh i said "some say" but there is already an open ticket on the bugtracker so clinging to my wording is pure nonsense

[michaln]

im sorry but i had to downgrade asap because it was so bad that it was not usable anymore, on 2 completely different hosts and guests. i will try to get some evidence in my spare time

That would be helpful.

not relevant? im sorry? that says that something that has to do with NAT emulation hogs the cpu very badly, and tbh i said "some say" but there is already an open ticket on the bugtracker so clinging to my wording is pure nonsense

This is not. I don't know if you bothered reading the ticket but it clearly talks about high *host* CPU utilization and unresponsive Qt GUI. The problem in this thread is with *guest* CPU utilization and there is absolutely no mention of the VirtualBox GUI being unresponsive. Mixing up totally different issues doesn't help resolve either of them.

billsc26: I'm still interested in what you have to say!

[billsc26]

The first time I ran the 4.3.2 test, I also got the 30 second time. While checking to make sure I really was running 4.3.2, I noticed that I was still using the 4.2.18 guest additions. When I installed the 4.3.2 additions and rebooted, that's when I really noticed how much slower the VM was. Checking task manager in the guest, I see "svchost" eating a lot of CPU time (one CPU @ 100% load). I waited about 30 minutes and it eventually died down to 3-5 % on a single CPU, but the SuperPI performance never recovered (!).

So it looks like it may have something to do with the guest additions, maybe?

Definitely maybe. How certain are you that the busy svchost wasn't Windows Update? Unfortunately XP doesn't make it easy to tell and you'd need something like Process Explorer to see what's really going on. All I can say with certainty is that Windows Update is known to cause high CPU utilization for unreasonably long time and it has nothing to do with virtualization (but VMs are affected just as much as physical systems). I notice in the logs that the 4.2 run was restored from a saved state, while the 4.3 run was not. That could easily cause Windows Update to behave differently.

And just to double check... you're saying that VBox 4.3 with 4.2 Guest Additions was just as fast as VBox 4.2, but the performance dropped once you installed 4.3 Additions?

Just to make sure I didn't make a mistake, I re-ran all my tests. I switched to a different, clean XP SP3 VM with nothing but a few basic utilities installed vs. the production VM where I first noticed the issue. I also checked the task manager (in the guest) to make sure the VM was idling before running SuperPI each time. My procedure was to start with 4.2.18, test, upgrade Vbox to 4.3.2, test, upgrade the VM to 4.3.2 GAs, test, downgrade Vbox to 4.2.18, test and then downgrade the VM to 4.2.18 GAs. Notably I just reset the VM when upgrading to 4.3.2 GAs and again when downgrading to the 4.2.18 GAs (I don't shut down the VM, reboot the host, etc.).

4.2.18 Vbox, 4.2.18 GA - 28 s
4.3.2 Vbox, 4.2.18 GA - 29 s

I made a mistake and accidentally reset the VM. When it restarted, I noticed it was considerably slower, so I re-tested without making any changes.

4.3.2 VBox, 4.2.18 GA - 34 s (after VM reset)

I installed the 4.3.2 GAs (which requires a reboot)

4.3.2 VBox, 4.3.2 GA - 35 s

I shut down the host and rebooted the host. Starting up the guest again (without a guest reboot)

4.3.2 VBox, 4.3.2 GA - 29 s

So maybe my testing procedure accidentally pointed the finger at the 4.3.2 GAs when it is the VM reset that causes the problem? I seem to be able to reproduce the slowdown by simply resetting the 4.3.2 VMs, which I am not able to reproduce in 4.2.18. I hope you find this information helpful.

[michaln]

Yes, it's very helpful, although I currently don't know what to make of it

[billsc26]

Well, it would be interesting to see how the VM is different before vs. after a reset. Maybe something is not being set up or cleared corrected between then two? Maybe it is some interaction with the GAs? Obviously guessing here - but now that I know what to look for, resetting the VM does seem to have a performance impact for me on 4.3.2 at least.

[socratis]

resetting the VM does seem to have a performance impact for me on 4.3.2 at least.

Here is an idea... Windows Update runs at an interval between 17 and 22 hours (I think I read it in a Microsoft article, but I could be wrong). If you reset the machine, the last-update timer is reset as well and that would trigger a check-for-updates action (maybe?). Just a guess...

[billsc26]

When the VM is "acting up" I see a significant slowdown, even before the guest OS has (fully) booted. For example, you know when you boot XP, it initially starts up with "native" window borders and then quickly refreshes with the chosen window borders? All this while displaying the "Press ctrl+alt+del..." screen? It takes a very noticeably longer time just to refresh the window. It's like watching an artist draw the window. And when the guest VM is booted, it will show ~0% CPU load, but still act slow - so this is NOT because the WU service is stealing cycles.