4.3.0 massive DPC latency

Discussions related to using VirtualBox on Windows hosts.
frank
Oracle Corporation
Posts: 3362
Joined: 7. Jun 2007, 09:11
Primary OS: Debian Sid
VBox Version: PUEL
Guest OSses: Linux, Windows
Location: Dresden, Germany
Contact:

Re: 4.3.0 massive DPC latency

Post by frank »

Another 4.3.3 test build can be found here. I would appreciate any statement if the DPC latency remains unchanged, got better or worse (the latter is unlikely). Thank you!
Allen_ASU
Posts: 11
Joined: 7. Nov 2013, 19:38

Re: 4.3.0 massive DPC latency

Post by Allen_ASU »

Installed 4.3.3 and it seems to be improved. LatencyMon still shows high latency, but music rarely skips now. Here is some details I notice in regards to this update.

1. General VM performance is much improved over 4.3.2
2. Booting of VMs is still verrry slow. During this time, LatencyMon shows off the chart latency and music skips. Once the VM gets to a logon screen (Windows) and the guest tools load, latency comes back down(still high, but not pegged on the monitor) and music will stop skipping.
3. VMs without guest tools installed are slow, DPC latency is very high, and music skips on the host.
4. Reducing the number of vCPUs helps reduce latency.
5. Turning off VT-x also reduces latency.
6. Host CPU usage is generally higher in 4.3.x over 4.2.18.

If there is anything else to look for, let me know. I can also post logs.
michaln
Oracle Corporation
Posts: 2973
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all
Contact:

Re: 4.3.0 massive DPC latency

Post by michaln »

Allen_ASU wrote:Installed 4.3.3 and it seems to be improved. LatencyMon still shows high latency, but music rarely skips now. Here is some details I notice in regards to this update
Please describe the host system briefly. What OS, CPU...? How exactly do you measure the DPC latency?
1. General VM performance is much improved over 4.3.2
Please quantify.
2. Booting of VMs is still verrry slow.
Please quantify.
During this time, LatencyMon shows off the chart latency and music skips. Once the VM gets to a logon screen (Windows) and the guest tools load, latency comes back down(still high, but not pegged on the monitor) and music will stop skipping.
So you're saying that the latest 4.3.3 build still has high DPC latency, much worse than 4.2.18?
3. VMs without guest tools installed are slow, DPC latency is very high, and music skips on the host.
Whether the Guest Additions are installed or not has zero direct impact on DPC latency. It may be that overall CPU utilization is lower with GAs and that indirectly reduces the incidence of DPC latency spikes, but even that is questionable.
4. Reducing the number of vCPUs helps reduce latency.
That, to be honest, makes zero sense, except again as a side effect of reduced overall CPU utilization.
5. Turning off VT-x also reduces latency.
This however makes perfect sense. Not using VT-x completely changes everything.
6. Host CPU usage is generally higher in 4.3.x over 4.2.18.
Please quantify.

Now just in case it's not obvious... the DPC latency spikes are directly related to how much CPU time the guest uses. With a more or less idle guest, there will rarely be any spikes no matter how the scheduling is handled by VirtualBox. When the guest is busy (such as when booting or performing lengthy calculations) is when the DPC latency may spike when the internal scheduling too aggressively executes guest code.
Allen_ASU
Posts: 11
Joined: 7. Nov 2013, 19:38

Re: 4.3.0 massive DPC latency

Post by Allen_ASU »

michaln wrote:
Allen_ASU wrote:Installed 4.3.3 and it seems to be improved. LatencyMon still shows high latency, but music rarely skips now. Here is some details I notice in regards to this update
Please describe the host system briefly. What OS, CPU...? How exactly do you measure the DPC latency?

Dell Precision T3600
Intel Xeon E5-2665
32GB DDR3
256GB SSD, RAID 0 7200RPM for VM's
Win8.1 x64

DPC latency monitoring is done with LatencyMon.

I originally suspected I had a faulty driver. I didn't figure out that it was virtualbox until my colleague with a similarly spec'ed T3600 did not have audio issues with 4.2.18 installed. Rolling back to 4.2.18 fixed the audio problems.


1. General VM performance is much improved over 4.3.2
Please quantify.

Just a seat-of-my-pants generalization.
2. Booting of VMs is still verrry slow.
Please quantify.

4.8.18 would boot a Win7 VM to the CTRL+ALT+DEL screen in less than a minute consistently. 4.3.2 would take up to 3 minutes to complete while the host system suffers from high latency issues such as lagged mouse, stutters in audio, etc. This doesn't happen all the time, but it is reproducible.
During this time, LatencyMon shows off the chart latency and music skips. Once the VM gets to a logon screen (Windows) and the guest tools load, latency comes back down(still high, but not pegged on the monitor) and music will stop skipping.
So you're saying that the latest 4.3.3 build still has high DPC latency, much worse than 4.2.18?

4.3.3 is better than 4.3.2 in this regard, but not compared to 4.2.18.
3. VMs without guest tools installed are slow, DPC latency is very high, and music skips on the host.
Whether the Guest Additions are installed or not has zero direct impact on DPC latency. It may be that overall CPU utilization is lower with GAs and that indirectly reduces the incidence of DPC latency spikes, but even that is questionable.

This is just an observation from normal usage. I generally have to build VM's to pull images for our SCCM site. Some of my work with VM's is without GA installed. With 4.3.x, guests without GA installed VM performance is generally terrible, host latency is consistency high, and the related latency issues are present at all times. VM performance is another seat-of-the-pants judgement compared to 4.2.18, aside from the obvious host issues.
4. Reducing the number of vCPUs helps reduce latency.
That, to be honest, makes zero sense, except again as a side effect of reduced overall CPU utilization.

This is just an observation. Another seat-of-the-pants judgement. It, however, does make sense if there is a bug in the VT-x code. The more vCPUs assigned to a guest the more chances the guest will call for bugged interrupt requests.
5. Turning off VT-x also reduces latency.
This however makes perfect sense. Not using VT-x completely changes everything.

Agreed. However, if VT-x enabled causes DPC spikes, wouldn't be logical to assume that there is a regression in the VT-x code? VT-x is suppose to increase performance of a host and its guests, not degrade it.
6. Host CPU usage is generally higher in 4.3.x over 4.2.18.
Please quantify.

Just by looking at CPU usage in task manager as VMs and the host sit idle. 4.2.18 would idle around 5-10% CPU usage. DPC latency shows fine in LatencyMon. 4.3.x (4.3.3 included) shows a consistent 20-25% usage while idle with DPC spikes occurring at a 2-3 sec interval.

Now just in case it's not obvious... the DPC latency spikes are directly related to how much CPU time the guest uses. With a more or less idle guest, there will rarely be any spikes no matter how the scheduling is handled by VirtualBox. When the guest is busy (such as when booting or performing lengthy calculations) is when the DPC latency may spike when the internal scheduling too aggressively executes guest code.
Agreed for the most part. Simply booting a VM should not cause issues on the host. The windows kernel should be able to handle the scheduling appropriately.
I realize that most of my information is subjective. If there is any useful quantitative information I can provide, please let me know what that is and I'd be happy to provide it.
AndrewG
Posts: 7
Joined: 24. Oct 2013, 10:25

Re: 4.3.0 massive DPC latency

Post by AndrewG »

I can post info and results with that new version later.

Originally i was watching a videos with media player classic(windows) and hearing the massive audio stutter. To investigate furter i loaded dpclat.exe to get actual numbers, graphs, etc of the latency. If you google it, its first hit, avail at thesycon.de.
michaln
Oracle Corporation
Posts: 2973
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all
Contact:

Re: 4.3.0 massive DPC latency

Post by michaln »

Allen_ASU wrote:Agreed for the most part. Simply booting a VM should not cause issues on the host. The windows kernel should be able to handle the scheduling appropriately.
Yes, booting a VM is expected to take up quite a bit of the host's CPU and I/O bandwidth, but it shouldn't cause the DPC latency spikes. Unfortunately given the design of VT-x, playing nice with the host scheduler is a lot harder than it ought to be :)
I realize that most of my information is subjective. If there is any useful quantitative information I can provide, please let me know what that is and I'd be happy to provide it.
For the moment it'd be interesting to know your host setup and how you determine the DPC latency, and how 4.2.18 compares with the latest 4.2.3 test build on your host.
Allen_ASU
Posts: 11
Joined: 7. Nov 2013, 19:38

Re: 4.3.0 massive DPC latency

Post by Allen_ASU »

michaln wrote:
Allen_ASU wrote:Agreed for the most part. Simply booting a VM should not cause issues on the host. The windows kernel should be able to handle the scheduling appropriately.
Yes, booting a VM is expected to take up quite a bit of the host's CPU and I/O bandwidth, but it shouldn't cause the DPC latency spikes. Unfortunately given the design of VT-x, playing nice with the host scheduler is a lot harder than it ought to be :)
I realize that most of my information is subjective. If there is any useful quantitative information I can provide, please let me know what that is and I'd be happy to provide it.
For the moment it'd be interesting to know your host setup and how you determine the DPC latency, and how 4.2.18 compares with the latest 4.2.3 test build on your host.

Absolutely. LatencyMon includes a reporting option. I'll attach it here. This one is on 4.3.3 with 2 VM's sitting idle in the background while I type this out on the host machine. Nothing processor intensive is running in the background on the host aside from the VMs. CPU is utilized around 20% in the host task manager.
4.3.3idle.txt
4.3.3idle
(14.72 KiB) Downloaded 23 times
Here is an attachment with logging as a bring up another VM. As you can see, average latency is still very good however the highest measured latency looks bad. When the spikes occur, the host has noticeable issues. ie, mouse jumps, audio stutters, clicks, etc.
4.3.3boot.txt
4.3.3 VM boot
(14.71 KiB) Downloaded 11 times
I'll work on installing 4.2.18 and do the same test with the same VMs.
Allen_ASU
Posts: 11
Joined: 7. Nov 2013, 19:38

Re: 4.3.0 massive DPC latency

Post by Allen_ASU »

AndrewG wrote:I can post info and results with that new version later.

Originally i was watching a videos with media player classic(windows) and hearing the massive audio stutter. To investigate furter i loaded dpclat.exe to get actual numbers, graphs, etc of the latency. If you google it, its first hit, avail at thesycon.de.

Be careful with this monitor. It can produce incorrect results on Win8.x systems due to some kernel timer changes made.
LatencyMon does not have this issue if you are on a Win8.x system.
michaln
Oracle Corporation
Posts: 2973
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all
Contact:

Re: 4.3.0 massive DPC latency

Post by michaln »

Thanks for the logs. As a sanity check, how does the 4.3.3 test build compare to 4.3.2?
Allen_ASU
Posts: 11
Joined: 7. Nov 2013, 19:38

Re: 4.3.0 massive DPC latency

Post by Allen_ASU »

Next is 4.2.18...
Attachments
4.3.2boot.txt
(14.97 KiB) Downloaded 10 times
4.3.2idle.txt
(14.7 KiB) Downloaded 10 times
Allen_ASU
Posts: 11
Joined: 7. Nov 2013, 19:38

Re: 4.3.0 massive DPC latency

Post by Allen_ASU »

And here is 4.1.18.

Much better latency on boot. :)
Attachments
4.2.18idle.txt
(14.68 KiB) Downloaded 8 times
4.2.18boot.txt
(14.75 KiB) Downloaded 11 times
Jonas H
Posts: 25
Joined: 9. Nov 2012, 14:49

Re: 4.3.0 massive DPC latency

Post by Jonas H »

Hi,

I have also experienced this problems after upgrading my host to Win 8.1 64-bit (guest WinSrv 2012 64-bit) and then VB 4.3.0 as the network functionality failed in 4.2.x. I've upgraded to 4.3.3 as described above and it works much better now. I have also used the LatencyMon with 4.3.2 and then after upgrading to 4.3.3, and then finally when also upgrading the Guest Additions to 4.3.3, please see the attached files (the first 3 here, the rest in the next message)

When do you think that 4.3.3 will be released officially?

Thanks for the quick response

Brgds

Jonas
Attachments
4.3.3.boot.txt
(8.65 KiB) Downloaded 9 times
4.3.2.idle.4.min.after.boot.txt
(8.78 KiB) Downloaded 10 times
4.3.2.boot.txt
(8.5 KiB) Downloaded 9 times
Last edited by Jonas H on 8. Nov 2013, 11:29, edited 1 time in total.
Jonas H
Posts: 25
Joined: 9. Nov 2012, 14:49

Re: 4.3.0 massive DPC latency

Post by Jonas H »

Here comes the rest of the files.

Brgds

Jonas
Attachments
4.3.3.idle.4.min.after.boot.Guest.Additions.4.3.3.txt
(8.65 KiB) Downloaded 8 times
4.3.3.boot.Guest.Additions.4.3.3.txt
(8.42 KiB) Downloaded 8 times
4.3.3.idle.4.min.after.boot.txt
(8.4 KiB) Downloaded 6 times
michaln
Oracle Corporation
Posts: 2973
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all
Contact:

Re: 4.3.0 massive DPC latency

Post by michaln »

Allen_ASU wrote:And here is 4.1.18.
Confused here... I'm guessing the comments are wrong but the names of the log files correctly indicate the VBox version, i.e. 4.3.2 (when you say 4.2.18) and 4.2.18 (when you say 4.1.18)?
michaln
Oracle Corporation
Posts: 2973
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all
Contact:

Re: 4.3.0 massive DPC latency

Post by michaln »

Okay, in the logs from Jonas H there's clearly a big improvement between 4.3.2 and the 4.3.3 test build. Both the maximum values and the average latencies are much lower.

With the logs from Allen_ASU, I see an improvement in the averages but not the maximums. I can't really tell which is "better".

Now here's the problem. On my machine, I can get a maximum latency of 8.5ms and a report that "your system appears to be having trouble handling real-time audio and other tasks" just by building VirtualBox, while VirtualBox itself isn't even running. Similarly on the first try of starting VirtualBox, I got a latency peak of 2.8ms, before any VM was started. When I started a Windows 7 VM with 2 VCPUs, the latency spiked at 4.5ms, which is bad but not nearly as bad as the latency incurred when building source code. The averages OTOH stayed low all the time.

The problem with the latency checking tools is that they don't (or can't) say what is responsible for the high latencies. When I was running VBox, I can't even say for sure if the DPC latency spike was caused by VBox or by whatever caused an even higher spike when building code.

I'll add that Allen_ASU's machine has much worse average latencies than Jonas H's despite being (admittedly an older/slower) Xeon.

So the real question is, does the 4.2.3 test build still cause music etc. to drop out where 4.2 did not?
Post Reply