64-bit Windows Server 2003 guest clock out of sync on 3.1.x

[BACON]

I hope it's appropriate for me to start a new thread here. I saw some other threads that were related to the same basic issue, but they either pertained to different configurations (ie a Windows host) or were supposedly fixed by 3.1.4.

I have a 64-bit Ubuntu Server 9.10 (Karmic Koala) host with three uniprocessor guests; two also run 64-bit Ubuntu Server 9.10 and the third runs 64-bit Windows Server 2003. All three were created under and had been running under VMware Server 1.0.x until recently when I switched to VirtualBox 3.1.2. The only changes I made to the guests in switching from VMware to VirtualBox was to uninstall VMware Tools, install Guest Additions, and on the Windows guest I had to change the system disk from SCSI to IDE to get it to boot. I still have to get around to changing all of my virtual disks from SCSI to SATA, and potentially convert them from VMDK to VDI format (where applicable, I have a number of writethrough virtual disks backed by LVM logical volumes on the host), but I'm thinking that wouldn't be the cause of my problem.

Ironically, one of the reasons I ditched VMware is because the Windows guest ran fine but VMware Tools was unable to keep the clocks in sync on the Ubuntu guests. Now, with VirtualBox, I have the opposite problem: the clocks on the Ubuntu guests are perfect but the Windows guest's clock just won't stay in sync. The first version of VirtualBox I used was 3.1.2, and with that the Windows guest's clock was running fast. I found this thread which referenced this bug which stated that running v3.1.0 of Guest Additions was a workaround for this problem. I tried that, but that just caused my clock to run very slow instead.

3.1.4 supposedly fixed that bug, though after upgrading my clock is still all over the place. What seems to be happening is this: when I first boot up the Windows guest the clock is fine. However, as soon as it experiences any significant load, then the clock starts losing time and in fact the entire guest is "stuck" running slowly. As a test, what I did was to start playing a YouTube video in the guest and gradually increase the video resolution (360p => 480p => 720p); after each "step" the processor usage would increase, and the clock would start slowing down a tiny bit (it would get maybe as much as a second or two behind), but Guest Additions was still able to periodically resync. However, after running the following batch file...

Code: Select all

@echo OFF

:LOOP

goto :LOOP

...for 20-30 seconds and then killing it the clock would start losing time at a dramatic rate (often times it seems like one second in the guest is closer to two or three seconds wall time). If I then resumed playback of that exact same YouTube video it was like it was playing back in slow motion. In fact, the entire guest seems to be in slow motion, and all this while there's little or no processor usage on the guest or the host!

So, something seems to be happening when that guest CPU hits 100% that permanently puts it into a time warp that even rebooting the guest won't fix. The other frustrating thing is that, for example, I rebooted the guest and host the other day, and after that the clock was running way fast. So, there's just no rhyme or reason to it and I just can't figure out what or where the problem is. I've tried doing a repair installation from the Windows CD and that didn't seem to help. I also tried using the HALu utility to change the guest's HAL from "ACPI Uniprocessor x64-based PC" to some other non-ACPI uniprocessor HAL, but after I rebooted it still came up with the same description. I've considered doing a complete reinstall of the virtual machine from scratch, but I don't want to go through all that work only for it to not fix anything. Is it possible that I'm having this issue because the virtual machine was migrated from VMware? Could ACPI or APIC be to blame? Any other ideas I could try? I was going to begrudgingly disable AMD's Cool 'n' Quiet in the host BIOS to see if that helped...but it turned out I had it disabled all along, so I guess processor frequency scaling can't be the culprit.

The following is the configuration of that Windows guest. By the way, I use VBoxHeadless to run all of my virtual machines.

Name: Win2003x64
Guest OS: Windows 2003 (64 bit)
UUID: 052a9fe4-def3-41de-aed2-32bded420633
Config file: /VirtualMachines/VirtualBox/Machines/Win2003x64/Win2003x64.xml
Hardware UUID: 052a9fe4-def3-41de-aed2-32bded420633
Memory size: 2048MB
VRAM size: 8MB
Number of CPUs: 1
Synthetic Cpu: off
CPUID overrides: None
Boot menu mode: message and menu
Boot Device (1): Floppy
Boot Device (2): DVD
Boot Device (3): HardDisk
Boot Device (4): Not Assigned
ACPI: on
IOAPIC: on
PAE: on
Time offset: 0 ms
Hardw. virt.ext: on
Hardw. virt.ext exclusive: on
Nested Paging: off
VT-x VPID: off
State: running (since 2010-02-22T04:18:07.866000000)
Monitor count: 1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off

Here's a section of a log file that seemed to be of use in other threads dealing with time synchronization issues:

00:00:02.565 TM: GIP - u32Mode=2 (AsyncTSC) u32UpdateHz=100
00:00:02.605 TM: cTSCTicksPerSecond=0x71c4bf8c (1 908 719 500) fTSCVirtualized=true fTSCUseRealTSC=false
00:00:02.605 TM: fMaybeUseOffsettedHostTSC=false TSCTiedToExecution=false TSCNotTiedToHalt=false

I get a ton of lines like this in the log:

95:09:18.014 TM: Giving up catch-up attempt at a 60 001 861 296 ns lag; new total: 116 880 374 084 799 ns

Thanks for the help.

[vbox4me2]

Try GA 3.0.12 or 2.2.4

[BACON]

Well, I had powered on the virtual machine and the clock stayed fine for a couple days, so I thought I'd just leave it alone and not touch it just to see what would happen. I think it went about five days or so with the clock staying perfectly in sync. It was not until I logged into the guest to make a few simple configuration changes that the guest got into its "slow" mode, and I think within five minutes after that the guest's clock was already about two minutes behind the host. I have since powered the guest off and back on (simply restarting it doesn't fix the problem) and everything has been running fine again. So, the moral seems to be that the guest is fine as long as I don't use it, which, of course, is not very useful at all.

Try GA 3.0.12 or 2.2.4

Is there an issue in particular you want me to try to isolate, or is this just a general workaround?

[vbox4me2]

A general workaround, if you don't need additional ga features 3.0.12 is good enough.

[gjl15]

Hello - I was wondering if there has been any updated information on this topic. I am running a Windows 2003 32-bit guest with VB 3.1.8 on Ubuntu 9.10. The clock on the 2003 guest stays in sync with the host if it it running on it's own. However, when I start an XP guest along with it (to do testing between the two) the clock on the 2003 guest slows significantly - even stopping for up to ~30 seconds. When I shut down the XP guest, the 2003's clock begins racing to catch up to the host, and speeds past it.

This is causing testing problems for me because I'm testing a client management software, where the server includes an Apache Tomcat web service. The client and server communicate via https, and occasionally, the communications appear to fail between the two - coincidentally when the server's time is dramatically out of sync (by ~30 minute or more). I'm wondering if the time difference is affecting agent- server communications.

I will try the 3.0.12 GAs as suggested.
Thanks
Geoff

[BACON]

Today I performed the following upgrades on my system, in this order:

• Used Aptitude to upgrade VirtualBox from 3.1.6 to 3.1.8 on the 64-bit Ubuntu Server 9.10 host
• In each guest, upgraded VirtualBox Guest Additions from 3.1.6 to 3.1.8
• Used Aptitude to install all other available updates (new kernel 2.6.31-21) on the host
• Used Aptitude to install all other available updates (new kernel 2.6.31-21) on the two Ubuntu Server 9.10 guests
• Upgraded the host from 64-bit Ubuntu Server 9.10 to 10.04 LTS using the installation CD
• Upgraded the two Ubuntu guests from 64-bit Ubuntu Server 9.10 to 10.04 LTS using the installation CD

Everything seems to be working fine after the upgrades, except my time problem is now worse than before. The two Ubuntu guests range from in sync to 10-15 seconds slow. The real problem is that before the 64-bit Windows guest's clock would stay in sync as long as I just booted it up and let it sit at the login screen. Now when I boot it up the clock stays in sync for five or ten minutes (as observed using w32tm on another Windows workstation) and then for no (apparent) reason it starts rapidly losing time. With all three guests running the Windows guest has also shown 10% processor usage on the host despite there being 0% processor usage in the guest (as observed using PsList on another Windows workstation), and I've also seen the Windows guest consume 100% of the processor on the host when it's the only guest running. At the moment the host is showing negligible processor usage for each of the three guests, so that's good, but no amount of powering on and off will get the Windows guest's clock to stay in sync, and it still seems like it's not even ticking for seconds at a time.

Here is a snippet of VBox.log towards the end of the Windows guests' boot sequence:

00:00:25.030 Guest Additions information report: additionsVersion = 0x00010004 osType = 0x00034000
00:00:25.055 Guest reported fixed hypervisor window at 0x0000000000400000 (size = 0x1000000, rc = VINF_SUCCESS)
00:00:28.776 Audio: set_record_source ars=0 als=0 (not implemented)
00:00:28.777 Audio: set_record_source ars=0 als=0 (not implemented)
00:00:29.221 Guest Log: VBoxVideo: using HGSMI
00:00:29.567 SharedFolders host service: connected, u32ClientID = 1
00:00:46.571 TM: u64DeltaPrev=-1302785666 u64PrevNanoTS=0x0000001ff0cfebd1 u64NanoTS=0x0000001fa328fd4f
00:00:46.571 TM: u64DeltaPrev=-1302786092 u64PrevNanoTS=0x0000001ff0cfed7b u64NanoTS=0x0000001fa328fd4f
00:00:46.571 TM: u64DeltaPrev=-1302786963 u64PrevNanoTS=0x0000001ff0cff0e2 u64NanoTS=0x0000001fa328fd4f
00:00:48.012 Guest Log: VBoxDisp[0]: VBVA enabled
00:00:48.012 Display::handleDisplayResize(): uScreenId = 0, pvVRAM=00007f39d7977000 w=1024 h=768 bpp=32 cbLine=0x1000
00:00:54.438 Guest Log: VBoxService.exe: 3.1.8 r61349 started. Verbose level = 0
00:01:43.711 TM: u64DeltaPrev=-1279744672 u64PrevNanoTS=0x0000002d3d3e165a u64NanoTS=0x0000002cf0f6bbba
00:01:47.461 TM: u64DeltaPrev=-151037254 u64PrevNanoTS=0x0000002dd9857e6b u64NanoTS=0x0000002dd084d925
00:13:27.292 TM: Giving up catch-up attempt at a 60 000 177 326 ns lag; new total: 60 000 177 326 ns

All three guests were running at the time, as is normal. The rest of the log after that consists entirely of "Giving up catch-up attempt" entries.

[BACON]

Just an update: with all three guests running, processor usage at 0% on the host, and the Windows guest's clock 640 seconds slow and continuing to lose time, I shut down the two Ubuntu guests. The remaining Windows guest began consuming 80-100% of the host's processor as its clock slowly began to gain time again. Here's VBox.log for the Windows guest, starting at about the time the other two guests were shut down:

00:57:13.820 TM: Giving up catch-up attempt at a 60 000 195 282 ns lag; new total: 2 040 081 695 026
00:57:27.368 Guest Log: VBOXNP: DLL loaded.
00:57:27.464 Guest Log: VBOXNP: DLL unloaded.
00:58:33.524 TM: Giving up catch-up attempt at a 60 000 152 421 ns lag; new total: 2 100 081 847 447
00:59:52.020 TM: Giving up catch-up attempt at a 60 003 557 287 ns lag; new total: 2 160 085 404 734
01:02:32.909 Guest Log: VBOXNP: DLL loaded.
01:02:32.926 Guest Log: VBOXNP: DLL unloaded.
01:14:17.238 Guest Log: VBOXNP: DLL loaded.
01:14:17.258 Guest Log: VBOXNP: DLL unloaded.
01:21:39.807 Guest Log: VBOXNP: DLL loaded.
01:21:39.823 Guest Log: VBOXNP: DLL unloaded.
01:29:17.268 Guest Log: VBOXNP: DLL loaded.
01:29:17.282 Guest Log: VBOXNP: DLL unloaded.

It took 10-15 minutes for the clock to catch up...and then it just kept going. It got up to seven seconds fast, then slowed to seven seconds slow, then 10 seconds fast, then 10 seconds slow, all the while consuming 80% of the host's processor. As I've been writing this the clock has gotten in sync and is actually staying in sync, though the 80% host processor usage stays the same. I was just about to start one of the Ubuntu guests just to see what would happen, but after checking the clock on the Windows guest one last time it's suddenly 720 seconds behind and slowly catching up again! Craziness!

Now I've started one of the Ubuntu guests, processor usage on the Windows guests is back down to 0% but it's losing time again and there's already a "giving up catch-up attempt" entry in the log. Something is definitely broken here. Either I get one guest with a (somewhat) acceptable clock and unacceptable processor usage or three guests with acceptable processor usage but one of which has an unacceptable clock.

[BACON]

After upgrading VirtualBox to 3.2.0 and the host and two Linux guests to Ubuntu Server 10.04, this problem has gotten worse, not better. Now none of my guests' clocks stay in sync. So, I opened ticket #6842.

[kbranch]

Hi,

I'm having the same kind of problem

My host hardware is: "AMD Athlon(tm) 64 X2 Dual Core Processor 4000+"
My Linux version is the latest i386 CentOS 5.5
My VBox version is 3.2.2r62298
No other hypervisors running

Guest system is CentOS 5.4

It had been working OK and now that I reboot it is creeping at perhaps 1% it's normal speed, even during the grub loading stage.

Reams and reams of this in the log:

00:06:04.661 TM: u64DeltaPrev=-324712250 u64PrevNanoTS=0x0003fc63fb5b0113 u64NanoTS=0x0003fc63e80049d9
00:06:04.661 TM: u64DeltaPrev=-324711892 u64PrevNanoTS=0x0003fc63fb5b6e71 u64NanoTS=0x0003fc63e800b89d
00:06:04.661 TM: u64DeltaPrev=-324710872 u64PrevNanoTS=0x0003fc63fb5c2e01 u64NanoTS=0x0003fc63e8017c29
00:06:04.661 TM: u64DeltaPrev=-324710870 u64PrevNanoTS=0x0003fc63fb5c650a u64NanoTS=0x0003fc63e801b334
00:06:04.661 TM: u64DeltaPrev=-324698081 u64PrevNanoTS=0x0003fc63fb5c6d25 u64NanoTS=0x0003fc63e801ed44
00:06:04.661 TM: u64DeltaPrev=-324709381 u64PrevNanoTS=0x0003fc63fb5d5b0c u64NanoTS=0x0003fc63e802af07
00:06:04.661 TM: u64DeltaPrev=-324711210 u64PrevNanoTS=0x0003fc63fb5d9ac9 u64NanoTS=0x0003fc63e802e79f
00:06:04.661 TM: u64DeltaPrev=-324711160 u64PrevNanoTS=0x0003fc63fb5e50f5 u64NanoTS=0x0003fc63e8039dfd
00:06:04.661 TM: u64DeltaPrev=-324698470 u64PrevNanoTS=0x0003fc63fb5e567a u64NanoTS=0x0003fc63e803d514
00:06:04.661 TM: u64DeltaPrev=-324693390 u64PrevNanoTS=0x0003fc63fb5e94e0 u64NanoTS=0x0003fc63e8042752