Time drift is very high with VirtualBox 7.0.4

Discussions related to using VirtualBox on Mac OS X hosts.
Post Reply
granada29
Volunteer
Posts: 690
Joined: 3. Mar 2015, 07:27
Primary OS: Mac OS X other
VBox Version: OSE other
Guest OSses: Linux, macOS, Windows

Time drift is very high with VirtualBox 7.0.4

Post by granada29 »

In my Linux VMs I generally disable the clock updates sent from the host via the guest additions and install chrony in the guest to discipline the machine clock.

With VirtualBox 6.1.x this gives a guest clock drift rate within 1 or 2 parts per million of the host clock (host is also running chrony) of around 15ppm, depending on the host hardware.

Using VirtualBox 7.0.4 the guest clock drift rate increases dramatically to between 2000 and 5000 ppm. Chrony is able to manage this huge drift rate but it is a surprising result all the same.

With the clock drifting at this rate I surmise that if the Guest Additions are providing the updates then there must be significant jumps in the guest clock (either forward or backward) each time an update is applied. Possibly this contributes to some of the problems seen with Windows guests.

I am not sure if the root cause is a deficiency in Apple's hypervisor or if this a VirtualBox problem that needs addressing. I'm reluctant at this stage to post a bug report on the issue but any comments or thoughts appreciated.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Time drift is very high with VirtualBox 7.0.4

Post by fth0 »

I have a few ideas what to investigate (without having any prior knowledge of the timer system of VirtualBox in NEM on macOS):

Start a VM, let it run for ~10 minutes, shut it down from within the guest OS and provide the corresponding (zipped) VBox.log file.

On a Linux host, I'd use watch -d -n 1 'VBoxManage debugvm "VM name" info clocks' to monitor the VirtualBox timer system. You could use a macOS alternative of watch or go without it. Using the watch command has the charm of displaying the changes in place and colorized for easy watching. ;)

Edit:
Added single quotes around the VBoxManage command to protect space characters in the VM name correctly.
Last edited by fth0 on 30. Dec 2022, 01:10, edited 1 time in total.
granada29
Volunteer
Posts: 690
Joined: 3. Mar 2015, 07:27
Primary OS: Mac OS X other
VBox Version: OSE other
Guest OSses: Linux, macOS, Windows

Re: Time drift is very high with VirtualBox 7.0.4

Post by granada29 »

With GetHostTimeDisabled the command VBoxManage debugvm "VM name" info clocks does not work, as expected.

chronyc shows the following.

Code: Select all

root@debianvb:~# chronyc -n tracking
Reference ID    : 0A400119 (10.64.1.25)
Stratum         : 2
Ref time (UTC)  : Wed Dec 28 23:46:51 2022
System time     : 0.000016956 seconds fast of NTP time
Last offset     : +0.000022472 seconds
RMS offset      : 0.002142525 seconds
Frequency       : 3379.057 ppm slow
Residual freq   : +0.072 ppm
Skew            : 5.421 ppm
Root delay      : 0.000519276 seconds
Root dispersion : 0.000114022 seconds
Update interval : 8.1 seconds
Leap status     : Normal
I'll disable GetHostTimeDisabled, remove chrony, capture a new log and show the results in my next post.
Attachments
VBox.log.zip
(37.35 KiB) Downloaded 1 time
granada29
Volunteer
Posts: 690
Joined: 3. Mar 2015, 07:27
Primary OS: Mac OS X other
VBox Version: OSE other
Guest OSses: Linux, macOS, Windows

Re: Time drift is very high with VirtualBox 7.0.4

Post by granada29 »

With the VM running 'normally', i.e. clock updates via Guest Additions and no chrony I see the following message every 10 seconds or so:

Code: Select all

bryan@number9 ~ % VBoxManage debugvm Debian info clocks
Cpu Tick:      1213134160814 (0x00011a746e43ae) 3514992276Hz ticking - virtualized - native api
Cpu Tick:      1213088924682 (0x00011a71bc040a) 3514992276Hz ticking - virtualized - native api
Cpu Tick:      1213088615376 (0x00011a71b74bd0) 3514992276Hz ticking - virtualized - native api
 Virtual:       346225944147 (0x0000509caca653) 1000000000Hz ticking
VirtSync:       346225737959 (0x0000509ca980e7) ticking - catchup
          offset 195919  catch-up rate 5 %
    Real:          191692571 (0x0000000b6cff1b) 1000Hz
If the offset is in usec, then that is a significant drift being corrected several times per minute. Even if it is nano seconds it's still large. Looks to me as though VirtualBox is doing it's best to keep up with a slow hypervisor but since I know very little about such things, I have no idea what correction could me made.
Attachments
VBox.log.zip
(36.02 KiB) Downloaded 2 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Time drift is very high with VirtualBox 7.0.4

Post by fth0 »

granada29 wrote:With GetHostTimeDisabled the command VBoxManage debugvm "VM name" info clocks does not work, as expected.
Well, I would not have expected that, and I also would have expected the command to either work or not work in general, independent of the GetHostTimeDisabled setting ... I just verified that myself (on a Linux host). Did you perhaps try the command too early? (You don't have to repeat the experiment, the VBox.log files already told me that the output would not differ considerably.)

VBoxManage debugvm "VM name" info clocks displays the VirtualBox internal clocks, which are not on the guest side of things, but control the timing of the virtual hardware on the VirtualBox side (on the host). The values are given in nanoseconds (ns), which means that the clocks Virtual and VirtSync only differ in less than a millisecond (~0.2 ms). This is good to know because the "giving up catch-up attempts" messages you may have seen in other users' issues in the past comes when this difference accounts to "huge" 60 seconds. ;)

Some questions:

Does the issue from your OP depend on the GetHostTimeDisabled setting?

I asked you to run the VM for ~10 minutes. Did you run the VM for only ~8 and ~6 minutes respectively, or do the timestamps of the VBox.log files indicate lost time?

Do I understand correctly that we're looking for "only" 5 ms per second?

Differences between the Virtual and VirtSync clocks are perfectly normal. For a (very simplified) example, if the guest OS writes data to a virtual hardware, and VirtualBox needs 1 ms to write the data to the physical hardware, the VirtSync clock would be 1 ms behind and then catch-up to the Virtual clock during the next 20 milliseconds (catch-up rate 5 %). If you have another (non-macOS) host with VirtualBox installed, you could watch the clocks there for a few minutes to get a feeling for the "normal" behavior.

Edit:
Corrected the catch-up rate in the example.
Last edited by fth0 on 30. Dec 2022, 01:23, edited 2 times in total.
granada29
Volunteer
Posts: 690
Joined: 3. Mar 2015, 07:27
Primary OS: Mac OS X other
VBox Version: OSE other
Guest OSses: Linux, macOS, Windows

Re: Time drift is very high with VirtualBox 7.0.4

Post by granada29 »

fth0 wrote: Well, I would not have expected that, and I also would have expected the command to either work or not work in general, independent of the GetHostTimeDisabled setting ...
It was a repeatable issue on macOS. I currently don't have a Linux host to see what happens there.
fth0 wrote: Does the issue from your OP depend on the GetHostTimeDisabled setting?
Definitely. On VirtualBox 6.1.x with GetHostTimeDisabled active a Linux VM will see drift rates very close to the hardware rates and will sychronise to UTC with about +/- 10 usecs offset. If GetHostTimeDisabled is not active (double negatives are confusing !!) the ntp algorithms are badly affected and there is much greater error in the offset from UTC. The recommendation for running software that disciplines the system clock is to always disable the hypervisor clock updates.
fth0 wrote: I asked you to run the VM for ~10 minutes. Did you run the VM for only ~8 and ~6 minutes respectively, or do the timestamps of the VBox.log files indicate lost time?
That was my approximation to ~10minutes :)
fth0 wrote: Do I understand correctly that we're looking for "only" 5 ms per second?
5 ms/s is 5000 ppm. On real hardware, clocks will typically drift < 30 ppm and this is also the case on VirtualBox 6.1.x where the VM drift rate is very close to the hardware drift rate.

Chrony is quite forgiving with high drift rates. That is not the case with ntpd, where I doubt sychronization could be achieved at all in a VM drifting at 5000ppm.

The main issue is that in VirtualBox 7.0.x this is a regression from the behaviour seen in VirtualBox 6.x.x
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Time drift is very high with VirtualBox 7.0.4

Post by fth0 »

Even after deeply looking into the VirtualBox source code, I do not understand how the GetHostTimeDisabled setting could influence the command that reads the VirtualBox internal clocks. But I have a theory regarding your problem:

I think that the guest OS somehow uses the virtual Time Stamp Counter (TSC) of one of the virtual CPUs to advance its clock(s) (*). If VirtualBox 6.1.40 has direct access to the host CPU (e.g. via a kernel extension), it reads the TSC from the current host CPU core and uses it as the base for the TSC of the vCPU. Since VirtualBox 7.0.4 has no direct access to the host CPU, it asks the Apple hypervisor for the TSC of the vCPU (via hv_vcpu_read_msr(MSR_IA32_TSC)).

In the output of VBoxManage debugvm "VM name" info clocks, the TSC values are shown as Cpu Tick values. Note that the values are probably not equal because (physical and virtual) CPU cores are being started one after another. One can perhaps verify my theory by executing the command repeatedly and evaluating the differences between consecutive TSC values ...

(*) I'm not sure if and how the paravirtualized clocks of the VirtualBox KVM paravirtualization provider are used for this, but they are independent of the VirtualBox Guest Additions AFAIK.
granada29
Volunteer
Posts: 690
Joined: 3. Mar 2015, 07:27
Primary OS: Mac OS X other
VBox Version: OSE other
Guest OSses: Linux, macOS, Windows

Re: Time drift is very high with VirtualBox 7.0.4

Post by granada29 »

Thanks for the research @fth0

I am going to leave it for now. I think it's pretty clear that this is a consequence of the change in hypervisor. For most people this won't be an issue but hopefully improvements will be made as VirtualBox 7.x.x matures.
Practical Knowledge
Posts: 2
Joined: 4. Jan 2023, 05:56

Re: Time drift is very high with VirtualBox 7.0.4

Post by Practical Knowledge »

I can confirm that I'm having an identical issue with my system.
The Host is MacOS 12.6
The Guest is ArchLinux
The guest correctly gets the time when it boots. Thereafter, I get a pretty significant drift (~5s of drift after 60 minutes of uptime).

All of the ArchLinux Internet-based clock synchronization tools are disabled (NTP, et. al.).

Ideally, I'd like the native VirtualBox drivers to synchronize the time. Do you think that's still possible?
granada29
Volunteer
Posts: 690
Joined: 3. Mar 2015, 07:27
Primary OS: Mac OS X other
VBox Version: OSE other
Guest OSses: Linux, macOS, Windows

Re: Time drift is very high with VirtualBox 7.0.4

Post by granada29 »

@Practical Knowledge

Have you installed the VirtualBox Guest Additions in your VM? If not, then they will do a good job of keeping your VM synchronized with the host. However, any NTP synchronization software is going to be fighting with those adjustments, so it is better to leave ntpd, chronyd etc disabled.
Post Reply