Time drift is very high with VirtualBox 7.0.4
-
- Volunteer
- Posts: 708
- Joined: 3. Mar 2015, 07:27
- Primary OS: Mac OS X other
- VBox Version: OSE other
- Guest OSses: Linux, macOS, Windows
Time drift is very high with VirtualBox 7.0.4
In my Linux VMs I generally disable the clock updates sent from the host via the guest additions and install chrony in the guest to discipline the machine clock.
With VirtualBox 6.1.x this gives a guest clock drift rate within 1 or 2 parts per million of the host clock (host is also running chrony) of around 15ppm, depending on the host hardware.
Using VirtualBox 7.0.4 the guest clock drift rate increases dramatically to between 2000 and 5000 ppm. Chrony is able to manage this huge drift rate but it is a surprising result all the same.
With the clock drifting at this rate I surmise that if the Guest Additions are providing the updates then there must be significant jumps in the guest clock (either forward or backward) each time an update is applied. Possibly this contributes to some of the problems seen with Windows guests.
I am not sure if the root cause is a deficiency in Apple's hypervisor or if this a VirtualBox problem that needs addressing. I'm reluctant at this stage to post a bug report on the issue but any comments or thoughts appreciated.
With VirtualBox 6.1.x this gives a guest clock drift rate within 1 or 2 parts per million of the host clock (host is also running chrony) of around 15ppm, depending on the host hardware.
Using VirtualBox 7.0.4 the guest clock drift rate increases dramatically to between 2000 and 5000 ppm. Chrony is able to manage this huge drift rate but it is a surprising result all the same.
With the clock drifting at this rate I surmise that if the Guest Additions are providing the updates then there must be significant jumps in the guest clock (either forward or backward) each time an update is applied. Possibly this contributes to some of the problems seen with Windows guests.
I am not sure if the root cause is a deficiency in Apple's hypervisor or if this a VirtualBox problem that needs addressing. I'm reluctant at this stage to post a bug report on the issue but any comments or thoughts appreciated.
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Time drift is very high with VirtualBox 7.0.4
I have a few ideas what to investigate (without having any prior knowledge of the timer system of VirtualBox in NEM on macOS):
Start a VM, let it run for ~10 minutes, shut it down from within the guest OS and provide the corresponding (zipped) VBox.log file.
On a Linux host, I'd use watch -d -n 1 'VBoxManage debugvm "VM name" info clocks' to monitor the VirtualBox timer system. You could use a macOS alternative of watch or go without it. Using the watch command has the charm of displaying the changes in place and colorized for easy watching.
Edit:
Added single quotes around the VBoxManage command to protect space characters in the VM name correctly.
Start a VM, let it run for ~10 minutes, shut it down from within the guest OS and provide the corresponding (zipped) VBox.log file.
On a Linux host, I'd use watch -d -n 1 'VBoxManage debugvm "VM name" info clocks' to monitor the VirtualBox timer system. You could use a macOS alternative of watch or go without it. Using the watch command has the charm of displaying the changes in place and colorized for easy watching.
Edit:
Added single quotes around the VBoxManage command to protect space characters in the VM name correctly.
Last edited by fth0 on 30. Dec 2022, 01:10, edited 1 time in total.
-
- Volunteer
- Posts: 708
- Joined: 3. Mar 2015, 07:27
- Primary OS: Mac OS X other
- VBox Version: OSE other
- Guest OSses: Linux, macOS, Windows
Re: Time drift is very high with VirtualBox 7.0.4
With GetHostTimeDisabled the command VBoxManage debugvm "VM name" info clocks does not work, as expected.
chronyc shows the following.
I'll disable GetHostTimeDisabled, remove chrony, capture a new log and show the results in my next post.
chronyc shows the following.
Code: Select all
root@debianvb:~# chronyc -n tracking
Reference ID : 0A400119 (10.64.1.25)
Stratum : 2
Ref time (UTC) : Wed Dec 28 23:46:51 2022
System time : 0.000016956 seconds fast of NTP time
Last offset : +0.000022472 seconds
RMS offset : 0.002142525 seconds
Frequency : 3379.057 ppm slow
Residual freq : +0.072 ppm
Skew : 5.421 ppm
Root delay : 0.000519276 seconds
Root dispersion : 0.000114022 seconds
Update interval : 8.1 seconds
Leap status : Normal
- Attachments
-
- VBox.log.zip
- (37.35 KiB) Downloaded 3 times
-
- Volunteer
- Posts: 708
- Joined: 3. Mar 2015, 07:27
- Primary OS: Mac OS X other
- VBox Version: OSE other
- Guest OSses: Linux, macOS, Windows
Re: Time drift is very high with VirtualBox 7.0.4
With the VM running 'normally', i.e. clock updates via Guest Additions and no chrony I see the following message every 10 seconds or so:
If the offset is in usec, then that is a significant drift being corrected several times per minute. Even if it is nano seconds it's still large. Looks to me as though VirtualBox is doing it's best to keep up with a slow hypervisor but since I know very little about such things, I have no idea what correction could me made.
Code: Select all
bryan@number9 ~ % VBoxManage debugvm Debian info clocks
Cpu Tick: 1213134160814 (0x00011a746e43ae) 3514992276Hz ticking - virtualized - native api
Cpu Tick: 1213088924682 (0x00011a71bc040a) 3514992276Hz ticking - virtualized - native api
Cpu Tick: 1213088615376 (0x00011a71b74bd0) 3514992276Hz ticking - virtualized - native api
Virtual: 346225944147 (0x0000509caca653) 1000000000Hz ticking
VirtSync: 346225737959 (0x0000509ca980e7) ticking - catchup
offset 195919 catch-up rate 5 %
Real: 191692571 (0x0000000b6cff1b) 1000Hz
- Attachments
-
- VBox.log.zip
- (36.02 KiB) Downloaded 4 times
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Time drift is very high with VirtualBox 7.0.4
Well, I would not have expected that, and I also would have expected the command to either work or not work in general, independent of the GetHostTimeDisabled setting ... I just verified that myself (on a Linux host). Did you perhaps try the command too early? (You don't have to repeat the experiment, the VBox.log files already told me that the output would not differ considerably.)granada29 wrote:With GetHostTimeDisabled the command VBoxManage debugvm "VM name" info clocks does not work, as expected.
VBoxManage debugvm "VM name" info clocks displays the VirtualBox internal clocks, which are not on the guest side of things, but control the timing of the virtual hardware on the VirtualBox side (on the host). The values are given in nanoseconds (ns), which means that the clocks Virtual and VirtSync only differ in less than a millisecond (~0.2 ms). This is good to know because the "giving up catch-up attempts" messages you may have seen in other users' issues in the past comes when this difference accounts to "huge" 60 seconds.
Some questions:
Does the issue from your OP depend on the GetHostTimeDisabled setting?
I asked you to run the VM for ~10 minutes. Did you run the VM for only ~8 and ~6 minutes respectively, or do the timestamps of the VBox.log files indicate lost time?
Do I understand correctly that we're looking for "only" 5 ms per second?
Differences between the Virtual and VirtSync clocks are perfectly normal. For a (very simplified) example, if the guest OS writes data to a virtual hardware, and VirtualBox needs 1 ms to write the data to the physical hardware, the VirtSync clock would be 1 ms behind and then catch-up to the Virtual clock during the next 20 milliseconds (catch-up rate 5 %). If you have another (non-macOS) host with VirtualBox installed, you could watch the clocks there for a few minutes to get a feeling for the "normal" behavior.
Edit:
Corrected the catch-up rate in the example.
Last edited by fth0 on 30. Dec 2022, 01:23, edited 2 times in total.
-
- Volunteer
- Posts: 708
- Joined: 3. Mar 2015, 07:27
- Primary OS: Mac OS X other
- VBox Version: OSE other
- Guest OSses: Linux, macOS, Windows
Re: Time drift is very high with VirtualBox 7.0.4
It was a repeatable issue on macOS. I currently don't have a Linux host to see what happens there.fth0 wrote: Well, I would not have expected that, and I also would have expected the command to either work or not work in general, independent of the GetHostTimeDisabled setting ...
Definitely. On VirtualBox 6.1.x with GetHostTimeDisabled active a Linux VM will see drift rates very close to the hardware rates and will sychronise to UTC with about +/- 10 usecs offset. If GetHostTimeDisabled is not active (double negatives are confusing !!) the ntp algorithms are badly affected and there is much greater error in the offset from UTC. The recommendation for running software that disciplines the system clock is to always disable the hypervisor clock updates.fth0 wrote: Does the issue from your OP depend on the GetHostTimeDisabled setting?
That was my approximation to ~10minutesfth0 wrote: I asked you to run the VM for ~10 minutes. Did you run the VM for only ~8 and ~6 minutes respectively, or do the timestamps of the VBox.log files indicate lost time?
5 ms/s is 5000 ppm. On real hardware, clocks will typically drift < 30 ppm and this is also the case on VirtualBox 6.1.x where the VM drift rate is very close to the hardware drift rate.fth0 wrote: Do I understand correctly that we're looking for "only" 5 ms per second?
Chrony is quite forgiving with high drift rates. That is not the case with ntpd, where I doubt sychronization could be achieved at all in a VM drifting at 5000ppm.
The main issue is that in VirtualBox 7.0.x this is a regression from the behaviour seen in VirtualBox 6.x.x
-
- Volunteer
- Posts: 5677
- Joined: 14. Feb 2019, 03:06
- Primary OS: Mac OS X other
- VBox Version: PUEL
- Guest OSses: Linux, Windows 10, ...
- Location: Germany
Re: Time drift is very high with VirtualBox 7.0.4
Even after deeply looking into the VirtualBox source code, I do not understand how the GetHostTimeDisabled setting could influence the command that reads the VirtualBox internal clocks. But I have a theory regarding your problem:
I think that the guest OS somehow uses the virtual Time Stamp Counter (TSC) of one of the virtual CPUs to advance its clock(s) (*). If VirtualBox 6.1.40 has direct access to the host CPU (e.g. via a kernel extension), it reads the TSC from the current host CPU core and uses it as the base for the TSC of the vCPU. Since VirtualBox 7.0.4 has no direct access to the host CPU, it asks the Apple hypervisor for the TSC of the vCPU (via hv_vcpu_read_msr(MSR_IA32_TSC)).
In the output of VBoxManage debugvm "VM name" info clocks, the TSC values are shown as Cpu Tick values. Note that the values are probably not equal because (physical and virtual) CPU cores are being started one after another. One can perhaps verify my theory by executing the command repeatedly and evaluating the differences between consecutive TSC values ...
(*) I'm not sure if and how the paravirtualized clocks of the VirtualBox KVM paravirtualization provider are used for this, but they are independent of the VirtualBox Guest Additions AFAIK.
I think that the guest OS somehow uses the virtual Time Stamp Counter (TSC) of one of the virtual CPUs to advance its clock(s) (*). If VirtualBox 6.1.40 has direct access to the host CPU (e.g. via a kernel extension), it reads the TSC from the current host CPU core and uses it as the base for the TSC of the vCPU. Since VirtualBox 7.0.4 has no direct access to the host CPU, it asks the Apple hypervisor for the TSC of the vCPU (via hv_vcpu_read_msr(MSR_IA32_TSC)).
In the output of VBoxManage debugvm "VM name" info clocks, the TSC values are shown as Cpu Tick values. Note that the values are probably not equal because (physical and virtual) CPU cores are being started one after another. One can perhaps verify my theory by executing the command repeatedly and evaluating the differences between consecutive TSC values ...
(*) I'm not sure if and how the paravirtualized clocks of the VirtualBox KVM paravirtualization provider are used for this, but they are independent of the VirtualBox Guest Additions AFAIK.
-
- Volunteer
- Posts: 708
- Joined: 3. Mar 2015, 07:27
- Primary OS: Mac OS X other
- VBox Version: OSE other
- Guest OSses: Linux, macOS, Windows
Re: Time drift is very high with VirtualBox 7.0.4
Thanks for the research @fth0
I am going to leave it for now. I think it's pretty clear that this is a consequence of the change in hypervisor. For most people this won't be an issue but hopefully improvements will be made as VirtualBox 7.x.x matures.
I am going to leave it for now. I think it's pretty clear that this is a consequence of the change in hypervisor. For most people this won't be an issue but hopefully improvements will be made as VirtualBox 7.x.x matures.
-
- Posts: 2
- Joined: 4. Jan 2023, 05:56
Re: Time drift is very high with VirtualBox 7.0.4
I can confirm that I'm having an identical issue with my system.
The Host is MacOS 12.6
The Guest is ArchLinux
The guest correctly gets the time when it boots. Thereafter, I get a pretty significant drift (~5s of drift after 60 minutes of uptime).
All of the ArchLinux Internet-based clock synchronization tools are disabled (NTP, et. al.).
Ideally, I'd like the native VirtualBox drivers to synchronize the time. Do you think that's still possible?
The Host is MacOS 12.6
The Guest is ArchLinux
The guest correctly gets the time when it boots. Thereafter, I get a pretty significant drift (~5s of drift after 60 minutes of uptime).
All of the ArchLinux Internet-based clock synchronization tools are disabled (NTP, et. al.).
Ideally, I'd like the native VirtualBox drivers to synchronize the time. Do you think that's still possible?
-
- Volunteer
- Posts: 708
- Joined: 3. Mar 2015, 07:27
- Primary OS: Mac OS X other
- VBox Version: OSE other
- Guest OSses: Linux, macOS, Windows
Re: Time drift is very high with VirtualBox 7.0.4
@Practical Knowledge
Have you installed the VirtualBox Guest Additions in your VM? If not, then they will do a good job of keeping your VM synchronized with the host. However, any NTP synchronization software is going to be fighting with those adjustments, so it is better to leave ntpd, chronyd etc disabled.
Have you installed the VirtualBox Guest Additions in your VM? If not, then they will do a good job of keeping your VM synchronized with the host. However, any NTP synchronization software is going to be fighting with those adjustments, so it is better to leave ntpd, chronyd etc disabled.