Clock stalled in illumos guest

[jimklimov]

I have a rather aged setup (maybe that's the culprit, but it can not be updated right now) with the host running OpenSolaris SXCE snv_117 and VirtualBox 3.0.12, and one of the guests running illumos (oi_151a). While no other guests show this problem, illumos sometimes starts to loop in the same few seconds for hours.
Visually the condition looks like this old bug: http://dl.dropbox.com/u/841166/archwtf/date2.png

In this case all timer-dependent programs stop working (like top, iostat, proper shutdown, etc.)
Just after boot the VM works okay, sometimes for hours, but whenever it gets any load (like trying to use "pkg update") it is likely to get into this "groundhog day" loop.
However, even though there is nothing going on in the VM, it eats up 100% of the host's CPU core. One more note: the VM has 8 virtual CPUs (host has 8 real cores).

Does this ring the bell for anyone?

I have seen similar bug reports, even still-opened 3-year-old bugs, but they mostly discuss linux or windows guests, and in particular tuning the timer source for linux. Any pointers if this can be done for illumos?

Thanks,
//Jim

[jimklimov]

For what it's worth, I scaled the openindiana (illumos) VM down to 1 CPU, disabled NTP and VirtualBoxGuest services and rebooted. For the past few days the clock-stall problem has not reemerged...

However, the VM clock is currently one hour off (minutes are correct), possibly due to different definitions of time zone data on host and guest. Maybe that's what puzzled NTP and/or VBGuest trying to adjust the clock?

[michaln]

Is the VM configured to run the emulated CMOS clock in UTC or local time zone? Do the guest settings match?

[jimklimov]

That used to differ indeed: the VM process was started with local time (Europe/Moscow) which recently changed to GMT+4, and guest's /etc/rtc_config defined that "HW clock" differently as GMT+3:

Code: Select all

zone_info=Europe/Moscow
zone_lag=-10800

I updated the guest's timezone files (/usr/share/zoneinfo) like I did on the host - that did not help.

I modified /etc/rtc_config to UTC "HW Clock" zone:

Code: Select all

zone_info=UTC
zone_lag=0

while "cosmetic" default timezone for processes remained (processed according to zoneinfo definitions) in /etc/TIMEZONE:

Code: Select all

TZ=Europe/Moscow

After doing these changes I reenabled "ntp" and after several hours "vboxguest" services yesterday and rebooted the VM, no problem so far (the VM process's host time zone remains at local time for now, probably I'll have a clock jump when I change that to UTC as well). Likewise, there was no problem (except some clock drift) while both services were kept disabled.

I think the clock-stall problem was some conflict between ntp and vboxguest trying to set the clock according to different assumptions...