virtualbox.org

Did you notice whether or not the kernel was updated in the last few days?
IIRC 9.10 had some problems. Yes I do remember and I switched to a different kernel from the Ubuntu kernel PPA to solve all of the timing issues.
It may be that you have a previous kernel still installed. You can either press the Esc key or hold down the left shift key when booting (I don't remember exactly when they switched) and select the previous kernel and see if ti works better as a test.

OK, did a little more experimentation.

Rebooting into another kernel did not help the problem.

However, exiting Virtualbox and starting it up again did help the problem temporarily -- for about 2 or 3 hours.

I tried getting rid of my Windows gtk application (Workrave) to see if that was the problem, but it didn't make much of a difference.

I did notice that if I pump up the Virtualbox.exe process priority to "High", it drifts slightly less. This makes me think that something is competing with virtualbox causing the wall clock drift -- I just don't know what.

It also seems that the longer I run Virtualbox, the more the time drifts. I guess, at some point (probably when I start losing 1 second for each second), I'll have to reboot/start Virtualbox to "reset" the time drift.

For the time being, I'm keeping my htpdate solution.