I made this a ticket last year (#7414) but was blown off by the support person who didn't think it was a real problem. (To be fair, I under-reported the memory footprint by a factor of one thousand.) So I'm trying here.
Am running vbox 4.0.8 on WinXP SP3. Fred Vorck's W2K is the guest. When I first start, vboxservice takes about 23MB of memory, according to Sysinternals TaskManager. But now, after a few days, vboxservice is now at 289MB. It grows without bounds, requiring a reboot every now and then to reset it.
In the bug tracker forum, one other person reported the same problem. The support person, Frank, told me how to disable some aspects of vboxservice in the Windows Registry. But I never did figure that out. (Couldn't find instances of vboxservice in regedit.)
There was one version of VirtualBox (3.1.4, I think it was) where vboxservice did not grow. But since then, it just keeps growing.
Does anyone else have this problem? Does anyone have an easy solution?