I had a somewhat similar problem with WinXP guest on Xubuntu 8.04.3 host with dual-core processor and 2 GB RAM; in my case the guest did not totally freeze but it quit responding at all to keyboard and mouse actions after several seconds running a program that did large amounts of (virtual) disk i/o in a tight loop. Going through the VB manual for ideas I came across section 11.2.3 (troubleshooting) that suggested possibly the much faster response to IRQs could be causing it. I used the suggested command:
- Code: Select all Expand viewCollapse view
VBoxManage setextradata <VM Name> "VBoxInternal/Devices/piix3ide/0/Config/IRQDelay" 1
to delay IRQ response by one millisecond, and it appears to have solved my problem completely. The title for Section 11.2.3 makes it appear to only apply to Win2K guests -- but it may solve your problem also. The delay can be increased by as much as 5 ms, but it will definitely slow down the guest actions. In my case the slowdown was minimal, and the ability to retain control of the guest is well worth it.
Note that my guest itself kept running normally; it just lost touch with the host keyboard and mouse, making it impossible to get out of fullscreen until the non-responsive program completed its actions and allowed the guest to recover.