Have the same problem here, with a freshly installed VBox 4.0.4. Occurs when I switch to the running VM window, only under the condition that the cursor is over the desktop area of the VM (i.e. doesn't happen when the cursor is over a VBox toolbar). In fullscreen mode, it doesn't happen when switching between virtual desktops on my host, but does happen when switching between windows on the same virtual desktop.
Auto-capture checkbox is, of course, unchecked, and the GUI/Input/AutoCapture option in VirtualBox.xml is accordingly set to false.
Host is Slackware 13.1, x86. DE is Xfce.
This is very irritating - I'd be grateful for any help.