Hi mpack.
Yeah, I tried it and it works for me.
When I examined the ls -l of my /usr/bin/ I noticed that VBoxManage, VirtualBox, VBoxHeadless, VBoxSDL, etc... are all executable PATH variables.
So I decided to experiment and take away VirtualBox because it's actually the one that launches the GUI.
Then I made a startup script in my user's Startup Applications in Ubuntu, VBoxManage startvm MS XP, to automatically start a Guest OS (Windows), that doesn't necessarily go through a GUI or menu. Note: I've also tried making alias and creating a sudoer to launch VirtualBox, but none of them worked effectively, I was always able to make a workaround.
I also removed the CD Rom drive in my Guest OS'es because I don't want users to be able to boot from the CD by hitting F12 on startup of Guest OS.
When I was studying all these, I was only led to one conclusion, that the VirtualBox GUI, is the main security concern so to solve that problem, disable it and your security issues will fix themselves.
Finally chmod 700 VirtualBox so ONLY the owner can access the GUI again. You can launch it in Ubuntu, just go to where you put it and type ./VirtualBox. It's a good thing it's a script.
Hope this helps someone.
