For some of those struggling, I discovered that Microsoft's Lync/Communicator really screws the clipboard up. If you get rid of Lync, the clipboard again starts working both ways again.
(BTW, it seems that Lync screws up the mouse, too. The problem is not VirtualBox specific. The problem is that clicking the mouse works only randomly. Again, getting out of Lync fixes the problem.)