Auto switching to workspace

[Xarin]

So this happens on both my laptop running Debian, and my desktop running windows 7 (using an extra workspace switcher). I will be working in one workspace and then get randomly moved to the one running my Virtual OS. I run the guest os in its own workspace so that I can use it as I would any other OS and just switch between work spaces using ctrl+alt+arrow key. This auto switching can bet a bit annoying and wondered if anyone knew why it was happening.

PS: This happens while I'm not using the computer as well if I just sit back and watch tv or something I'll see it randomly switch over for no reason at all.