I first lowered my VRAM down to 64MB, down from 128MB, and the CPU usage dropped down to around 80% when idle.
I lowered it again to 32MB and now my CPU usage hovers around 20-25%, shooting up when something displays like the Synaptic software update notification.
If I check the processes, I still see Xorg taking a large chunk of the CPU whenever I do something and for a few seconds afterwards. I see Xorg at 50-70%, then dropping to about 9%, sometimes shooting up to 70% again.
If I simply move the cursor, I see Xorg shooting to 70% and CPU usage increasing up to 90%. If I stop moving the cursor, CPU usage usually settles down to 20-25%, although barely nothing is running.
If I continually move the cursor, System monitor stops refreshing until I stop, displaying at around 85% before dropping down.
I think this has something to do with my Ubuntu install. Seeing this, I am tempted to recreate the VM from scratch.