I have tried everything that the Internet suggests:
* Installed guest additions. Caused me to only be able to choose 1280x800, previously I could choose higher resolutions but I got scrollbars.
* Ran
andVBoxManage setextradata global GUI/MaxGuestResolution any
. After doing this, many more video modes appeared in my guest, but whenever I choose one of them the only effect is that I only see the top left part of the window until I go in and out of fullsscreen, at which point the 1280x800 resolution is restoredVBoxManage.exe setextradata "<VM Name>" CustomVideoMode1 2560x1600x32
* Ran
. This seemed to have no effect whatsoever.VBoxManage controlvm <VMName> setvideomodehint 2560 1600 32
When I connect my external monitor, I have no problem setting the VM screen resolution to the native resolution of that monitor.
I have noticed that the guest VM thinks the monitor is a "Generic Non-PnP Monitor".
Running VirtualBox 4.3.10 r93012 on OSX 10.9.2. Guest is Windows Server 2012 Standard.