Uninstalled 5.0, installed 5.2 RC1, restarted OS.
Now when I start VirtualBox from mstc session (xrdp) I get empty app frame (no VirtualBox GUI inside of it) and then it freezes or crashes.
$ virtualbox --help
Oracle VM VirtualBox Manager 5.2.0_RC1
$ uname -a
Linux box 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
This was with the deb package for Xenial 64-bit.
I also installed the AMD64 run package. This installs but when I execute virtualbox from mstsc/xrdp (I start xterm and from it execute virtualbox), I get Qt WARNING's such as "Unbalanced save/restore" and "Painter not active".
Edit: later I noticed another more telling message - about 15 bit color mode not being supported. Indeed, my mstsc was 15 bit mode. I added 1 bit and now it's working. (It's kind of ridiculous that this thing can't work with 15 bits; I'd expect it to work with 256 colors).