VirtualBox VRDP performance

[velis]

A while ago I was testing VRDP on my main development computer (Windows). So I enabled it for a test Windows guest and for a test Linux guest and then accessed it from another computer on my home GbE. Everything seemed perfect. It was fast, responsive, and hardly distinguishable from working directly at the console (of those test hosts). Screen resolution 3440x1440 (20MB per uncompressed screen image)

Then I changed two variables:
My host is now Linux (Ubuntu 17.10). Graphics card is a 2D accelerator only.
Network speed to said host is 50 Mbit/s

In that host I have two VMs: Windows 10 and Linux (Ubuntu)

I again enabled VRDP thinking that 50Mbit is not that much worse than gigabit.

The problem:
And I got this beautiful Matrix movie effect. Looks great, but overall renders the screen about 2 seconds. Seems like 64x64 pixel blocks. Naturally completely useless for any scrolling.

If I connect from Windows to Windows (guest RDP server), everything is again great. But not Windows to VB VRDP for same Windows guest.
But any combination involving Linux nets me the Matrix effect.

Is this normal and to be expected?
If so, any suggestions on how to make the Linux guest more real-time? We tried VRDP, VNC, X forwarding so far.

Thanks,
Jure