[Fixed] Dev snapshot 124007: segfaults/SIGSEGV on VRDP connection

[TheBloke]

Dev snapshot 124007/5.2.97, with extpack (5.2.97r124007) installed: guest seg faults whenever an VRDP connection is made.

I have confirmed this problem on both Solaris 11.3 and Mac OS 10.13.6 hosts. Choosing to report for Mac OS as I have a full exception/stack trace log.

This issue may well be known, and I realise Dev snapshots are not intended to be stable. However I thought I'd report it in case it's not yet known, as I had hoped to switch to Dev 124007 because it fixes another bug I am experiencing in 5.2.16 stable: Windows HyperV guests hang with cpus > 8. That bug is fixed in 124007 so I had hoped to upgrade from 5.2.16, but the inability to use VRDP is a deal breaker for me as I run VBox in a headless environment and need working VRDP.

I tried disabling all features I don't need, but this did not help:

Code: Select all

VBoxManage modifyvm Win10-test --vrdeproperty Client/DisableUSB=1
VBoxManage modifyvm Win10-test --vrdeproperty Client/DisableAudio=1
VBoxManage modifyvm Win10-test --vrdeproperty Client/DisableClipboard=1
VBoxManage modifyvm Win10-test --vrdeproperty Client/DisableUpstreamAudio=1

I have tested RDPing from two different clients: Microsoft Remote Desktop Client on Mac OS, and the default Windows 10 RDP client. Both result in the same crash.

Attached is:
1. A VBox log of a Win 10 guest that crashed after a VRDP connection was opened. No errors are logged when the crash occurs, but perhaps the preceding messages are of some use.
2. Mac OS crash/exception log for VirtualBoxVM, occuring just after a VRDP connection was made. I ran several tests that always resulted in a crash, but have so far only seen this crash log notification once.

Thanks in advance.

[socratis]

Confirming with both VirtualBox Dev. Builds r124007 and r124160, on an OSX 10.11.6 host, connecting with Microsoft's RDP 8.0.43 (Build 27325), on the localhost (127.0.0.1:3389), to a Win10-64, 1607 (14393.1480) guest.

Crash of the VM doesn't happen on the initiation of the RDP connection, but it will happen eventually; the VM will abort. No crash in the VirtualBox process, just the VM goes "Aborted".

I can get it to abort reproducibly if I simply close the MS RDP client. And it shows in the log. The last lines of the VBox.log:

00:00:42.772473 VMMDev: Guest Log: Display with ID=0 already enabled
00:04:39.283817 RTC: period=0x10 (16) 2048 Hz
00:04:40.389784 RTC: period=0x200 (512) 64 Hz
00:04:40.420785 RTC: period=0x10 (16) 2048 Hz
00:04:40.436294 RTC: period=0x200 (512) 64 Hz
00:06:38.548843 VRDP: Received the Disconnect Request packet.
00:06:38.548843 VRDP: Connection closed: 1
00:06:38.548843 VBVA: VRDP acceleration has been disabled.

The end...

[TheBloke]

I just tested Mac OS version 124235 with extpack 124237 and the issue remains there.

@socratis found that the crash was not immediate on initiation of RDP connection. For me, it is: the crash always happens within about 0.5 seconds of opening an RDP connection to the VM, such that I never see an RDP window; the RDP client immediately declares that the connection has been closed, and I usually get a Mac OS crash handler report (for VirtualBoxVM) pop up at exactly the same time. Monitoring the VM process with top -s 0 confirmed that the process disappears almost immediately after I connect via RDP.

So unlike socratis, I don't get as far as being able to disconnect RDP myself.

When I initiate an RDP connection I always see these lines added to VBox.log before the VM crashes:

Code: Select all

00:00:53.889385 VRDP: New connection:
00:00:53.889526 VRDP: Connection opened (IPv4): 0
00:00:53.890110 VRDP: Negotiating security method with the client.
00:00:53.891254 VRDP: failed to access the server certificate file:
00:00:53.891352 VRDP: Connection closed: 0
00:00:53.910542 VRDP: New connection:
00:00:53.910659 VRDP: Connection opened (IPv4): 1
00:00:53.911224 VRDP: Negotiating security method with the client.
00:00:53.913795 VRDP: Methods 0x0000001b
00:00:53.913833 VRDP: Channel: [rdpdr] [1004]. Accepted.
00:00:53.913856 VRDP: Channel: [rdpsnd] [1005]. Accepted.
00:00:53.913878 VRDP: Channel: [cliprdr] [1006]. Accepted.
00:00:53.913900 VRDP: Channel: [drdynvc] [1007]. Accepted.
00:00:53.913921 VRDP: Unsupported SEC_TAG: 0xC006/8. Skipping.
00:00:53.913942 VRDP: Unsupported SEC_TAG: 0xC00A/8. Skipping.
00:00:53.916241 VRDP: Client seems to be MSFT.
00:00:53.916282 VRDP: Logon: Marvin.local (192.168.0.20) build 66653. User: [tomj] Domain: [] Screen: 0
00:00:53.917001 AUTH: User: [tomj]. Domain: []. Authentication type: [Null]
00:00:53.917025 AUTH: Access granted.
00:00:53.920358 VRDP: feature: Client/DisableAudio
00:00:53.920775 VRDP: feature: Client/DisableUSB
00:00:53.921120 VRDP: feature: Client/DisableClipboard
00:00:53.922071 VBVA: VRDP acceleration has been requested

Looking at VBox.Log for some VMs running on 5.2.16 with no VRDP crash problems suggests that the log lines that would normally follow "VRDP acceleration has been requested" are:

Code: Select all

00:00:09.362500 VMMDev: SetVideoModeHint: Got a video mode hint (1920x1200x32)@(0x0),(1;0) at 0
00:00:09.362562 VRDP: SunFlsh disabled.
00:00:09.362604 VRDP: Supported video redirection channel [TSMF]
00:00:09.371138 VRDP: Input[2] enabled: 3
00:00:09.371180 VRDP: INPUT created for 2
00:00:09.392160 VRDP: INPUT for 2: flags 0x0, ver 0x20000, contacts 10
00:00:09.392203 VRDP: INPUT closing for 2
00:00:09.394973 VRDP: Input[2] closed: 3

[socratis]

Funny thing? I tried with 5.2.16, and I get a crash as well! I definitely remember being able to RDP in a VM with no issues, so I'll try to downgrade VirtualBox and see what's going on here...

[socratis]

Even more interesting findings... Only Win10 aborts! All my other VMs are working just fine. From OSX, Ubuntu*, Fedora*, Win* (except Win10), DOS*, etc. It's the Win10 VM that aborts. Only! With the latest dev. builds as well as with 5.2.16.

Investigating...

[klaus]

Socratis' findings (which he narrowed down to affecting win8/10 guests with 3d enabled) lead into a different direction than the original report (which has 3d disabled), so they are most likely about different issues. Too bad that the original report wasn't reproducible yet.

[TheBloke]

Yes when socratis said he could find issues on 5.2.16 I thought it must be a different issue.

I have done some more digging and the issue appears to be related to specific VM settings. EDIT: It's Audio disabled = crash; see next post.

I did a complete uninstall/reinstall on my Mac OS workstation, including deleting my test VMs. I then re-created two test VMs (one Windows, one Ubuntu) and I could RDP into them just fine.

Then I tried installing 124236 on Solaris and tested an existing VM (that works fine in 5.2.16), and it crashed on RDP initiation just as before.

So then I copied that VM definition from my Solaris 11.3 server to the Mac OS host, imported it and tested it on Mac OS, and it again crashed there. So on Mac OS I now have a couple of test VMs that work fine with incoming RDP (created from scratch through the UI just now), and one that always crashes (copied from my Solaris server and then imported on Mac OS, originally created with VBoxManage). I made a couple of hand edits to the .vbox file before importing on Mac OS, for example to change the name of the bridged network interface.

Attached is that .vbox file: a VM definition that will crash on RDP connection initiation on both Solaris 11.3 and Mac OS, tested with versions 124236 and 124235 respectively, both using extpack 124237.

I will now try changing some of the settings in this vbox file to see if I can make it work and work out which setting(s) are making the difference between crash or not.

Thanks again.

[TheBloke]

I've got it! To reproduce the crash, disable Audio in the VM.

This config does NOT crash on incoming RDP connection:

Code: Select all

<AudioAdapter codec="AD1980" driver="CoreAudio" enabled="true" enabledIn="false"/>

This config DOES crash on incoming RDP connection (in latest dev builds, but not in 5.2.16):

Code: Select all

<AudioAdapter codec="AD1980" driver="CoreAudio" enabled="false"/>

I have confirmed this is the case also on Solaris 11.3, where I got my existing VM to work fine with VRDP in 124236 by running the following command:

Code: Select all

VBoxManage modifyvm OpenSuse-A --audio null

I hope that helps it to be reproduced and fixed, when you have time. Although it is not that urgent for me at least, because it's no problem for me to simply turn on Audio in all my VMs, now that I know it will fix this issue.

Thanks again for your help.

PS. Has audio output through VRDP chnaged in the latest dev builds, is that why this issue occurred? I always had audio disabled in my VMs before, because I always access through VRDP and I thought audio only worked for a local system. But I now realise audio will work through VRDP when it's enabled? At least on my OpenSUSE VM it does. That's a useful feature.

[socratis]

I've got it! To reproduce the crash, disable Audio in the VM.

Excellent detective work! Kudos!

Confirmed with a variety of clients and sound card options. Host OSX 10.11.6, VirtualBox 5.2.97 r124235. Disabling audio aborts the VM when an RDP connection is initiated.

Abort does not occur with 5.2.16 r123759.

[sunlover]

Dev snapshot 124007/5.2.97, with extpack (5.2.97r124007) installed: guest seg faults whenever an VRDP connection is made.

Thanks. Reproduced and fixed. The crash happen if audio is disabled for the VM. As a temporary workaround you can enable audio in the VM settings and use 'Null Audio Driver'.

[TheBloke]

Thanks a lot for investigating and getting it fixed. And thanks socratis for your help and kind words!

[socratis]

Funny thing? Your post actually fixed TWO bugs in RDP! That's literally the incarnation of the phrase "Two birds with one stone"!

I wish all posts were like that, we'd soon have "Zarro boogs found"...

Marking as [Fixed], hopefully in the next Dev. Snapshot (> r124236).