Serial port (TCP) relay traffic hangs Windows 10 guest

gtoombs · Post by **gtoombs** » 1. Mar 2024, 17:46

I have searched the forums, and the only things that seem vaguely (but not exactly) related are

Adding COM port to XP guest crashes the guest - different guest, and seems like a different reproduction sequence.
Ticket 18632 Guru meditation with SW virtualization and serial port (link to virtualbox domain banned, which is very funny)

Following that post, I have confirmed that "AMD-V" is enabled in the BIOS.

The setup is

VirtualBox 7.0.14 r161095
Windows 11 host
Windows 10 guest
Two serial ports enabled -
- COM1, IRQ 4, I/O port 0x3F8, TCP, server mode (do not connect to existing socket)
- COM2, IRQ 3, I/O port 0x2F8, TCP, server mode (do not connect to existing socket)

Experimentally I have found that I need four virtual CPUs to avoid the guest OS idling at 100%. I am running a custom desktop application in the guest that requires a serial-to-serial bridge. Aside from severely choppy audio, it runs well - until it has traffic on the serial bus.

The guest never crashes when the port opens, or even when there's initial serial traffic. It only crashes after a sizable amount of bidirectional traffic has occurred.

If I use the (flaky and terrible for its own reasons) HHD Software virtual serial port bridge in the guest, there is no flatline.

If I only transmit from the application out of COM1 and dump its traffic via TCP in the host, and do not use COM2, there is no flatline.

Decreasing the application's baud rate from 115200 to 9600 delays but does not prevent the flatline.

Running a simple host-side script that relays traffic between the two TCP servers works initially, and the application functions well for anywhere from a few seconds to several minutes; then the flatline occurs 100% of the time. When it does crash, Windows does not report any load out of the ordinary (~30% CPU, ~50% memory), but it permanently hangs and has to be hard-powered off. From the host relay side, no socket abnormalities are ever reported until the guest is stopped and the sockets hit an EOF.

An excerpt of the log during a flatline doesn't show anything obvious between boot and when I hard-power off the guest:

Code: Select all

00:00:32.434519 GUI: UIMachineLogicNormal::sltCheckForRequestedVisualStateType: Requested-state=1, Machine-state=6
00:01:01.317574 Audio: Warning: Scheduling hint of stream '[WasAPI] Front:0' is bigger (10ms) than used period size (0ms)
00:06:35.738095 VMMDev: Guest Log: VBOXNP: DLL loaded.
00:06:35.892049 VMMDev: Guest Log: VBOXNP: DLL unloaded.
00:06:38.856369 VMMDev: Guest Log: VBOXNP: DLL loaded.
00:08:25.903856 Changing the VM state from 'RUNNING' to 'SUSPENDING'
00:08:25.916028 AIOMgr: Endpoint for file 'C:\Users\myuser\VirtualBox VMs\Windows Sim\Snapshots/{f0782286-2586-412d-a257-087f305e779a}.vdi' (flags 000c0781) created successfully
00:08:25.917154 PDMR3Suspend: 13 255 238 ns run time
00:08:25.917165 Changing the VM state from 'SUSPENDING' to 'SUSPENDED'
00:08:25.917175 Console: Machine state changed to 'Paused'
00:08:27.893813 GUI: Request for close-action to power VM off.
00:08:27.893973 GUI: Powering VM off..
00:08:27.903101 Console: Machine state changed to 'Stopping'
00:08:27.903592 Console::powerDown(): A request to power off the VM has been issued (mMachineState=Stopping, InUninit=0)
00:08:27.903886 Changing the VM state from 'SUSPENDED' to 'POWERING_OFF'

I have separately tried using named FIFOs for relay purposes, and overall this would be preferable, but their permissions model in Windows seems broken because I can never open them from a non-super application.