Serial device data not transmitted to Win XP guest (#17116)

[ruga]

---

[ModEdit, related ticket #17116: Serial device data not transmitted to Win XP guest]

---

VirtualBox V5.1.14
Host: Win 7 Ultimate
Guest: Win XP Pro

We attach a power supply unit (PSU) to serial port COM1 of the physical machine running Win 7.
The PSU sends status data to COM1 permanently.

In the guest (WinXP) the virtual serial port COM1 is mapped to the real serial port COM1.
This is the serial port configuration in the guest:

• "Enable Serial Port": Checked
  "Port Number": COM1 IRQ: 4 I/O Port: 0x3F8
  "Port Mode": Host Device
  "Connect to existing pipe/socket": Checked
  "Path/Address": COM1

In the guest an application should monitor the state of the PSU.
Now the Problem:
When we start the VM the application doesn't receive the status data initially.
Only if we send any character from the VM to COM1, the status data is received by the monitor application as expected.
Further we can see, if we don't send a character to COM1, close the VM and start it again, the PSU status data is received at once.
This succeeds as long as we don't reboot the host.
After rebooting the host the next time and starting the VM receiving the PSU status data fails again.

It seems, that VirtualBox requires sending some data to the serial port first, before it is able to receive data from the serial port.

By the way, running the monitoring application in the host works perfectly.

What do we do wrong ? Any help will be appreciated.

[mpack]

Is the serial port on the host native/internal? I.e. not a USB serial adapter?

[ruga]

No, there is no USB serial adapter used. The serial interface of the PSU is directly connected by a serial cable to the RS232 interface of the host.

[ruga]

For your additional info, the settings of the physical host serial port COM1 are:

• Baud: 9600
  Data bits: 8
  Parity: none
  Stop bits: 1
  Flow control: none
  Use FIFO buffers: checked
  • Receive buffer: high
    Transmit buffer: high

[mpack]

Where did this XP image come from? It does not seem to be a normal install from a setup CD. For example, the hdd is a VHD, is named differently than the VM, and the VM seems to be using EBoot - presumably to prevent a trial version of XP from expiring.

If it isn't a standard install of XP then we have no way of predicting how it should behave.

[ruga]

We use this Win XP image only for demonstrating the serial port behaviour based on a relative restricted system context, because our original main system tailored for our products and based on Win XP is a something bigger one.
We got this image several years ago as vmdk hdd for test purposes and transferred it to a vhd hdd. So we don't know how it has been created originally. But, presumably it's not a trial version.

The EBoot hdd has been added to the VM only for test purposes. In the initial testing phase of the VM, if our main System (here substituted by the Win XP System) has stopped its startup by error, we could boot from EBoot hdd and access the main system components causing the error within the mounted main system.
But ok, we are going to remove the EBoot hdd from the VM.

Besides, I'm going to replace the VM by a VM with a newly installed Win XP. But I have no doubt, that the error will still remain.

[mpack]

Besides, I'm going to replace the VM by a VM with a newly installed Win XP. But I have no doubt, that the error will still remain.

I don't know what makes you so confident of that. We have many people successfully using serial ports in XP VMs, which is partly why I believe there is something odd about your XP installation. For example if it started off as a VMWare or VirtualPC VM then it might have UART drivers that are specific to those platforms.

[jorgensen]

Maybe this could be of help - viewtopic.php?f=2&t=26860
Have you uninstalled the prior guest additions?
Otherwise try an USB serial adapter, and install the driver at the guest.

[ruga]

Now, I have generated a completely new Win XP system in the VM. It behaves exactly as the erroneous prior VM (see attachments).
After having started it, you have to send a character to the serial device at first. Only then, the data permanently sent by the serial device are received by the application in the VM.

I was relatively sure that this new XP guest System would react in this erroneous way, because the original guest system (with many applications "on board" and showing this error for the first time) has based on a completely new generated XP too.

@jorgensen:
Unfortunately, we can't use a USB serial adapter, because the guest system has to be unchanged in its "main features" compared to the native system running directly on a machine. The XP based system comes from a customer expecting that he will get back the system running in a VM, but completely unchanged in its functionality.
Thanks for your hint link. I'm going to study it.

[ruga]

I continued analyzing.

In general:
Running a serial port monitor you can watch the IOCTL traffic between an application and the serial port driver.
For analyzing reasons, our application is a COM test tool, by which we can send/receive data to/from the serial port.

SCENARIO 1:
Fig. 1 shows the IOCTL traffic, when the COM test tool runs in the HOST and waits for data from the physical serial port.
This constellation is used for comparison reasons later.
We see that after the start the tool causes some initial IOCTL traffic regarding the port settings.
From line 36 on it gets interesting: the tool sends a Get_COMMStatus call to the host driver and gets for answer, that there would be 28 characters (=1c) in the InQueue of the driver (lines 36-37). After having got the same answer a second time (lines 38-39), the tool sends a Read call to the driver.
The Driver returns the queued 28 characters (lines 40-41).
From now on, data are permanently sent from the driver to the tool caused by alternating Get_COMMStatus and Read calls.

SCENARIO 2:
If we start the serial port monitor and the COM test tool in the VM and watch the traffic between the COM test tool and the virtual serial port driver in the GUEST
we get this protocol (fig. 2-1):
No difference compared to the first protocol in lines 0-35. But: the Get_COMMStatus call in line 36 returns with "0 characters"(!) in the InQueue. This doesn't
change for ever.
In this scenario the simultaneous IOCTL traffic between VirtualBox and the physical serial port driver is interesting perhaps. It is logged in fig. 2-2.
Up to line 1479 the log entries result from the VM start and the monitor start. From line 1479 on the log entries represent the situation described by fig. 2-1.
The entry sequences lines 1483 - 1503 are repeated endless in the following.

SCENARIO 3:
In scenario 2, if we send a character from COM test tool to the virtual serial port in the GUEST, the COM test tool receives the permanently sent data of the physical device (see fig. 3). This is logged from line 377 on.

[ruga]

... and here comes figure 3.

[ruga]

Since noone has a good advice for me with respect to the described IOCTL sequences, it seems for me, that there could be a bug in the communication between the virtual serial port and the serial device.
So I would prefer to open a bug ticket.

[socratis]

Added related ticket in the first post, #17116: Serial device data not transmitted to Win XP guest.
Thanks @ruga for including the discussion link to the ticket.

[mpack]

The thought occurs... in the XP guest, was the serial port enabled while XP was installed? If not, then did you remember to "Search for new hardware" after it was enabled? ISTR that with certain options, XP - at least when running in a VirtualBox VM - doesn't always auto-detect new legacy (i.e. non plug and play) hardware.