[Solved] Serial port connection

Discussions about using non Windows and Linux guests such as FreeBSD, DOS, OS/2, OpenBSD, etc.
Post Reply
gtq
Posts: 5
Joined: 8. Dec 2017, 19:25

[Solved] Serial port connection

Post by gtq »

Hi,

I'm testing the serial port in VBox (5.2.2 r119230 (Qt5.6.3)). The VM is configured with a client mode COM1 port (i.e. Port Mode: TCP, "connecting to existing pipe/socket", "Path/Address: 192.168.1.2:2023").

I run a simple TCP server that runs on 192.168.1.2 port 2023. It reads and then prints whatever the client sends. I want to use it as a log.

The weird thing is the log server doesn't receive/print anything from client unless the client sends a LF byte (i.e. ascii 0x0A), or the VM is shutdown.

For example, I can write 600 bytes to the serial port's THR register in a tight loop, nothing shows up on my TCP server side. However, if I send an LF byte after these 600 bytes, all bytes are printed on log server immediately.

The COM1 was configured as: 115200/8-bit-1-stop-no-parity/14-byte-receive-trigger. TX/RX interrupts are enabled.

The log server runs on a Linux VM. I tried read() and select()/recv(), also set/unset the TCP_NODELAY option. Same results.

It seems the data sent thru the VBox serial port are being cached by the device driver. The cache is flushed to remote only if an LF is written to the COM1's THR register.

But is this correct behavior? I don't think I have to do this with a real serial port.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Serial port connection

Post by socratis »

From my experience with serial ports many, many years ago, there's always a "Termination character" that you have to set in order to indicate the termination of the string sent. This is typically set to 0x0a. So I guess, the answer to your last question is yes, you have to include the termination character.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Serial port connection

Post by mpack »

That is not the behaviour of a serial port (which is character based, not line or packet based). However it may be the behaviour of the guest OS serial drivers. I.e. it sounds Unixy to me.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Serial port connection

Post by socratis »

mpack wrote:That is not the behaviour of a serial port (which is character based, not line or packet based).
You are most probably absolutely right, since all the serial port connection encounters that I had, were instrument based, and they definitely like a line-based approach. As in:
  • SNAP?1,2,4,10,11;\0x0a
They would definitely hang up if the termination character was not there, similar to an incomplete sentence. That BTW was for controlling a DSP Lock-In Amplifier from 20+ years ago, still have the sources... ;)
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
gtq
Posts: 5
Joined: 8. Dec 2017, 19:25

Re: Serial port connection

Post by gtq »

mpack wrote:That is not the behaviour of a serial port (which is character based, not line or packet based). However it may be the behaviour of the guest OS serial drivers. I.e. it sounds Unixy to me.
I'm not using any guest OS. I wrote a very simple MBR and jump to UART code immediately.

Ideally, even if I only write a single byte to VM's COM1 THB register, this byte should be delivered to the remote TCP server immediately. Using 0x0a as "Termination Char" leads to problems for sending binary data.

Is it possible the Nagle's algorithm(or something similar -- e.g. an internal cache) on VM side delays the data transmission?

@socratis, I read National Semiconductor's PC 16550D UART spec (June 1995), Xilinx's AXI UART 16550 spec v2.0 (Oct 2016), and Xilinx's 16550 device driver (https://github.com/Xilinx/embeddedsw/tr ... tns550/src). None of them mentioned or implemented any "Termination Char".
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Serial port connection

Post by mpack »

gtq wrote: Is it possible the Nagle's algorithm(or something similar -- e.g. an internal cache) on VM side delays the data transmission?
I think you misunderstood my previous comment. I responded to Socratis when he said that this was his recollection of how serial ports worked back in the day. It wasn't a comment on how a virtual UART works.

I have no problem believing that a virtual UART must buffer data for acceptable performance, and no problem accepting that in "pipe mode" there are additional constraints. However I have no long experience of using pipes for communications so I've no idea what the norms are with them. Quite possibly they use the Unix text terminal model, and only do a tx on EOL.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Serial port connection

Post by socratis »

gtq wrote:@socratis, I read ... spec v2.0 (Oct 2016)... None of them mentioned or implemented any "Termination Char".
You remember what I said about controlling instruments? Maybe it was a requirement for every instrument driver out there. I wasn't talking to the UART, I was using the UART to talk to instruments. The example of the SR830 DSP Lock-in Amplifier that I brought up, was from 1993 working on a 80486! Quite a few differences from last year's spec... ;)
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
gtq
Posts: 5
Joined: 8. Dec 2017, 19:25

Re: Serial port connection

Post by gtq »

Sorry for the misunderstanding. @socratis, so the "termination char" is part of the DCE device protocol (e.g. a modem connected to the serial port, or the Amplifier you mentioned)?

I believe you're right VBox is emulating this "termination char" or Unix text terminal mode, but I think it should only emulate the UART chipset, not the "termination char". If I connect 2 VMs via RS232 cable, I should be able to send data without "termination char".

If my understanding is correct this time, should this be considered a bug?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Serial port connection

Post by mpack »

I would suspect that it's the stream driver in the guest OS, not in VirtualBox. I'm no Unix expert, but ISTR that stream oriented ports can be configured for byte mode.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Serial port connection

Post by socratis »

I did a quick search and I found a "Linux Serial Programming HOWTO" (or in searhcable PDF). In there they talk about three modes of communicating:
  • 2.3.1. Canonical Input Processing
    • This is the normal processing mode for terminals, but can also be useful for communicating with other dl input is processed in units of lines, which means that a read will only return a full line of input. A line is by default terminated by a NL (ASCII LF), an end of file, or an end of line character. A CR (the DOS/Windows default end−of−line) will not terminate a line with the default settings.
  • 2.3.2. Non−Canonical Input Processing
    • Non−Canonical Input Processing will handle a fixed amount of characters per read, and allows for a character timer. This mode should be used if your application will always read a fixed number of characters, or if the connected device sends bursts of characters.
  • 2.3.3. Asynchronous Input
    • The two modes described above can be used in synchronous and asynchronous mode. Synchronous is the default, where a read statement will block, until the read is satisfied. In asynchronous mode the read statement will return immediatly and send a signal to the calling program upon completion. This signal can be received by a signal handler.
which seems to coincide with my memory cells that still remember a couple of things about serial ports. So, what you're looking for is the "Non-canonical form" with a 1-byte character per read, no timeout and I'm not quite sure about the asynchronous vs. synchronous mode, but that would depend on how your daemon is implemented. Or, you could append a termination character to each and every byte to make the communications immediate. That will add an overhead to your communications, but you'll be the judge of its importance.

Generally speaking the termination character has nothing to do with the communication, it's a "convention" between the sender and the receiver. They got to speak the same language and have the same expectations.

The HOWTO includes code and examples, so it might be worth taking a look at it...
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Serial port connection

Post by mpack »

The only question is who is doing this queuing. The OP claims that there is no OS as such, that he's booting straight into his own code. Now I know for a fact that VirtualBox doesn't require a LF before it will transmit on the serial port (*), so I guess that makes it a feature of the pipe. I suggest doing some digging into pipes on the host OS.

(*) The 16550D UART has receive and transmit FIFOs, from memory they are either 14 or 16 characters. The UART doesn't know anything about lines or EOLs, it just sends bytes. Transmit happens either when the tx fifo fill level is reached, or when 4 character times have elapsed since the last write. The 16550D was great, I miss it. Most of the modern MCUs I work with have a half assed microcode UART that requires me to set up a DMA channel to get the equivalent of FIFOs, but all so far have overlooked the vital timeout feature.
gtq
Posts: 5
Joined: 8. Dec 2017, 19:25

Re: Serial port connection

Post by gtq »

Just figured out it's my bad. When I receive the data on TCP server, I printed the data with "std::cout << data". I didn't know "std::cout" caches data by lines. Same thing with printf and Python sys.stdout.write().

I usually use these print statements to print large amount data. Never noticed this cache behavior before. What a shame!

Thank you guys for help,
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Serial port connection

Post by socratis »

gtq wrote:I usually use these print statements to print large amount data. Never noticed this cache behavior before.
OK, you got my curiosity up... ;)
What do you use instead of those statements then?
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
gtq
Posts: 5
Joined: 8. Dec 2017, 19:25

Re: Serial port connection

Post by gtq »

You mean how to fix the print caching problem? Flush the output stream like this: std::cout << data << std::flush. fflush() should work for printf().
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Serial port connection

Post by socratis »

Flush the buffers... Great, thanks for the reply.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Post Reply