Maybe I'm missing something, or perhaps I wasn't clear enough?
In the settings for the VM, I have serial port 1 enabled, Port Number is COM1, port mode is Host Device, Port/File path is /dev/ttyp0 (this is a slave port under linux, with pttyp0 being the master, used for looping back to the host machine)
using that port (ttyp0) on the host machine with minicom, it connects as expected, so the device is setup correctly on the host. When setting the above values in the VM config, I get the error I listed above.
According to 18.104.22.168: You can connect the virtual serial port to a physical serial port on your host. (On a Windows host, this will be a name like COM1; on Linux or OpenSolaris hosts, it will be a device node like /dev/ttyS0). VirtualBox will then simply redirect all data received from and sent to the virtual serial port to the physical device.
This appears to be what I've done, yet VBox seems to be unable to access it.