Creating PDM PCI device - MMIO problem

Discussions related to using the OSE version of VirtualBox.
Post Reply
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hello,

I'm trying to build some virtual PCI device, to emulate a real PCI device, so I can reuse Windows drivers... It is a multiport UART card, but that isn't important. I'd be happy to post code, once I get it working, as this has been a heck of a learning curve!

Anyways, my problem is the MMIO PCI bar. When I create a standard PCI IO BAR, I can set the size down to 8 ports, no problem. However, when I do MMIO, it seems the minimum size if 4096 bytes, or _4K.

Here is the MMIO part (0x80 is the width (128)).

Code: Select all

    rc = PDMDevHlpPCIIORegionCreateMmio(pDevIns, 0 /*iPciRegion*/, 0x80, PCI_ADDRESS_SPACE_MEM,
                                        PLX_MMIOWrite, PLX_MMIORead, NULL /*pvUser*/,
                                        IOMMMIO_FLAGS_READ_PASSTHRU | IOMMMIO_FLAGS_WRITE_PASSTHRU,
                                        "DigiClassicBoardPCI8", &pThis->hMmioPLX);
And the IO part.

Code: Select all

    rc = PDMDevHlpPCIIORegionCreateIo(pDevIns, 1 /*iPciRegion*/, 0x80 /*cPorts*/,
                                      PLX_IOWrite, PLX_IORead, NULL /*pvUser*/,
                                      "DigiClassicBoardPCI8", NULL /*paExtDescs*/, &pThis->hIoPLX);
    AssertRCReturn(rc, rc);
When I run the device in DOS, I get the BAR info. BAR is the physial plug location, hBAR is where I write all 1's to the BAR, and read it back.
BAR=F0804000 hBAR=FFFFF000
BAR=0000D081 hBAR=FFFFFF81

IO clearly works, but MMIO is 4K.

Real hardware looks like (which is correct).
BAR=F0800100 hBAR=FFFFFF80
BAR=00001101 hBAR=FFFFFF81

I understand that BAR moves around. It's the hBAR that is giving me problems (hBAR, again, the readback after writing all 1's).

If I go to 8K, 16K, 32K, etc, the hBAR is correctly set. I just can't seem to go below 4K. The problem is that the driver calls BS on the port size, and refuses to load fully.

So, my question is, is it possible to go below a 4096 byte window?

My device was placed on Bus 0 by PDM. I register it with a simple PDMDevHlpPCIRegister(pDevIns, pPciDev);

Is Bus 0 special? Is it MSI / MSI-X? PCIe? Does that have a 4K minimum limit? I see a few _4K consts in the code referring to a spec. Do I need to attach another PCI bus? If so, I don't know how to set that part up. I figured out how to attach UARTs and their associated downstream (upstream?) drivers, but PCI doesn't seem to have settings, so I don't know how to call it out.

Any help or leads would be greatly appreciated. Thank you for you time, too!
--Ben
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Re: Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hello,

Well, I took a stab in the dark, and added another PCI bus (I figured the worst would be a VM crash), and explicitly attached my device to it, but it didn't help..

vboxmanage setextradata DOS "VBoxInternal/Devices/pcibridge/0/a" 1
vboxmanage setextradata DOS "VBoxInternal/Devices/xx/0/PCIBusNo" 1

Without the explicit pcibridge, the VM errors out for there being no Bus 01..

I don't see any config items in the code for pcibridge .

--Ben
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Re: Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hello,

I think I traced it, somewhat (helpers and pointers, etc)..

PDMDevHlpPCIIORegionCreateMmio -> pfnPCIIORegionRegister -> pdmR3DevHlp_PCIIORegionRegister-> devpciR3CommonIORegionRegister


pdmR3DevHlp_PCIIORegionRegister does

Code: Select all

/*
     * We're currently restricted to page aligned MMIO regions.
     */
    if (   ((enmType & ~(PCI_ADDRESS_SPACE_BAR64 | PCI_ADDRESS_SPACE_MEM_PREFETCH)) == PCI_ADDRESS_SPACE_MEM)
        && cbRegion != RT_ALIGN_64(cbRegion, PAGE_SIZE))
    {
        Log(("pdmR3DevHlp_PCIIORegionRegister: caller='%s'/%d: aligning cbRegion %RGp -> %RGp\n",
             pDevIns->pReg->szName, pDevIns->iInstance, cbRegion, RT_ALIGN_64(cbRegion, PAGE_SIZE)));
        cbRegion = RT_ALIGN_64(cbRegion, PAGE_SIZE);
    }
which might set the minimum size to PAGE_SIZE, which gets passed to devpciR3CommonIORegionRegister, which does

Code: Select all

 pRegion->size        = cbRegion;
So, if this is the cause, is it possible to have the size smaller than PAGE_SIZE? I might try to gross hack it, compile a test version, and see. I'm running a release build, so I can't seem to see if the Log in there emits anything...

Anyone,
Am I on the right path? Can PDM create PCI devices with a memory foot print smaller than 4096 (4K) bytes?

Thanks,
Ben
klaus
Oracle Corporation
Posts: 1115
Joined: 10. May 2007, 14:57

Re: Creating PDM PCI device - MMIO problem

Post by klaus »

If you need inspiration, especially in the early stages: have a look at src/VBox/Devices/Serial/DevOxPcie958.cpp. It's the draft of a 16 port multiport serial device emulation. It compiles, and Linux detects it, but otherwise it wasn't really tested (it is shipping as part of VirtualBox, but there is no easy way to enable it, just the usual CFGM tweaking through extradata).

I wonder how much difference there can be with multiport serial cards... fundamentally they're likely all rather similar, because the building block for each port is pretty much always a 16550 or newer. Which means that you wouldn't have to implement the serial port engine as such. It is already separated out, ready to be reused. So you can focus on the specifics.

Regarding PCI buses: especially during development better stay away from manually created complex bus structures. It makes your life much easier to put it simply on the root bus. Later when the device is properly integrated (which needs making changes to the API code) then the assignment of devices to buses will take care of creating the necessary bridges.
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Re: Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hello klaus,

I lifted a lot of code and ideas from DevOxPcie958 : ) Thank you to the VirtualBox for including it in the build!

I did try using it in Windows. I figured out how to attach it, and attach the Char and "Host Serial" drivers.

// Named Pipe (Windows) - On LUN0
vboxmanage setextradata [VMName] "VBoxInternal/Devices/[SomeSerialBoard]/0/LUN#0/AttachedDriver/Config/IsServer" 1
vboxmanage setextradata [VMName] "VBoxInternal/Devices/[SomeSerialBoard]/0/LUN#0/AttachedDriver/Config/Location" \\.\pipe\CommPipe
vboxmanage setextradata [VMName] "VBoxInternal/Devices/[SomeSerialBoard]/0/LUN#0/AttachedDriver/Driver" NamedPipe
vboxmanage setextradata [VMName] "VBoxInternal/Devices/[SomeSerialBoard]/0/LUN#0/Driver" Char

// Serial (Windows) - On LUN1
vboxmanage setextradata [VMName] "VBoxInternal/Devices/[SomeSerialBoard]/0/LUN#1/Config/DevicePath" \\.\COM6
vboxmanage setextradata [VMName] "VBoxInternal/Devices/[SomeSerialBoard]/0/LUN#1/Driver" "Host Serial"

Ok, back... I found a driver for Windows for DevOxPcie958, but it just says there was an error, and won't start the card.

I then ran with that idea, and made an ISA version of it (8 UARTS). The shared IRQs don't work in the default UART driver in Windows, so I can only activate 1 UART, but that did work, and serial data came across. I set it up to pass in PortBase and IRQ per UART, and then I could get 2 UARTs functional, but IRQ clashing prevented the rest from starting in windows.

I then tried making a PCI function for each, and using the MF driver in Windows, but I don't know Windows drivers well enough to pull off writing the .inf for that. The VM also wouldn't assign resources to functions 1-7 of the device. Neither DOS nor Windows showed the BARs configured. Maybe something in my code was off, but it did show each function with class ID and VID, etc.

So, my most recent idea was to just take a Digi Classic PCI 8 card, and duplicate it (they have a driver writers guide on their website which details the inner workings!). The IO ports come up right, and they work in DOS, but again, Windows won't light it up. The driver reports a resource allocation problem. So, I looked in DOS (using a PCI probe program I wrote) to see how the BARs were being configured. That is when I noticed VirtualBox assigning a minimum 4096 byte memory window. So, I'm thinking the driver is checking that, and not agreeing with the memory window size, and shutting down the device.


If I could figure out the Windows drivers (Win 2K, XP, and Win7) for a custom serial port, I'd just do that, but making fake hardware seems MUCH simpler, especially when a chunk of the heavy code is already done by VirtualBox! Using Toaster from the DDK to make a custom bus driver, then layer on a custom re-write of the serial.sys to handle IRQ sharing isn't helpful.. The crazy part is that the DDK source has support for multiport cards, but the inf file has to be specially crafted, and they don't really detail that part.

From what I can find on the net, and from digging into the pile of at least 4 different serial port cards on my desk, most vendors use the 16550 family UART, and just baseaddress + UartPort * UartIndex , plus some reg for a collection of IRQ bits. It is so simple to set up the glue logic for that. I even have a card that takes a SPI version of that Uart, and exposes it to the PCI bus through a FPGA.

This is maddeningly simple, it would seem, and there are dozens of vendors with their own multiport serial drivers, but no one wants to say the few (maybe dozen) lines of code they changed. Many include the signed DDK bits!!! VSPE is a great redirector, but I need to be able to set baud rate, so I need some way to pass that to external hardware. It's crazy that such a simple and low level interface is so hard to code for. As hardware ages, I think it's one of the most important to emulate and expose.

I'm sorry for the rant. I'm sure we all should be getting back to work.

I'm going to try to modify PDMDevHlp.cpp to output logging in release mode, and see what happens in pdmR3DevHlp_PCIIORegionRegister.


klaus, thank you for taking the time to read and reply to my post.

--Ben
klaus
Oracle Corporation
Posts: 1115
Joined: 10. May 2007, 14:57

Re: Creating PDM PCI device - MMIO problem

Post by klaus »

From our experience drivers do not check the size of a BAR (especially the MMIO ones). Most OSes don't report it in an easily digestible way, and finding out is possible only by changing the content of PCI config space which is difficult to do when the resources have been allocated already. Either way, it's not the business of the driver to worry about sizes, just about the start address.

As mentioned, the DevOxPcie958 code isn't really finished or tested. So it could easily be doing something not quite right which makes the Windows driver unhappy (also there might be bit rot... no one really used the thing since 2018 and there were several rounds of mechanical changes which were never validated).

Logging the accesses by the guest driver usually gives a hint what's going wrong. If the guest driver doesn't access the device at all then often it's incorrect PCI config space content. Otherwise it's something with the returned values not matching the expectations.

Turning a PCI device into an ISA one usually needs a lot of changes, because the PCI specific helpers won't work for ISA devices.
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Re: Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hello klaus,

I have LogRel on all of my port (mem / IO) handlers, and nothing talks to the hardware from the Windows side. They only way I see accesses is when I probe it through DOS. So, I think I'm covered there.

I tried building a debug version of VirtualBox, to get more debug info, but it would fail to start the VM (any VM). I even pulled my custom hardware out.

So, I went after pdmR3DevHlp_PCIIORegionRegister in src\VBox\VMM\VMMR3\PDMDevHlp.cpp , and changed the Log/LogFlow to LogRel, and I got the following

00:00:01.079958 DigiClassicBoardPCI8 Construct #0
00:00:01.079964 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: pPciDev=000000000671a120:{0x40} iRegion=0 cbRegion=0000000000000080 enmType=0 fFlags=0x6, hHandle=0x5 pfnMapUnmap=0000000000000000
00:00:01.079969 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: aligning cbRegion 0000000000000080 -> 0000000000001000
00:00:01.079973 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: returns VINF_SUCCESS
00:00:01.079976 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: pPciDev=000000000671a120:{0x40} iRegion=1 cbRegion=0000000000000080 enmType=1 fFlags=0x5, hHandle=0x4a pfnMapUnmap=0000000000000000
00:00:01.079980 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: returns VINF_SUCCESS
00:00:01.079982 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: pPciDev=000000000671a120:{0x40} iRegion=2 cbRegion=0000000000000100 enmType=1 fFlags=0x5, hHandle=0x4b pfnMapUnmap=0000000000000000
00:00:01.079986 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: returns VINF_SUCCESS
00:00:01.079989 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: pPciDev=000000000671a120:{0x40} iRegion=4 cbRegion=0000000000000100 enmType=0 fFlags=0x6, hHandle=0x6 pfnMapUnmap=0000000000000000
00:00:01.079992 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: aligning cbRegion 0000000000000100 -> 0000000000001000
00:00:01.079995 pdmR3DevHlp_PCIIORegionRegister: caller='DigiClassicBoardPCI8'/0: returns VINF_SUCCESS

So, the regions are resized.

I then commented out cbRegion = RT_ALIGN_64(cbRegion, PAGE_SIZE); , compiled, and ran my Windows VM. The card's children UARTs were then detected, and Windows was happy!

HOWEVER, it didn't work, and the VM CPU was railed.

In the port logs, I saw accesses outside the memory window (My range is 0-0xff, but stuff like 0x310 showed up...). Odd... The UARTS didn't function, either. I think the driver uses MMIO, instead of similarly mapped IO ports..

But after more digging, I found this
00:00:01.143320 PCI: Setting up resources and interrupts
00:00:01.143922 AssertLogRel V:\VirtualBox-6.1.22\src\VBox\VMM\VMMR3\IOMR3Mmio.cpp(225) int __cdecl IOMR3MmioMap(struct VM *,struct PDMDEVINSR3 *,unsigned __int64,unsigned __int64): !(GCPhys & PAGE_OFFSET_MASK)
00:00:01.143932 Misaligned! GCPhys=00000000f0805100 LB 0000000000001000 DigiClassicBoardPCI8 (DigiClassicBoardPCI8[#0])
00:00:01.143937 AssertLogRel V:\VirtualBox-6.1.22\src\VBox\Devices\Bus\DevPciIch9.cpp(2707) int __cdecl devpciR3UpdateMappings(struct PDMPCIDEV *,bool): RT_SUCCESS_NP(rc)
00:00:01.143993 VERR_IOM_INVALID_MMIO_RANGE (-2605) - The specified MMIO range was invalid. It was either empty or it was out of bounds.


So, I'm assuming here, VirtualBox uses paged memory, and bumps out when accesses to that range happen, so that they can be handed off to their proper handlers, for non-memory mapped memory. I figured something like that was the reason, I just wanted to see what it would do.. I'm guessing that failed mapping causes odd side effects in other mapping, and causes my device code to get called for other stuff. I'm not sure there.

So, I need a way to let VirtualBox make that 0x1000 byte page grab happen (or at least align to the page, and not have overlap), or modify the stuct that holds cbRegion, to take another parameter to "lie" to the BAR handler about the size, to make the driver happy. Windows sends the device driver a resource allocation table on creation, with IOs, memory, and IRQ lines assigned to the device, so I know the driver could make a decision based on that. Or, is there a way to know when the VM has started, and layouts are done, and I can go back and down-size pRegion->size? Finding all accesses to ->size is hard, as the word size is used, a lot!

As for the PCI and ISA devices, y'all have given PLENTY of examples with serial ports, LPT, floppy, etc. I have a simple ISA device for debugging that MS Toaster driver. Just spit out data to port 0x200! Once I figure out more of this serial port stuff, I'm gonna go after some more boring hardware, such as IO cards. I have plenty of aging systems with IO cards in ISA and PCI slots, and virtualizing that stuff to a USB device or something would be awesome. You know, using some FTDI parts, I could bump out ISA card ranges, to real ISA cards over USB. Sometimes the old hardware is still good, but the CPU has just gone crazy after 30 years of flipping flops.

Thank you again for your time and guidance, I really appreciate it!
Ben
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Re: Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hmm, on second thought, maybe the VM backend makes the Guest align the device on a page, to make sure the Host can easily just map memory 1:1... That would be harder to fix, as it is the Guest BIOS / OS..

--Ben
klaus
Oracle Corporation
Posts: 1115
Joined: 10. May 2007, 14:57

Re: Creating PDM PCI device - MMIO problem

Post by klaus »

MMIO regions need to be page aligned (and page sized) in practice. It's the unit of the access interception. With PCI there should be no issue with false sharing, but there might be assumptions in the VirtualBox code about having just a single party per page.

Either way, it's super weird that the removing of the size aligning has any real effect. Because the BAR allocation should end up with 4K alignment in any case (leaving the rest of the page unused), because all other MMIO resources of all other PCI devices in VirtualBox are at least 4K in size, so there should be absolutely no reason why this one will end up unaligned.

You can get the PCI config space summary of a running VM (even if the guest OS hangs or some such) with

Code: Select all

VBoxManage debugvm "vmname" info pci
Without knowing what's where it's hard to say much. It's already quite unexpected that a debug build doesn't work for you. It's something which developers use all the time (with a release build of the kernel drivers usually). Again, without knowing what errors you get straight on VM start it's hard to tell.
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Re: Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hello klaus,

Thank you for the PCI debug command! I noticed when the VM exits poorly, it dumps a bunch of good data, but normal exits don't give much (in a way, why should they, everything was fine). I don't know what is wrong with my debug build. It would emit an assert on timers having the same names, and other stuff. I can re-run and post the logs if anyone would be interested. I copy the builds to a test PC, in their own folders, so I still have it.

Anyways, I have some success to report!

I used a disassembler to peek into the serial port driver, and it was indeed looking at memory ranges to validate the hardware. It has two memory ranges (one is a PLX PCI bridge chip config, and the other is the UART bank), and they have different sizes. The driver knows this, and maybe it uses those sizes to know which range is which function. It looks like the resource ordering in Resources enumerated by Windows are in BAR order, but I don't know if that is a guarantee... Maybe it is just pure double check...

So, in working with the 4K minimum land grab for MMIO in the VM, I sized one BAR at 4K, and the other at 8K (original was 128 bytes and 256 bytes). Then, I used a hex editor to update the values checked by the driver (it was laid out as such that it wasn't difficult). I then blue screened the VM because apparently Windows drivers must have valid checksums. I found a tool to correct that... I then reinstalled the driver in the VM, without the .cat file, and it took it! So, I took the signed vendor driver, modified it, and installed it unsigned.

The test UART coming out over a named pipe had full duplex data, and the piped physical UART passed data, too! The baud rates were 8x too fast, due to the original card having a 9 something MHz clock, but a quick mod to UartCore.cpp fixed that.

So, overall, it kinda works. It has been one heck of a learning experience building VirtualBox from source (Qt was a nightmare), building virtual hardware, digging into VBox source for weird details to appease the host, and having something complicated work. Thank you again, klaus, for your help, time, and patience with me.

I can post my code, if that's ok. It is certainly not up to the level seen in other files in the VBox tree. I might rename some functions, as they still have the the base ox958R3 names... But, I suppose it could be an encouragement to others (or a warning that this stuff is indeed difficult). Certainly not vetted for production use, or even proper use of virtual hardware design technique. I write VB code usually ;)

Thank you, again, again, and again, very much for your time,
Ben
klaus
Oracle Corporation
Posts: 1115
Joined: 10. May 2007, 14:57

Re: Creating PDM PCI device - MMIO problem

Post by klaus »

Resource enumeration in Windows (and all other OSes I've seen so far) for PCI devices is always strictly in BAR order. Since the BARs have certain defined meaning most drivers see no reason to check the details. As long as the resource is set up it's good to go.

Since we now have a real world driver which trips over MMIO bar size being "rounded up" by VirtualBox we should find a solution which avoids this, and still guarantees that there is no other MMIO ending up in the same page. In the legacy BIOS this will worst case need rounding up the size, ensuring proper alignment. With EFI it needs to be investigated what happens. For that it would be great to have such a device...

So even if your code isn't anywhere near production ready it would still be useful for testing purposes. To find out what needs changing in VirtualBox to make this easier.
SrBIOS
Posts: 18
Joined: 30. Aug 2021, 20:06
Primary OS: MS Windows other
VBox Version: OSE other
Guest OSses: DOS / Windows

Re: Creating PDM PCI device - MMIO problem

Post by SrBIOS »

Hello,

Here is my code for the Digi Classicboard8 PCI.

It is not to be used as a model of good code / driver design, but it works. It may have a fifo glitch, but the Digi driver expects a 16C650, and I used 750.

Hopefully this can help someone writing a PCI device, make something better!

Thank you again to klaus, and the whole VirtualBox team for creating this crazy mechanism to add crazy hardware in!

--Ben
Attachments
DigiC8.cpp
Code to faux emulate a Digi Classicboard8 PCI
(29.85 KiB) Downloaded 83 times
Post Reply