PCI passthrough changing BAR size

[rpoffen]

I am trying to pass a PCI device from a Linux (Centos 7.2) host to a Centos 6 guest. On the host, the device appears in lspci -v as ..
06:0b.0 Processing accelerators: Credence Systems Corporation Device 5454 (rev ba)
Subsystem: Credence Systems Corporation Device 5454
Flags: bus master, 66MHz, medium devsel, latency 32
Memory at f220b000 (32-bit, non-prefetchable)
Memory at f2000000 (32-bit, non-prefetchable) [size=2M]
Capabilities: <access denied>
Kernel driver in use: pci-stub

but on the guest, it appears as
06:0b.0 Class 1200: Credence Systems Corporation Device 5454 (rev ba)
Subsystem: Credence Systems Corporation Device 5454
Flags: bus master, 66MHz, medium devsel, latency 32
Memory at e1900000 (32-bit, non-prefetchable) [size=4K]
Memory at e1a00000 (32-bit, non-prefetchable) [size=2M]
Capabilities: <access denied>
Kernel driver in use: pci_access
Kernel modules: pci_access

Note that BAR0 has been changed from 512 bytes to 4K.

Is there a way to fix this?

[klaus]

The "growing" of BARs to 4K is an unavoidable side effect of PCI passthrough. The IOMMU can only work with 4K pages, so VirtualBox must make sure that nothing else lives in the same page.

[michaln]

Note that the PCI 3.0 specification says: "Devices are free to consume more address space than required, but decoding down to a 4 KB space for memory is suggested for devices that need less than that amount." It is legal to create a 512-byte BAR, but it is unwise because it can lead to problems.

[rpoffen]

Thanks for the answers. Funny that KVM doesn't do this. I thought that VirtualBox and KVM largely use the same technology.
Unfortunately we have software that expects the BAR size to be advertised at 512 bytes, and fails to work properly under VirtualBox.

[klaus]

Could be that KVM relies on the guest OS leaving BARs alone, i.e. that the placement created by the PCI BIOS (which could easily do a little more alignment than the BAR needs minimally) is left unchanged. Or they managed to find a way to use software to cleanly split up the BAR accesses, but that can't be cheap.

[rpoffen]

Sounds reasonable.
At this point, for what I am trying to do, I can get KVM to work, but there are limitations. For example, our hardware has PLX PCIe switches in the fabric, and we actually use some of the device functionality available through its BAR. So in this respect, it is both a switch and an endpoint device. KVM does not allow these to be passed through. I tried using VMWare ESXi, but I can't seem to control how the bus/device/function are mapped, I need these to be predictable, and they come up in random bus/device/function mapping and I have not been able to figure out how to change this behavior, I asked about this on the VMWare forums, but have not gotten any answers. At least with KVM or VirtualBox, there are ways to control this mapping.