PCI device Bus Mastering

[Joey123]

I have created a PCI device addon and am trying to use the bus mastering which was implemented in VBox version 4.3.0 onwards. I have found the read and write functions in VBox/vmm/pdmdev.h however there doesn't appear to be any functionality to see how much memory the VM has.

As the amount of memory is subject to change between each virtual machine it would seem that this is a necessary function for those wishing to do PCI bus mastering? Have I just missed something in the API or does anyone know to find the amount of memory the VM has (without re-parsing the machine.vbox file)?

Thanks

[michaln]

Why should a device care how much RAM there is?

Also, bus mastering has always been there. How else would emulated devices like AHCI or OHCI/EHCI work?

[Joey123]

The device has its own mapped bus so in order for it be able to map the bus mastered ram in correctly it needs the size of the ram.

You are correct about bus mastering being present in previous versions, I didn't see it because the help functions that make using it easier are not present.

[michaln]

I think you'll need to back out a bit and explain what you're doing, and how your device differs from standard bus-mastering devices like NICs and storage controllers that VirtualBox emulates. Right now I can't answer your question because to be honest, I still don't understand it.

I'm also curious how your PCI device finds the RAM size on a physical system?

[Joey123]

Apologies for the delayed reply I was called away from this.

I have looked at VBox/Storage/ISCSI.cpp (which I believe is the storage controller) and have been unable to see any bus mastering (though with 5000 lines of code I may well have missed it), could you point me to the source file for a device which you know has PCI bus mastering?

I have a simulation containing a bus model which I am attempting to connect to virtualbox, this bus does some other things which are needed as well so I am not able to remove it. This simulator should be able to access both bus mastered memory and the memory contained on one of the PCI bars so I map them both on my bus to the same physical areas as on the VM bus in order for the driver to use them correctly. In my current set-up the PCI memory is created and mapped on my bus at the physical address that the MMIOMap callback provides however I can't seem to find any such callback to get the address or size of the DDR.

I know from building Linux kernels that Intel machines RAM usually starts at address 0 on its address bus, so using this address and the size as my VM memory (1Gb) I have managed to get it all to work, others who may use this will have virtual machines with differing amounts of memory so ideally I would like to find the size of the DDR by interrogating the VM.

In the real hardware I'm not sure how this works, I suspect memory requests go to the system bus where they are then routed to the correct memory area. However if possible I would like to avoid giving the simulator access to the whole virtualbox address bus and just restrict it to a few areas of memory.

[michaln]

Apologies for the delayed reply I was called away from this.

No problem...

I have looked at VBox/Storage/ISCSI.cpp (which I believe is the storage controller) and have been unable to see any bus mastering (though with 5000 lines of code I may well have missed it), could you point me to the source file for a device which you know has PCI bus mastering?

Ah, you looked at the wrong place. VBox/Storage is the purely host side implementation of storage backends. What you want is VBox/Devices/Storage. Now that I think about it, all of the controllers there (ATA, AHCI, BusLogic SCSI, LSI Logic SCSI/SAS) are either capable of bus mastering (ATA) or use bus mastering exclusively (all the rest).

I have a simulation containing a bus model which I am attempting to connect to virtualbox, this bus does some other things which are needed as well so I am not able to remove it. This simulator should be able to access both bus mastered memory and the memory contained on one of the PCI bars so I map them both on my bus to the same physical areas as on the VM bus in order for the driver to use them correctly. In my current set-up the PCI memory is created and mapped on my bus at the physical address that the MMIOMap callback provides however I can't seem to find any such callback to get the address or size of the DDR.

Hmm, I'm still a bit lost. It's not obvious to me what's where. Is the "simulator" something that runs inside the VM, or outside?

And by MMIOMap callback you mean the callback given to PDMDevHlpPCIIORegionRegister()?

I know from building Linux kernels that Intel machines RAM usually starts at address 0 on its address bus, so using this address and the size as my VM memory (1Gb) I have managed to get it all to work, others who may use this will have virtual machines with differing amounts of memory so ideally I would like to find the size of the DDR by interrogating the VM.

Yes, RAM usually starts at physical address 0 and goes up to some more or less arbitrary "top of memory" address. Potentially with one or more memory holes, but I think that's not relevant here. Technically speaking this is just a convention, but I don't believe anything called a "PC" uses a different memory layout. Memory on PCI devices is then mapped somewhere above system RAM, and it's up to the OS to decide where since it's all relocatable.

There is no interface in VirtualBox for devices to query system RAM size because there's no such interface in physical machines. A PCI device doesn't know or care how much RAM there is. It's the operating system's business to map memory resources such that they don't overlap, but that's all.

In the real hardware I'm not sure how this works, I suspect memory requests go to the system bus where they are then routed to the correct memory area. However if possible I would like to avoid giving the simulator access to the whole virtualbox address bus and just restrict it to a few areas of memory.

Yes. From the host (CPU) point of view, there are no differences between memory addresses. The CPU reads/writes memory and the chipset routes the accesses to system RAM or onto the (PCI) bus. PCI memory accesses are then claimed by bridges or devices as appropriate.

From the PCI device point of view, it's very similar. A bus mastering device just accesses memory at a given address and again it's the chipset's job to route that to system memory or potentially other memory resources on the PCI bus. The device itself can't tell which is which.

Short of using some kind of IOMMU, there is no way to restrict bus-mastering devices' access to memory. Any bus master can access any physical memory address in the system.

[Joey123]

Ah, you looked at the wrong place. VBox/Storage is the purely host side implementation of storage backends. What you want is VBox/Devices/Storage. Now that I think about it, all of the controllers there (ATA, AHCI, BusLogic SCSI, LSI Logic SCSI/SAS) are either capable of bus mastering (ATA) or use bus mastering exclusively (all the rest).

Thanks I can't believe I missed the devices folder XD

Hmm, I'm still a bit lost. It's not obvious to me what's where. Is the "simulator" something that runs inside the VM, or outside?

The way this model was set up means that the simulator can be a simulator running within VBox, a simulator connected to via a socket or the real hardware which can be connected to (the only noticeable difference between them all is accuracy of the modelling and speed).

And by MMIOMap callback you mean the callback given to PDMDevHlpPCIIORegionRegister()?

Yes I did.

There is no interface in VirtualBox for devices to query system RAM size because there's no such interface in physical machines. A PCI device doesn't know or care how much RAM there is.

I suspected as much but hoped there might be something to do this or some way of restricting the view to avoid having to alter the bus model too much, I will have to give it access to the whole bus then like the hardware does.

Thanks for all your help