DMA Streaming in the PCIE Kernel Driver

Question

I am working on an FPGA driver for the Linux kernel. The code seems to work fine on x86, but on x86_64 I have some problems. I have implemented streaming DMA. So it looks like

get_user_pages(...); for (...) { sg_set_page(); } pci_map_sg(); 

But pci_map_sg returned addresses like 0xbd285800 that are not PAGE_SIZE , so I can't send the full first page, as the PCIE spec says

"Requests should not indicate an Address / Length combination that causes access to the memory space to cross the 4 KB border."

Is there a way to get consistent addresses, or am I just missing something important?

The source code is DMA .

Answer 1

The first possibility that comes to mind is that the user input buffer does not start at the page border. If your starting address is 0x800 bytes across the page, then the offset on the first call to sg_set_page will be 0x800. This will create a DMA address ending in 0x800. This is a common thing, and it is not a mistake.

Since pci_map_sg combines pages, this first segment can be more than one page. Importantly, pci_map_sg creates contiguous blocks of DMA address memory, but it does not create a list of low-level PCIe transactions. On x64, you are likely to get a large area because most x64 platforms have IOMMUs.

Many of the devices I'm dealing with have DMA mechanisms that allow me to specify a logical transmission length of several megabytes. Typically, a DMA implementation at a PCIe endpoint is responsible for starting a new PCIe transaction at each 4 KB boundary, and the programmer can ignore this limitation. If the resources in the FPGA are too limited to handle this, you might consider writing driver code to convert a list of Linux memory blocks into a (much longer) PCIe transaction list.