DMA PCIe Data Transfer from PC to FPGA

Question

I am trying to get DMA transfer between FPGA and Linux x86_64 machine.

On the PC side, I do this initialization:

//driver probe ... pci_set_master(dev); //set endpoint as master result = pci_set_dma_mask(dev, 0xffffffffffffffff); //set as 64bit capable ... //read pagePointer = __get_free_page(__GFP_HIGHMEM); //get 1 page temp_addr = dma_map_page(&myPCIDev->dev,pagePointer,0,PAGE_SIZE,DMA_TO_DEVICE); printk(KERN_WARNING "[%s]Page address: 0x%lx Bus address: 0x%lx\n",DEVICE_NAME,pagePointer,temp_addr); writeq(cpu_to_be64(temp_addr),bar0Addr); //send address to FPGA wmb(); writeq(cpu_to_be64(1),bar1Addr); //start trasnfer wmb(); 

The bus address is a 64-bit address. On the FPGA TLP side, I send 1 DW to read:

 Fmt: "001" Type: "00000" R|TC|R|Attr|R|TH : "00000000" TD|EP|Attr|AT : "000000" Length : "0000000001" Requester ID Tag : "00000000" Byte Enable : "00001111"; Address : (address from dma map page) 

The completion that I will return from the PC is:

 Fmt: "000" Type: "01010" R|TC|R|Attr|R|TH : "00000000" TD|EP|Attr|AT : "000000" Length : "0000000000" Completer ID Compl Status|BCM : "0010" Length : "0000000000"; Requester ID Tag : "00000000" R|Lower address : "00000000" 

so basically this is completion without data and with Unsupported Request status. I don’t think that something is wrong with the TLP design, but I don’t see any problem on the part of the driver. The kernel I use has a PCIe error message, but I don't see anything in the dmesg output. What's wrong? Or, is there a way to find why I am receiving this unsupported Completion request?

Marco

Answer 1

This is an excerpt from one of my projects (it works!). This is VHDL and is slightly different, but hopefully this will help you:

 -- First dword of TLP Header tlp_header_0(31 downto 30) <= "01"; -- Format = MemWr tlp_header_0(29) <= '0' when pcie_addr(63 downto 32) = 0 else '1'; -- 3DW header or 4DW header tlp_header_0(28 downto 24) <= "00000"; -- Type tlp_header_0(23) <= '0'; -- Reserved tlp_header_0(22 downto 20) <= "000"; -- Default traffic class tlp_header_0(19) <= '0'; -- Reserved tlp_header_0(18) <= '0'; -- No ID-based ordering tlp_header_0(17) <= '0'; -- Reserved tlp_header_0(16) <= '0'; -- No TLP processing hint tlp_header_0(15) <= '0'; -- No TLP Digest tlp_header_0(14) <= '0'; -- Not poisoned tlp_header_0(13 downto 12) <= "00"; -- No PCI-X relaxed ordering, no snooping tlp_header_0(11 downto 10) <= "00"; -- No address translation tlp_header_0( 9 downto 0) <= "00" & X"20"; -- Length = 32 dwords -- Second dword of TLP Header -- Bits 31 downto 16 are Requester ID, set by hardware PCIe core tlp_header_1(15 downto 8) <= X"00"; -- Tag, it may have to increment tlp_header_1( 7 downto 4) <= "1111"; -- Last dword byte enable tlp_header_1( 3 downto 0) <= "1111"; -- First dword byte enable -- Third and fourth dwords of TLP Header, fourth is *not* sent when pcie_addr is 32 bits tlp_header_2 <= std_logic_vector(pcie_addr(31 downto 0)) when pcie_addr(63 downto 32) = 0 else std_logic_vector(pcie_addr(31 downto 0)); tlp_header_3 <= std_logic_vector(pcie_addr(31 downto 0)); 

Let's ignore the obvious difference that I made MemWr of 32 words instead of reading dword. Another difference that caused me problems the first time I did this is that you need to use the 3DW header if the address is below 4 GB .

This means that you need to check the address that you get from the host and determine whether to use the 3DW header (only with LSB addresses) or the full 4DW header mode.

If you do not need to transmit an unholy amount of data, you can always set the dma address mask to 32 bits in the case of 3DW, Linux must reserve a large amount of memory below 4 GB by default.