Passing a structure by value to a P / Invoked library on a 64-bit version of Linux?

Question

I am trying to get my own dependencies for the C # library that I use to compile on Linux x86_64. The code itself is platform independent and easy to compile.

However, by first trying to run my project on Linux with a compiled dependency, I started getting weird results from the library and segfault later. After some research, it turns out that the parameters in the P / Invoke functions were not passed in the correct order. It seems that they are being transferred back.

I tried to compile the native dependency in several ways and explicitly define various calling conventions. Nothing seems to work.

Extern Method Definition for C #

[DllImport(InteropUtil.PLATFORM_DLL)] public static extern NavStatus dtqFindPath(IntPtr query , NavmeshPoint startPosition , NavmeshPoint endPosition , IntPtr filter , [In, Out] uint[] resultPath , ref int pathCount , int maxPath); 

C ++ Relevant Definition

 #if _MSC_VER // TRUE for Microsoft compiler. #define EXPORT_API __declspec(dllexport) // Required for VC++ #else #define EXPORT_API // Otherwise don't define. #endif extern "C" { EXPORT_API dtStatus dtqFindPath(dtNavMeshQuery* query , rcnNavmeshPoint startPos , rcnNavmeshPoint endPos , const dtQueryFilter* filter , dtPolyRef* path , int* pathCount , const int maxPath) { return query->findPath(startPos.polyRef , endPos.polyRef , &startPos.point[0] , &endPos.point[0] , filter , path , pathCount , maxPath); } } 

G ++ compiler settings

 g++ -shared -o cai-nav-rcn.so.1 -g -fPIC -I Detour/Include -I DetourCrowd/Include -I Nav/Include Detour/Source/*.cpp DetourCrowd/Source/*.cpp Nav/Source/*.cpp 

In the bottom output, the dtqFindPath line clearly shows the out-of-order parameters. maxPath should be 100 ( 0x64 ), but instead 1298. 1298 is the first int in the startPos structure. 100 instead of the path value.

Partial GDB Output

 Thread 1 (Thread 0x7fef64330740 (LWP 3923)): #0 0x00007fef63823ce9 in waitpid () from /usr/lib/libpthread.so.0 #1 0x00000000004ae448 in ?? () #2 0x0000000000503b8b in ?? () #3 0x00000000004226b2 in ?? () #4 <signal handler called> #5 0x00007feef052339c in dtNavMeshQuery::findPath (this=0x5405610, startRef=88101520, endRef=4203419680, startPos=0x7fff5fd3975c, endPos=0x7fff5fd3974c, filter=0x7fef64176ec0, path=0x64, pathCount=0x44d6595341be38e0, maxPath=1298) at Detour/Source/DetourNavMeshQuery.cpp:958 #6 0x00007feef0534d19 in dtqFindPath (query=0x5405610, startPos=..., endPos=..., filter=0x7fef64176ec0, path=0x64, pathCount=0x44d6595341be38e0, maxPath=1298) at Nav/Source/DetourNavMeshQueryEx.cpp:234 #7 0x0000000041ec2140 in ?? () ... #17 0x0000000005405610 in ?? () #18 0x0000000000000000 in ?? () 

I already compared the sizes of the rcnNavmeshPoint and NavmeshPoint at both ends, they are the same. The parameters included in the P / Invoke call are in the correct order, checked using the debugger.

You can also include that the library I'm trying to use is CritterAI .

So my question is this: what should I change to align the calling convention between these two pieces of code?

---

Update

I highlighted the problem. These are structures that are not properly transmitted. I created SSCCE to demonstrate this:

interop.cpp

 #include <cstdio> #if _MSC_VER #define EXPORT_API __declspec(dllexport) #else #define EXPORT_API #endif struct s { unsigned int a; float b[3]; }; extern "C" { EXPORT_API void testStruct(s str) { printf("STRUCT NATIVE\n"); printf("SIZE: %u\n", sizeof(s)); printf("%u, (%f, %f, %f)\n", str.a, str.b[0], str.b[1], str.b[2]); } } 

cs.cs

 using System; using System.Runtime.InteropServices; namespace InteropTest { [StructLayout(LayoutKind.Sequential)] public struct v { public float X; public float Y; public float Z; } [StructLayout(LayoutKind.Sequential)] public struct s { public uint A; public v B; } public class Test { [DllImport("./test.so")] public static extern void testStruct(s str); unsafe static void Main(string[] args) { s mStr; mStr.A = 22; mStr.BX = 33f; mStr.BY = 44f; mStr.BZ = 55f; Console.WriteLine("STRUCT MANAGED"); Console.WriteLine("SIZE: " + sizeof(s)); Console.WriteLine(mStr.A + ", (" + mStr.BX + ", " + mStr.BY + ", " + mStr.BZ + ")"); testStruct(mStr); } } } 

Compiled with

 g++ -shared -o test.so -g -fPIC interop.cpp && mcs /unsafe cs.cs && ./cs.exe 

Logout on my system

 STRUCT MANAGED SIZE: 16 22, (33, 44, 55) STRUCT NATIVE SIZE: 16 22, (33.000000, 0.000000, 3.179688) 

Several other tests show that the structures are “skipped”, where printing str.a print the value of the next non-structural parameter. The rest of the structure seems like garbage.

Answer 1

Note: The answer below addresses the original version of the question.

On Linux x86_64, there is only one calling convention. It is known as System V AMD64 ABI . Regardless of the mismatch, this is certainly not in defiant conventions. Perhaps the structure declarations do not match, or something else is wrong that we cannot see.

The next thing I would like to do, in your place, would be to write a simple test code. I would write a C ++ function, getting a couple of int parameters. Make sure they are transferred in the correct order. Convince yourself that the calling convention is not a problem, and then dig deeper to find out the true cause of the problem.