Why is PIvoke not crashing in case of violation of the calling convention (in .NET 3.5)?

Question

My solution has an unmanaged C ++ DLL that exports a function, and a managed application that PInvokes performs that function.

I just converted the solution from .NET 3.5 to .NET 4.0 and got this PInvokeStackImbalance. "The call to the PInvoke function [...] has an unbalanced stack exception. As it turned out, I called the __cdecl'ed function, since it was __stdcall:

Part C ++ (called):

__declspec(dllexport) double TestFunction(int param1, int param2); // by default is __cdecl 

Part C # (caller):

 [DllImport("TestLib.dll")] // by default is CallingConvention.StdCall private static extern double TestFunction(int param1, int param2); 

So, I fixed the error, but now I'm interested in how it works in .NET 3.5? Why is the (repeated) situation when no one (neither the caller nor the caller) clears the stack, caused a stack overflow, or some other incorrect behavior, but simply worked fine? Is there some kind of check in PInvoke, as mentioned by Raymond Chen in the article? It is also interesting why the opposite type of violation agreement (having __stdcall callle, which is PInvoked, like __cdecl) does not work at all, throwing only an EntryPointNotFoundException.

Answer 1

After some research:

The helper, which saves the situation from failure, is another register - EBP, a base pointer that points to the beginning of the stack frame. All access to local variables of the function is carried out using this pointer (with the exception of optimized code, see below). Before the function returns, the stack pointer reset to the value of the base pointer.

Before a function (such as PInvoke) calls another function (an imported DLL function), the stack pointer points to the end of the local variables of the function of the calling function. The caller then pushes the parameters onto the stack and calls this other function.

In the described situation, when a function calls another function like __stdcall, although it is actually __cdecl, no one clears the stack from these parameters. Thus, after returning from the called subscriber, the stack pointer points to the end of the block of specified parameters. This is similar to the caller function (PInvoke), which has just received a few local variables.

Since access to the local variables of the caller is through the base pointer, it will not break anything. The only thing that can happen is that the call function will be called many times in a row. In this case, the stack will grow and may overflow. But since PInvoke calls the DLL function only once and then returns, the stack pointer is simply reset to the base pointer, and all is well. Edit: As indicated here , the code can also be optimized to store local variables only in CPU registers. In this case, EBP is not used and thus an invalid ESP may result in an invalid address being returned.

Answer 2

PInvokeStackImbalance is no exception. This is an MDA warning implemented by the debug administration assistant. If this MDA is active, this is optional, you can configure it from the Debug + Exceptions dialog. It will never be active when you run without a debugger.

Getting an unbalanced stack can cause quite unpleasant problems, ranging from strange data corruption to getting SOE or AVE. It is very difficult to diagnose too. But it can also cause no problems, the stack pointer is restored when the method returns.

Code compiled into 64-bit tends to be stable; much more function arguments are passed through registers instead of the stack. It will fail if forced to run on x86, the new default for VS2010.

Answer 3

It is worth noting that the reason this changed between 3.5 and 4 is because the default behavior for PInvoke has changed. In 3.5 and earlier, he checked things like Alex described and fixed them. This causes some overhead, since validation must be performed every time PInvoke is called. In .NET 4, the behavior has changed so as not to perform this check, in order to remove performance on the right calls. Instead, an MDA warning has been added.

Old behavior can be enabled again using the app.config NetFx40_PInvokeStackResilience parameter ( http://msdn.microsoft.com/en-us/library/ff361650.aspx ).

Answer 4

When using DllImport , the default value is the actual WinApi , not StdCall . WinApi is not really an agreement, but is a standard system agreement. Perhaps it is possible that _cdecl is introduced in .Net 3.5 WinApi, and now it represents __stdcall

I really don't think so, because I remember that I always always specified __stdcall (or rather, WINAPI) when using P / Invoke. I'm not quite sure why it worked in .Net 3.5. (Perhaps DllImport was lazy then and simply “missed” the calling convention - that would be strange)