How does cdecl calling convention damage ESP?

Question

My application crashes because the library function that I call modifies the ESP, although it is declared as cdecl.

The library (libclang.dll) is compiled using MinGW, and I use it from the VC ++ project. Functions are exported as C functions, and Dependency Walker tells me that they have the correct cdecl calling convention. Functions are imported in my project using dllimport, including the Clang file "index.h". It seems that not all functions distort ESP, so some functions are successful, others lead to crashes.

Here is the assembly of the working function:

// call to clang_getNumDiagnostics(TU); - works! 5AF3EFAB mov esi,esp 5AF3EFAD mov eax,dword ptr [ebp-30h] 5AF3EFB0 push eax 5AF3EFB1 call dword ptr [__imp__clang_getNumDiagnostics (5AF977E0h)] 5AF3EFB7 add esp,4 5AF3EFBA cmp esi,esp 5AF3EFBC call @ILT+7135(__RTC_CheckEsp) (5AF16BE4h) 

The next function call will change esp (appendix 4) and thus lead to a crash due to a runtime check in __RTC_CheckEsp.

 // call to clang_getTranslationUnitCursor(TU); - fails! 5AF3EFC1 mov esi,esp 5AF3EFC3 mov eax,dword ptr [ebp-30h] 5AF3EFC6 push eax 5AF3EFC7 lea ecx,[ebp-234h] 5AF3EFCD push ecx 5AF3EFCE call dword ptr [__imp__clang_getTranslationUnitCursor (5AF9780Ch)] 5AF3EFD4 add esp,8 5AF3EFD7 cmp esi,esp 5AF3EFD9 call @ILT+7135(__RTC_CheckEsp) (5AF16BE4h) 

I already posted a question for this problem, but I thought that I asked specifically about the calling cdecl convention to get more specific information about the possibility of esp overflow, as I think it might be the source of the problem ... So, excuse this "double mail" .

The source may also be in the wrong function called (possibly due to a problem in the def file that I created with dlltool and then created lib lib) - the ordinals were different from what Dependency Walker showed - I tried with the corrected ordinals, but without changes). I feel this is hardly the source of the problem, because another function is working fine and returning the correct values ​​...

Thanks!

[Update]

As requested assembly for __imp__clang_getTranslationUnitCursor

 6660A4A0 push ebp 6660A4A1 mov ebp,esp 6660A4A3 push edi 6660A4A4 push ebx 6660A4A5 mov eax,dword ptr [ebp+8] 6660A4A8 mov ebx,eax 6660A4AA mov al,0 6660A4AC mov edx,14h 6660A4B1 mov edi,ebx 6660A4B3 mov ecx,edx 6660A4B5 rep stos byte ptr es:[edi] 6660A4B7 mov eax,dword ptr [ebp+8] 6660A4BA mov dword ptr [eax],12Ch 6660A4C0 mov eax,dword ptr [ebp+8] 6660A4C3 mov edx,dword ptr [ebp+0Ch] 6660A4C6 mov dword ptr [eax+10h],edx 6660A4C9 mov eax,dword ptr [ebp+8] 6660A4CC pop ebx 6660A4CD pop edi 6660A4CE pop ebp 6660A4CF ret 4 

[Update 2] Since both VC ++ and GCC use cdecl by default, and there is no way to enforce another default calling convention in GCC without explicitly specifying it in the function declaration (which is not done for problem functions), I really sure cdecl is used everywhere.

I found this link which points out some differences that may explain why some functions work and others - "t:

Visual C ++ / Win32

• Objects larger than 8 bytes are returned in memory.

• When returning to memory, the caller passes a pointer to the memory cell as the first parameter (hidden). The called memory fills the memory and returns a pointer. the caller displays a hidden pointer along with the rest of the arguments.

MinGW g ++ / Win32

• Objects larger than 8 bytes are returned in memory.

• When returning to memory, the caller passes a pointer to the memory cell as the first parameter (hidden). The called memory fills the memory and returns a pointer. the called one pushes a hidden pointer from the stack upon return.

Maybe a problem? is there any way to solve this problem? or do I need to change Clang Index.h and switch to stdCall?

[Update 3]

Below is the corresponding GCC-Bug . It seems that in 4.6 (64 bit) and 4.7 (32 bit) you can use the new ms_abi attribute of the function to fix the problem described in [Update 2].

Answer 1

GCC and Visual C ++ do not implement the same cdecl calling cdecl . Wikipedia explains :

There are some differences in the interpretation of cdecl, especially in how to return values. As a result, x86 programs compiled for different operating system platforms and / or using different compilers may not be compatible, even if they both use the cdecl convention and do not access the underlying environment. [...] For transmission "in memory", the caller allocates memory and passes a pointer to it as a hidden first parameter; the caller fills the memory and returns a pointer, invoking a hidden pointer upon return.

The last sentence is important: the GCC version of cdecl makes the calling link a clean hidden pointer, while the Cdecl version for Visual C ++ leaves it to clear the caller.

Answer 2

according to the clang site, at startup you can create it using msvc, so why not just rid yourself of any problems and create msvc libclang, which will provide the correct calling and ABI conventions. alternately, you can use the makefile to build with gcc via msvc.

Answer 3

It looks like you identified the issue in your update # 2. If you want to work around this problem without changing the source of the clang library, you can write your own wrapper function that is compiled in GCC using a calling convention that the two compilers actually agree to return values ​​through the memory buffer:

 // compile this wrapper in MinGW - it will be able to call the // original clang_getTranslationUnitCursor() correctly, since // it'll have the same idea of how __cdecl shoudl handle values // returned in a memory buffer // // Since both MSVC and GCC __stdcall functions seem to handle return // values via memory in the same way, this wrapper should be callable // by MSVC CXCursor __stdcall wrapper_getTranslationUnitCursor(CXTranslationUnit tu) { return clang_getTranslationUnitCursor(tu); } 

Hope you don't have to do this for too many features.

An alternative is shell compilation in MSVC, which uses an assembly language module (or built-in assembly) to handle the difference in the call.

I wonder if clang will consider this something that needs to be fixed in their code?