How can I make a custom calling convention in C?

Question

Without going into details, let's say I need to use a non-standard calling convention from C code. Functions that use this convention can return multiple values ​​on the stack. It would be simple to put each function in a shell that uses the built-in assembly to make a call by sending the result through the pointer parameters specified in the shell. Unfortunately, this solution does not generalize very well, and I need something that works in the general case. Should I just refuse and use macros to encapsulate the packaging, or is there a more general way to write, say, a variable function invokethat handles the dirty work of managing the stack?

Answer 1

Whichever approach you choose, you need to write a wrapper in the assembly. It is not possible to mess with the stack with C. I like your idea of ​​writing a single shell invoke(in asm) that does all the dirty work and then wraps it with C.