Am I calculating the return address position in an unprotected case?
This part is true, at least until the address is placed in an int. The correct type for retptr will be long with x86-64 asm, so the pointer can contain a 64-bit address.
You can double check this by running the following program:
#include <stdio.h> void function(int a, int b, int c) { char buf1[5]; char buf2[10]; int *retptr; retptr = (void*)(buf2 + 40); printf("retptr points to: %p\n", (long*)(long)*retptr); (*retptr) += 8; } int main(void) { int x; printf("ret address is %p\n", &&label); x = 0; function(1,2,3); label: x = 1; printf("%d\n", x); return 0; }
By running this, you should be able to confirm that the address immediately after the function in is the one that is also held by retptr .
I believe that the reason you are not getting the expected 0 lies in this line:
(*retptr) += 8;
On my 64-bit system, x = 1 compiles as:
40058a: c7 45 fc 01 00 00 00 movl $0x1,-0x4(%rbp) 400591: 8b 45 fc mov -0x4(%rbp),%eax 400594: 89 c6 mov %eax,%esi
The first line loads 1 into x , and the other two lines pass the value of x as an argument to printf() . Notice how it is 7 bytes, not 8. If you change the increment to 7, you should see 0 , as expected.
Effectively, by adding 8, the ret command set the instruction pointer to point to 45 , not 8b . Then this code will look like this:
45 fc rex.RB cld 89 c6 mov %eax,%esi
I'm not quite sure what is happening at this point, and I suspect that it depends on the processor model. Mine seems to skip instructions to mov %eax,%esi , and therefore printf displays the value of any %eax . If you look at the function() %rax , it turns out that %rax used to store the retptr value and that the seemingly random value is being printed.