How to write self-modifying code in x86 assembly

Question

I am looking to write a JIT compiler for a hobby virtual machine that I have been working on recently. I know a little assembly, (I am mainly a C programmer. I can read a large assembly with reference to opcodes that I don’t understand, and write some simple programs.), But it’s hard for me to understand a few examples of self-modifying code that I found in The internet.

This is one such example: http://asm.sourceforge.net/articles/smc.html

The following sample program performs about four different modifications at startup, none of which are clearly explained. Linux kernel interrupts are used several times and are not explained or detailed. (The author sent the data in several registers before calling interrupts. I assume that he passed arguments, but these arguments are not explained at all, leaving the reader to guess.)

What I'm looking for is the simplest, easiest example in the code of a self-modifying program. Something I can look at and use to understand how self-modifying code should be written in the x86 assembly and how it works. Are there any resources that you can point me to, or any examples that you can give that will adequately demonstrate this?

I use NASM as my assembler.

EDIT: I also run this code on Linux.

Answer 1

Wow, that turned out to be a lot more painful than I expected. 100% of the pain was linux, protecting the program from overwriting and / or executing data.

The two solutions shown below. And a lot of googling was involved, so a few simple ones put a few bytes of commands and executed them, my mprotect and page alignment were rejected from Google search queries, which I had to study for this example.

Self-modifying code is straightforward, if you take a program or at least only two simple functions, compile and then parse it, you will get operation codes for these instructions. or use nasm to compile assembler blocks, etc. From this, I defined the opcode to load eax right away, then return.

Ideally, you just put these bytes in some ram and execute that ram. To force Linux to do this, you need to change the protection, which means that you need to send a pointer that is aligned on the mmap page. Therefore, select more than you need, find the aligned address within this selection, located on the page border, and mprotect from this address and use this memory to place your operation codes and then execute.

the second example takes an existing function compiled into a program, again, due to the protection mechanism, you cannot just point to it and change the bytes, you must protect it from writing. Thus, you need to back up to the previous call to the mprotect page with this address and enough bytes to cover the code that needs to be changed. Then you can change the bytes / operation codes for this function in any way (until you switch to any function that you want to continue to use) and execute it. In this case, you can see that fun() works, then I change it to just return the value, call it again and now it has been changed.

 #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/mman.h> unsigned char *testfun; unsigned int fun ( unsigned int a ) { return(a+13); } unsigned int fun2 ( void ) { return(13); } int main ( void ) { unsigned int ra; unsigned int pagesize; unsigned char *ptr; unsigned int offset; pagesize=getpagesize(); testfun=malloc(1023+pagesize+1); if(testfun==NULL) return(1); //need to align the address on a page boundary printf("%p\n",testfun); testfun = (unsigned char *)(((long)testfun + pagesize-1) & ~(pagesize-1)); printf("%p\n",testfun); if(mprotect(testfun, 1024, PROT_READ|PROT_EXEC|PROT_WRITE)) { printf("mprotect failed\n"); return(1); } //400687: b8 0d 00 00 00 mov $0xd,%eax //40068d: c3 retq testfun[ 0]=0xb8; testfun[ 1]=0x0d; testfun[ 2]=0x00; testfun[ 3]=0x00; testfun[ 4]=0x00; testfun[ 5]=0xc3; ra=((unsigned int (*)())testfun)(); printf("0x%02X\n",ra); testfun[ 0]=0xb8; testfun[ 1]=0x20; testfun[ 2]=0x00; testfun[ 3]=0x00; testfun[ 4]=0x00; testfun[ 5]=0xc3; ra=((unsigned int (*)())testfun)(); printf("0x%02X\n",ra); printf("%p\n",fun); offset=(unsigned int)(((long)fun)&(pagesize-1)); ptr=(unsigned char *)((long)fun&(~(pagesize-1))); printf("%p 0x%X\n",ptr,offset); if(mprotect(ptr, pagesize, PROT_READ|PROT_EXEC|PROT_WRITE)) { printf("mprotect failed\n"); return(1); } //for(ra=0;ra&lt;20;ra++) printf("0x%02X,",ptr[offset+ra]); printf("\n"); ra=4; ra=fun(ra); printf("0x%02X\n",ra); ptr[offset+0]=0xb8; ptr[offset+1]=0x22; ptr[offset+2]=0x00; ptr[offset+3]=0x00; ptr[offset+4]=0x00; ptr[offset+5]=0xc3; ra=4; ra=fun(ra); printf("0x%02X\n",ra); return(0); } 

Answer 2

Since you are writing a JIT compiler, you probably don't need self-modifying code, you want to generate executable code at runtime. These are two different things. Self-modifying code is code that changes after it is already running. Self-modifying code has a large performance limitation for modern processors and therefore would be undesirable for the JIT compiler.

Generating executable code at runtime should be a simple mmap () issue in some memory with permissions PROT_EXEC and PROT_WRITE. You can also call mprotect () on some memory that you have allocated for yourself, as dwelch did above.

Answer 3

You can also see projects such as GNU lightning . You give it the code for a simplified machine like RISC, and it generates the right machine dynamically.

A very real problem that you should think about is related to foreign libraries. You may need to support at least some system level calls / operations for your virtual machine. Kitsune tip is a good start to get you thinking about system level challenges. You are probably using mprotect to ensure that the modified memory becomes legally executable. (@KitsuneYMG)

Some FFI, which allows you to call dynamic libraries written in C, should be enough to hide many OS details. All these problems can affect your design a bit, so it's best to start thinking about them sooner.

Answer 4

A slightly simpler example based on the example above. Thanks dwelch a lot helped.

 #include <stdio.h> #include <string.h> #include <stdlib.h> #include <sys/mman.h> char buffer [0x2000]; void* bufferp; char* hola_mundo = "Hola mundo!"; void (*_printf)(const char*,...); void hola() { _printf(hola_mundo); } int main ( void ) { //Compute the start of the page bufferp = (void*)( ((unsigned long)buffer+0x1000) & 0xfffff000 ); if(mprotect(bufferp, 1024, PROT_READ|PROT_EXEC|PROT_WRITE)) { printf("mprotect failed\n"); return(1); } //The printf function has to be called by an exact address _printf = printf; //Copy the function hola into buffer memcpy(bufferp,(void*)hola,60 //Arbitrary size); ((void (*)())bufferp)(); return(0); } 

Answer 5

I have never written self-modifying code, although I have a general idea of ​​how this works. Basically, you write instructions that you want to execute into memory, and then go there. The processor interprets the bytes you wrote, the instructions, and (tries) to execute them. For example, viruses and anti-copy programs can use this technique.
Regarding system calls, you were right, arguments are passed through registers. For a reference to linux system calls and their argument, just check here .

Answer 6

This is written in the AT & T assembly. As you can see from the program, the result has changed due to self-modifying code.

Compilation: gcc -m32 modify.s modify.c

the -m32 option is used because the example works on 32-bit machines

Aessembly:

 .globl f4 .data f4: pushl %ebp #standard function start movl %esp,%ebp f: movl $1,%eax # moving one to %eax movl $0,f+1 # overwriting operand in mov instuction over # the new immediate value is now 0. f+1 is the place # in the program for the first operand. popl %ebp # standard end ret 

C test program:

  #include <stdio.h> // assembly function f4 extern int f4(); int main(void) { int i; for(i=0;i<6;++i) { printf("%d\n",f4()); } return 0; } 

Output:

 1 0 0 0 0 0