Exit the application on the OS without sharing memory

Question

I am writing a monolithic OS (it is a joke to call it an OS, but it has very minimal school-level functionalities). When I say monolithic, I meant that it was compiled as a single binary blob and does not support the file system, etc. Currently, I just have a rudimentary simple user space, which is nothing more than an infinite while loop.

I plan to make my OS a little more useful and I want to write custom applications that may stop, like regular applications on a full-blown OS.

I have no glibc or equivalent. My current library in user space is the code I wrote. Now my problem is how to add a framework for user space applications, which will allow them to stop working at a fixed point.

I know how programs compile on regular systems and what happens when a program terminates. However, in my case, I don’t have the luxury of compiling programs against libraries, and if the program terminates, my instruction pointer just goes wildly around.

I am currently doing all applications to make a "callback", and I am pre-populating the application stack with the patch address (at startup). Is there a better way to deal with the problem?

Along with the answer, I would be more than happy to get clarity about some OS concepts.

I am working on an x86 platform emulator and set up a static binary table. (I have virtual memory support)

Answer 1

Manually processing the first frame of the stack and returning to any cleanup code for the process you need to start seems like a perfectly reasonable method. If your OS has "syscalls", then the user-space process cleanup code (possibly called exit() ) probably ends with a call to syscall _exit() . You still need to handle the case where a program tries to execute code in "la-la land" because it can still happen (however, it does before you have a page protection system, this can be a difficult problem).