To give an answer that makes sense, we’ll first look at some concepts.
Most modern processors have a memory management module (MMU), which is used for a number of purposes.
One goal is to match the virtual address (the one that the processor sees) and the physical address (where the chips are actually connected). This is called address translation.
Another goal is to set access attributes for specific locations in virtual memory (things like Read-Write or Read-Only memory are not available)
With the MMU, you can have the so-called "unity mapping" where the virtual address of the processor matches the physical address (i.e. you do not use address translation). For example, if a processor accesses 0x10000, it refers to a physical location of 0x10000.
A "flat" memory model usually refers to the fact that any virtual address that the CPU accesses is unique. Thus, for a 32-bit CPU, you are limited to a maximum of 4G address space.
Most often (although not necessarily) is used to indicate the only correspondence between virtual and physical memory.
In contrast, in the world of workstations, most operating systems (Linux / Windows) use an “overlapping” memory model. For example, any program (process) you run on Windows will have a starting address of 0x10000.
How can windows handle 10 processes from address 0x10000?
This is because each process uses an MMU to map a 0x10000 virtual address to different physical addresses. For P1, it can be 0x10000 = 0x10000, and P2 can be 0x10000 = 0x40000, etc.
In RAM, programs are located on different physical addresses, but the virtual address space of the CPU looks the same for each process.
As far as I know, Windows and Standard Linux always use an overlapping model (i.e. they don't have a flat model). It is possible that uLinux or another special kernel may have a flat model.
Now protection has nothing to do with a flat and protected model. I would say that most overlapping model OSs will use protection so that one process does not affect (i.e., writes to memory) another.
With VxWorks 6.x and the implementation of real-time processes, even with a flat-memory model, individual RTP files are protected from each other (and kernel applications) with protection.
If you do not use RTP and run everything in the vxWorks kernel, then protection is not used.
So, how does protection work (whether in VxWorks RTP or other OSs)? In essence, RTP / Process exists inside a “memory bubble” with a specific range of (virtual) addresses that contains code, data, a bunch, and another sorted memory cell.
If RTP / Process tries to access a memory cell outside its bubble, the MMU throws an exception and the OS (or signal handler) is called. A typical result is segment exclusion / bus exclusion.
But how can a process send a packet to an Ethernet port if it cannot escape its memory? It depends on the processor architecture, but essentially the user-side socket library (RTP) (for example) makes a “system call” —this is a special instruction that switches the processor to kernel space and supervisor mode. At the moment, some device driver (which is usually located in the kernel) is launched to output data to some hardware device. Once this is done, the system call will return, and we will return to the RTP / process space in which the user code is executed.
The OS takes care of all MMU programming, handling system calls, etc. This is invisible to the application.