forkIO creates a light, unbound green thread. Green threads have very little overhead; The GHC runtime is able to efficiently multiplex millions of green threads through a small pool of OS workflows. A green thread can live on more than one OS thread during its existence.
forkOS creates a linked thread: a green thread for which FFI calls are guaranteed to be executed in the same fixed OS thread. Linked streams are commonly used when interacting with C libraries that use local streaming data and expect all API calls to come from a single stream. From a document defining associated GHC flows:
The idea is that each associated Haskell thread has a dedicated OS thread. It is guaranteed that any FFI calls made by the associated Haskell thread are created by the associated OS thread, although pure-Haskell execution can, of course, be performed by any OS thread. Group Thus, external calls can be guaranteed by the same OS if all of them are performed in one connected Haskell thread.
[...]
[F] or each OS thread, there is no more than one associated Haskell thread.
Note that the above quote does not exclude the possibility that the OS thread associated with the linked thread may act as a worker for unrelated Haskell threads. Nor does it guarantee that the associated non-FFI code stream will execute on any particular thread.
forkProcess creates a new process, like fork on UNIX.