The answer will depend on the target architecture and specific OS. Since the question is marked as Linux, you are fairly biased in asking a question that at first glance seems more general.
In a complex OS or RTOS, such as Linux or QNX Neutrino, with support for MMU protection, memory protection mechanisms such as the previously mentioned protection pages can be used. Such OSs require, of course, goals with an MMU.
Simpler operating systems and typical RTOS scheduling kernels without MMU support can use a number of methods. The simplest thing is to place a protective signature at the top of the stack, which is checked for modification when the scheduler starts. This is a bit-and-miss, it requires that the stack overflow actually changes the signature, and that the damage received does not cause a crash before the next scheduler starts. Some systems with built-in debugging resources may place an access breakpoint in the signature word and throw an exception when it hits.
In development, a common method is to first populate each stream stack with a signature and periodically check the stream for a "high level" and issue a warning if it exceeds a certain percentage level.