In Linux / MMU (in! MMU you cannot grow a stack), the stack grew in a page error handler. For x86, whether to grow the stack is determined by the following code from arch/x86/mm/fault.c:do_page_fault() :
if (error_code & PF_USER) { if (unlikely(address + 65536 + 32 * sizeof(unsigned long) < regs->sp)) { bad_area(regs, error_code, address); return; } } if (unlikely(expand_stack(vma, address))) { bad_area(regs, error_code, address); return; }
expand_stack() checks the usual RLIMITS (RLIMIT_AS, RLIMIT_STACK, RLIMIT_MEMLOCK), whether LSM will allow to grow the stack, whether to recompile too much, etc., and finally increase the stack.
source share