The debugger has knowledge about where the code for functions in your program starts and ends, either because this information is provided when debugging data, or because it uses any external characters visible in the executable file to provide elementary information.
When the stack is in the correct state, it contains the return address of the calling function and, somewhere above it, the return address for the higher level calling function, etc. While you are executing various debugger commands, it uses these return addresses (and other information about the stack and the state of the process) to show you the names of these functions. This requires finding the return address in the debugger's knowledge of where the functions are located.
Once you overflow the buffer and corrupt the stack, the correct return address will be destroyed. Instead, you have a different address (one indicates your shellcode if your exploit worked). When the debugger tries to figure out which function is in this address, it fails because the address is not in any of the functions in your program.
When this error occurs, the debugger prints the error message that you see.
Usually, the debugger can still perform the basic functions: it can display registers and memory in your program, it can still execute single-stage and set breakpoints, etc. He will have problems with what requires a more complex interpretation: he cannot understand where the frames of the stack are, he cannot find local variables by name, etc.
source share