Debugging a running daemon using gdb

Question

I am developing a high bandwidth C server application that works like a daemon. In some cases, the application crashes (always without a kernel). How can I debug a running daemon using gdb to find the location that SIGSEGV generates?

Explanatory notes:

• I know how to connect using gdb to a running process using the attach command

• After joining the process, it stops. If I run continue, gdb will remain locked if the program does not work. If I press CTRL-C, the process ends and I cannot just disconnect gdb.

So the question is this: is there a way to continue the process without gdb freezing, but can it disconnect if the process does not work?

Answer 1

Try asynchronous mode and continue with & ":

Save below non-stop.gdb

set target-async on set pagination off set non-stop on 

Then run:

 $ gdb -x non-top.gdb (gdb) !pgrep YOUR-DAEMON 1234 (gdb) attach 1234 (gdb) continue -a & (gdb) 

Answer 2

This attach / detach page says that the detach command will work inside gdb .

If you want to catch the segmentation error in the application, you will have to start the application from the debugger. Then, when the signal is caught, you can use where or bt to see the application stack trace. Of course, you cannot continue the application after its failure, how to restore it? If you plan to cause an error soon, you can connect to the running process and wait for the error in the debugger again.

If you need a stack trace after an error occurs, you really need the main file, as there will be no process to attach. Now, if your daemon is running as part of the system, it may be difficult to get the configuration for a kernel dump, and you may also not want other applications to leave kernel dumps everywhere. Therefore, I would advise you to stop the system daemon and run it again in your user space, then you can let it reset the kernel. If it is really important that it starts as part of the system, then see if only one sub-shell starts running the daemon and use ulimit -c in this sub-shell to set the appropriate maximum size for the main dump.

Answer 3

Another way to debug your application is to use the main file for debugging using GDB.

To create the main file during segmentation, you can follow these steps:

1) Copy the parameters below into your script that starts the daemon.

 ulimit -c unlimited mkdir -p <path_to_core_file>, eg : /etc/user/ankit/corefiles chmod 777 /etc/user/ankit/corefiles echo "/etc/user/ankit/corefiles/%e.%s.core" > /proc/sys/kernel/core_pattern 

2) Launch your application using a script and wait for the kernel dump file to be created. Once you get a kernel dump, you can debug it using gdb by doing the following:

3) Getting back trace with GDB

 gdb -c <core_file>, where core_file is the file generated after segmentation fault 

4) Reverse trace

Next, we want to know what the stack is when the program crashed. Running bt at the gdb prompt will give you backtrace. If GDB did not load the characters for the binary, it will give an error with a question mark, similar to this "??????". To fix this, you will need to download the characters.

Here's how to load debugging symbols.

 symbol-file /path/to/binary sharedlibrary 

5) Get BackTrace for all threads

 thread apply all bt full 

NOTE. Make sure the binary is composed using debug symbols.