Analyze the back trace of a failure due to a failed library

Question

In my application, I have a setup signal handler for capturing Segfaults and printing bactraces. My application downloads some plugin libraries when the process begins.

If my application crashes using segfault, due to an error in the main binary executable, I can analyze the back trace with

addr2line -Cif -e ./myapplication 0x4... 

It accurately displays the function and source_file file: line_no

However, how to analyze whether an error occurs due to an error in the plugin, as in the opposite direction below?

 /opt/myapplication(_Z7sigsegvv+0x15)[0x504245] /lib64/libpthread.so.0[0x3f1c40f500] /opt/myapplication/modules/myplugin.so(_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi+0x6af)[0x7f5588fe4bbf] /opt/myapplication/modules/myplugin.so(_Z11myplugin_reqmodP12CONNECTION_TP7Filebuf+0x68)[0x7f5588fe51e8] /opt/myapplication(_ZN10Processors7ExecuteEiP12CONNECTION_TP7Filebuf+0x5b)[0x4e584b] /opt/myapplication(_Z15process_requestP12CONNECTION_TP7Filebuf+0x462)[0x4efa92] /opt/myapplication(_Z14handle_requestP12CONNECTION_T+0x1c6d)[0x4d4ded] /opt/myapplication(_Z13process_entryP12CONNECTION_T+0x240)[0x4d79c0] /lib64/libpthread.so.0[0x3f1c407851] /lib64/libc.so.6(clone+0x6d)[0x3f1bce890d] 

Both application and plugin libraries were compiled using gcc and are not included. My application, when executed, downloads plugin.so using dlopen. Unfortunately, a crash occurs on a site where I cannot run the application under gdb.

I was desperately looking for an answer, but all sites discussing backtrace and addr2line exclude scenarios where the analysis of erroneous plugins may be required. I hope that some kind hack knows the solution to this dilemma and can share some thoughts. That would be so invaluable to other programmers.

Tons of thanks in advance.

Answer 1

Here are some tips to help you debug this:

The address in your reverse direction is the address in the address space of the process at the time of its failure. This means that if you want to translate it into a "physical" address relative to the beginning of the .text section of your library, you must subtract the starting address of the corresponding pmap section from the address of your trace.

Unfortunately, this means that you need the pmap process before it crashes. I admittedly have no idea if the loading addresses for libraries on the same system are permanent if you close and run it (in fact, there are security features that randomize this), but as you noticed, it is of course Not portable across systems.

In your position, I would try:

• Rename symbol names using c++filt -n or manually. Now I don’t have a shell, so here is my manual attempt: _ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi has ICAPSection::process(CONNECTION_T *, Filebuf *, int) . This may be helpful. If not:
• use objdump or nm (I'm sure they can do this) to find the address matching the name you are looking for, then add the offset ( +0x6af to fit your glass), then look up the resulting address with addr2line .

Answer 2

The us2012 answer was a pretty trick needed to solve the problem. I'm just trying to reformulate it here to help any other newbie, struggling with the same problem, or if someone wants to suggest improvements.

Backtrace clearly shows that a flaw exists in the myplugin.so code. And the return line indicates that it exists:

 /opt/myapplication/modules/myplugin.so(_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi+0x6af)[0x7f5588fe4bbf] 

The problem of placing a line corresponding to this error cannot be defined as simplified as:

 addr2line -Cif -e /opt/myapplication/modules/myplugin.so 0x7f5588fe4bbf 

The correct procedure here would be to use nm or objdump to determine the address pointing to a malformed name. (Replacing how we made2012 is not really needed). Thus, using:

 nm -Dlan /opt/myapplication/modules/myplugin.so | grep "_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi" 

I get:

 0000000000008510 T _ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi /usr/local/src/unstable/myapplication/sources/modules/myplugin/myplugin.cpp:518 

It is interesting to note that myplugin.cpp: 518 actually points to the line where the opening of the "{" function of ICAPSection :: process (CONNECTION_T *, Filebuf *, int)

Next, we add 0x6af to the address (detected by nm output above) 0000000000008510 using the linux shell command

  printf '0x%x\n' $(( 0x0000000000008510 + 0x6af )) 

And this leads to 0x8bbf

And this is the actual source_file: line_no file of the faulty code and can be precisely defined using addr2line as:

 addr2line -Cif -e /opt/myapplication/modules/myplugin.so 0x8bbf 

What is displayed:

 std::char_traits<char>::length(char const*) /usr/include/c++/4.4/bits/char_traits.h:263 std::string::assign(char const*) /usr/include/c++/4.4/bits/basic_string.h:970 std::string::operator=(char const*) /usr/include/c++/4.4/bits/basic_string.h:514 ?? /usr/local/src/unstable/myapplication/sources/modules/myplugin/myplugin.cpp:622 

I'm not too sure why the function name is not displayed here, but myplugin.cpp: 622 was exactly where the error was.