Why is my Linux application pulling in the wrong library?

Question

I have an application that I create using the NetCDF C ++ library, and NetCDF pulls out the HDF-4 library. However, he pulled the wrong HDF-4 library.

Here's how my application is connected:

/apps1/intel/bin/icpc -gxx-name=/apps1/gcc-4.5.0/bin/g++ -shared -o lib/libMyCustom.so -Llib -L/apps1/boost-1.48.0/lib -Wl,-rpath=/apps1/boost-1.48.0/lib -L/apps1/gdal-1.8.0-jasper/lib -Wl,-rpath=/apps1/gdal-1.8.0-jasper/lib -L/new_apps1/hdf4/lib -Wl,-rpath=/new_apps1/hdf4/lib -L/new_apps1/netcdf/lib -Wl,-rpath=/new_apps1/netcdf/lib -lboost_system -lboost_serialization -lboost_date_time -lboost_thread -lgdal -ldf -lmfhdf -lnetcdf_c++ MyProj/obj/ProjUtility.o MyProj/obj/ProjMetadataException.o MyProj/obj/ProjTimestampUtil.o 

I set my LD_LIBRARY_PATH very shortly:

 LD_LIBRARY_PATH=/new_apps1/hdf4/lib:/new_apps1/hdf5/lib: /apps1/intel/composerxe/lib/intel64:/apps1/gcc-4.5.0/lib64:/apps1/gcc-4.5.0/lib 

And here is an excerpt from the output of ldd -v:

  libdf.so.0 => /new_apps1/hdf4/lib/libdf.so.0 (0x00002af5baabc000) libmfhdf.so.0 => /new_apps1/hdf4/lib/libmfhdf.so.0 (0x00002af5bad61000) libnetcdf_c++.so.5 => /new_apps1/netcdf/lib/libnetcdf_c++.so.5 (0x00002af5baf85000) libhdf5.so.6 => /new_apps1/hdf5/lib/libhdf5.so.6 (0x00002af5bd1e7000) libgif.so.4 => /usr/lib64/libgif.so.4 (0x0000003a6bc00000) libpng12.so.0 => /usr/lib64/libpng12.so.0 (0x0000003a71000000) libnetcdf.so.6 => /new_apps1/netcdf/lib/libnetcdf.so.6 (0x00002af5bd682000) libhdf5_hl.so.6 => /new_apps1/hdf5/lib/libhdf5_hl.so.6 (0x00002af5be272000) /new_apps1/hdf4/lib/libdf.so.0: libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6 libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6 /new_apps1/hdf4/lib/libmfhdf.so.0: libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6 /new_apps1/netcdf/lib/libnetcdf_c++.so.5: libgcc_s.so.1 (GCC_3.0) => /apps1/gcc-4.5.0/lib64/libgcc_s.so.1 libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6 libstdc++.so.6 (CXXABI_1.3) => /apps1/gcc-4.5.0/lib64/libstdc++.so.6 libstdc++.so.6 (GLIBCXX_3.4) => /apps1/gcc-4.5.0/lib64/libstdc++.so.6 /new_apps1/hdf5/lib/libhdf5.so.6: libm.so.6 (GLIBC_2.2.5) => /lib64/libm.so.6 libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6 libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6 /new_apps1/netcdf/lib/libnetcdf.so.6: libm.so.6 (GLIBC_2.2.5) => /lib64/libm.so.6 libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6 libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6 /new_apps1/hdf5/lib/libhdf5_hl.so.6: libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6 

So far, everything in LD_LIBRARY_PATH, rpath, and ldd has indicated that this points to the HDF I want to reference (/new_apps1/hdf4/lib/libmfhdf.so.0). But when I run, Valgrind tells me that he is dying in the HDF-4 OLD library (probably this is segfault for some reason), instead of the HDF-4 library that I am trying to link to:

  Invalid read of size 4 at 0x67CF765: NC_var_shape (in /apps1/hdf-4.2.6/lib/libmfhdf.so.0.0.0) by 0x91327CA: nc_get_NC (v1hpg.c:1113) by 0x91303C0: l3nc__open_mp (nc.c:1096) by 0x915B279: nc3d__open_mp (dapdispatch3.c:336) by 0x914A752: nc3d_open (ncdap3.c:94) by 0x911F8A2: l4nc_open_file (nc4file.c:2338) by 0x916A290: nc4d_open_file (ncdap4.c:122) by 0x911CDDF: nc__open (nc4file.c:2407) by 0x69E85F8: NcFile::NcFile(char const*, NcFile::FileMode, unsigned long*, unsigned long, NcFile::FileFormat) (netcdf.cpp:384) by 0x710F0B8: getData(std::string const&) (ProjTimestampUtil.cc:593) by 0x70E9BEA: (anonymous namespace)::parseOptions(int, char**) (ProjUtility.cc:190) by 0x70EAAFB: main(int, char**) (ProjUtility.cc:243) Address 0x1051 is not stack'd, malloc'd or (recently) free'd Process terminating with default action of signal 11 (SIGSEGV) Access not within mapped region at address 0x1051 at 0x67CF765: NC_var_shape (in /apps1/hdf-4.2.6/lib/libmfhdf.so.0.0.0) by 0x91327CA: nc_get_NC (v1hpg.c:1113) by 0x91303C0: l3nc__open_mp (nc.c:1096) by 0x915B279: nc3d__open_mp (dapdispatch3.c:336) by 0x914A752: nc3d_open (ncdap3.c:94) by 0x911F8A2: l4nc_open_file (nc4file.c:2338) by 0x916A290: nc4d_open_file (ncdap4.c:122) by 0x911CDDF: nc__open (nc4file.c:2407) by 0x69E85F8: NcFile::NcFile(char const*, NcFile::FileMode, unsigned long*, unsigned long, NcFile::FileFormat) (netcdf.cpp:384) by 0x710F0B8: getData(std::string const&) (ProjTimestampUtil.cc:593) by 0x70E9BEA: (anonymous namespace)::parseOptions(int, char**) (ProjUtility.cc:190) by 0x70EAAFB: main(int, char**) (ProjUtility.cc:243) 

Where else does my application get path information when dynamically pulling in other libraries?

Answer 1

I'm not quite sure about the details of -rpath and LD_LIBRARY_PATH, and their priority, but I found some useful environment variables:

• LD_DEBUG=all - This env variable enables detailed debugging of dynamic linkers. Now running ldd in your application will lead to the conclusion about the details of how all its dependencies find their dependencies.
• LD_DEBUG_OUTPUT=<filename_prefix> - Used in conjunction with LD_DEBUG to specify output files for recording debug information.

The variable LD_DEBUG env helped me track that / apps 1 / gdal-1.8.0-jasper / lib / libgdal.so.1 was compiled with the -rpath parameter, which pulled out the old (wrong) versions of my library. He gave this useful debug output:

 search path=/pathXYZ/lib/tls/x86_64:/pathXYZ/lib/tls:/pathXYZ/lib/x86_64: /pathABC/jasper/lib:/pathABC/hdf5/lib/tls/x86_64:/pathABC/hdf5/lib/tls: /pathABC/hdf5/lib/x86_64:/pathABC/hdf5/lib:/pathABC/netcdf/lib/tls/x86_64: /pathABC/netcdf/lib/tls:/pathABC/netcdf/lib/x86_64:/pathABC/netcdf/lib (RPATH from file /apps1/gdal-1.8.0-jasper/lib/libgdal.so.1) 

So the rpath of how the GDAL library was compiled seems to end my LD_LIBRAR_PATH. Until I can get my lab to rebuild libgdal correctly, I found this env var that helped me load the versions of the "right" library you need:

• LD_PRELOAD=<path/to/libName.so> - point this to the place of the library (or a list of libraries separated by spaces), which should be loaded before all the others. See the ld.so man page .