Executable cannot find dynamically linked mkl library, but ldd does

Question

I have a code which was dynamically linked with the mkl library. when running the code, it reports mkl not found.

./bmdl
/g/software/EMTO/5.7/intel_12.1/ser/bin/bmdl: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory

But when I use ldd to check the dynamically linked libraries in the executable, it shows the mkl library is found

ldd bmdl
libmkl_intel_lp64.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_intel_lp64.so (0x00002b975d76d000)
libmkl_sequential.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_sequential.so (0x00002b975df53000)
libmkl_core.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_core.so (0x00002b975e631000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003785600000)
libm.so.6 => /lib64/libm.so.6 (0x0000003784e00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003784a00000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000378a600000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003785200000)
/lib64/ld-linux-x86-64.so.2 (0x0000003784600000)

Any idea what could be wrong?

Output from readelf -l ./bmdl

Elf file type is EXEC (Executable file)
Entry point 0x4034b0
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001c0 0x00000000000001c0  R E    8
  INTERP         0x0000000000000200 0x0000000000400200 0x0000000000400200
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x00000000000e4eb4 0x00000000000e4eb4  R E    200000
  ....

More debugging

$ export LD_DEBUG=libs,files
$ ./bmdl
./bmdl: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
$ ldd ./bmdl
     15133: 
     15133: file=libtermcap.so.2 [0];  needed by /bin/sh [0]
     15133: find library=libtermcap.so.2 [0]; searching
     15133:  search path=/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/tls/x86_64:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/tls:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/x86_64:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/tls/x86_64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/tls:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/x86_64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64        (LD_LIBRARY_PATH)
     ....      

Seems LD_DEBUG doesn't have effect on running ./bmdl alone.

I just realized the old bmdl has a 'setgid' flag and my new built doesn't have it. Maybe that was the cause?

-rwxr-sr-x 1 root gants 1123992 Jul 23 16:14 /scratch/helpdesk/bmdl

I removed the setgid bit from the old bmdl and running ./bmdl doesn't complain about library not found. Now the question is why setgid can interfere with dynamically linked library?

It happens that setgid with a dynamically linked executable can cause security problems and is greatly limited by GNU glibc. For example, LD_LIBRARY_PATH will be ignored. Maybe the old built never worked before?!

Answer 1

The code is dynamically linked and has the setgid attribute. setgid with a dynamically linked executable can cause security problems and is greatly limited by GNU glibc. For example, LD_LIBRARY_PATH will be ignored. That is why the code keeps on complaining some shared library not found.

Answer 2

Using MKL reqires env variables including INCLUDE, MKLROOT, LD_LIBRARY_PATH, LIBRARY_PATH, CPATH, FPATH and NLSPATH being properly set.

This can be done by a single script provided by Intel.

If you use intel compiler,

  $ source ${intel_dir}/bin/compilervars.sh intel64

If you use MKL only with gcc compiler,

  $ source ${intel_dir}/mkl/bin/mklvars.sh intel64

You can add this cmd line to your .bashrc so you don't need to run it every time.

Answer 3

But when I use ldd to check the dynamically linked libraries in the executable, it shows the mkl library is found

It is highly improbable that you've told us the whole story, because ldd (on Linux) is just a small shell script around ld-linux.so. If ld-linux.so can find the shared library when invoked by ldd, ld-linux.so should also be able to do that when the executable is invoked directly (ld-linux.so is what actually maps shared libraries when you run a.out).

The only plausible explanations I can think of:

• you execute bmdl in a different environment from the one you execute ldd in, or
• you have a modified ldd on your PATH, which perhaps modifies the environment before running "real" ldd.

How do I know which linker is used?

readelf -l bmdl

and look for "Requesting program interpreter".

[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2].

Very strange.

The next debugging suggestion: set LD_DEBUG=libs,files and see where ld-linux is searching. You can do that for both ldd and bmdl, and see where the difference comes from.