shared library locations for matlab mex files:
Question
I am trying to write a matlab mex function which uses libhdf5; My Linux install provides libhdf5-1.8 shared libraries and headers. However, my version of Matlab, r2007b, provides a libhdf5.so from the 1.6 release. (Matlab .mat
files bootstrap hdf5, evidently). When I compile the mex, it segfaults in Matlab. If I downgrade my version of libhdf5 to 1.6 (not a long-term option), the code compiles and runs fine.
question: how do I solve this problem? how do I tell the mex compilation process to link against /usr/lib64/libhdf5.so.6 instead of /opt/matlab/bin/glnxa64/libhdf5.so.0 ? When I try to do this using -Wl,-rpath-link,/usr/lib64
in my compilation, I get errors like:
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../x86_64-pc-linux-gnu/bin/ld: warning: libhdf5.so.0, needed by /opt/matlab/matlab75/bin/glnxa64/libmat.so, may conflict with libhdf5.so.6
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
mex: link of 'hdf5_read_strings.mexa64' failed.
make: *** [hdf5_read_strings.mexa64] Error 1
ack. the last resort would be to download a local copy of the hdf5-1.6.5 headers and be done with it, but this is not future proof (a Matlab version upgrade is in my future.). any ideas?
EDIT: per Ramashalanka's excellent suggestions, I
A) called mex -v
to get the 3 gcc
commands; the last is the linker command;
B) called that linker command with a -v
to get the collect
command;
C) called that collect2 -v -t
and the rest of the flags.
The relevant parts of my output:
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/crti.o
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/crtbeginS.o
hdf5_read_strings.o
mexversion.o
-lmx (/opt/matlab/matlab75/bin/glnxa64/libmx.so)
-lmex (/opt/matlab/matlab75/bin/glnxa64/libmex.so)
-lhdf5 (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/libhdf5.so)
/lib64/libz.so
-lm (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/libm.so)
-lstdc++ (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/libstdc++.so)
-lgcc_s (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/libgcc_s.so)
/lib64/libpthread.so.0
/lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2
-lgcc_s (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/libgcc_s.so)
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/crtendS.o
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/crtn.o
So, in fact the libhdf5.so
from /usr/lib64
is being referenced. However, this is being overriden, I believe, by the environment variable LD_LIBRARY_PATH
, which my version of Matlab automagically sets at run-time so it can locate its own versions of e.g. libmex.so
, etc.
I am thinking that the crt_file.c
example works either b/c it does not use the functions I am using (H5DOpen
, which had a signature change in the move from 1.6 to 1.8 (yes, I am using -DH5_USE_16_API
)), or, less likely, b/c it does not hit the parts of Matlab internals that need hdf5. ack.
Solution
The following worked on my system:
Install hdf5 version 1.8.4 (you've already done this: I installed the source and compiled to ensure it is compatible with my system, that I get gcc versions and that I get the static libraries - e.g. the binaries offered for my system are
icc
specific).Make a target file. You already have your own file. I used the simple
h5_crtfile.c
from here (a good idea to start with this simple file first a look for warnings). I changedmain
tomexFunction
with the usual args and includedmex.h
.Specify the static 1.8.4 library you want to load explicitly (the full path with no -L for it necessary) and don't include
-lhdf5
in theLDFLAGS
. Include a-t
option so you can ensure that there is no dynamic hdf5 library being loaded. You also need-lz
, withzlib
installed. For darwin we also need a-bundle
inLDFLAGS
:mex CFLAGS='-I/usr/local/hdf5/include' LDFLAGS='-t /usr/local/hdf5/lib/libhdf5.a -lz -bundle' h5_crtfile.c -v
For linux, you need an equivalent position-independent call, e.g.
fPIC
and maybe-shared
, but I don't have a linux system with a matlab license, so I can't check:mex CFLAGS='-fPIC -I/usr/local/hdf5/include' LDFLAGS='-t /usr/local/hdf5/lib/libhdf5.a -lz -shared' h5_crtfile.c -v
Run the
h5_crtfile
mex file. This runs without problems on my machine. It just does a H5Fcreate and H5Fclose to create "file.h5" in the current directory, and when I callfile file.h5
I getfile.h5: Hierarchical Data Format (version 5) data
.
Note that if I include a -lhdf5
above in step 3, then matlab aborts when I try to run the executable (because it then uses matlab's dynamic libraries which for me are version 1.6.5), so this is definitely solving the problem on my system.
Thanks for the question. My solution above is definitely much easier for me than what I was doing before. Hopefully the above works for you.
OTHER TIPS
I am accepting Ramashalanka's answer because it led me to the exact solution which I will post here for completeness only:
- download the hdf5-1.6.5 library from the hdf5 website, and install the header files in a local directory;
- tell mex to look for "hdf5.h" in this local directory, rather than in the standard location (e.g.
/usr/include
.) - tell mex to compile my code and the shared object library provided by matlab, and do not use the
-ldfh5
flag inLDFLAGS
.
the command I used is, essentially:
/opt/matlab/matlab_default/bin/mex -v CC#gcc CXX#g++ CFLAGS#"-Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include" CXXFLAGS#"-Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include " -O -lmwblas -largeArrayDims -L/usr/lib64 hdf5_read_strings.c /opt/matlab/matlab_default/bin/glnxa64/libhdf5.so.0
this gets translated by mex into the commands:
gcc -c -I/opt/matlab/matlab75/extern/include -DMATLAB_MEX_FILE -Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include -O -DNDEBUG hdf5_read_strings.c
gcc -c -I/opt/matlab/matlab75/extern/include -DMATLAB_MEX_FILE -Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include -O -DNDEBUG /opt/matlab/matlab75/extern/src/mexversion.c
gcc -O -pthread -shared -Wl,--version-script,/opt/matlab/matlab75/extern/lib/glnxa64/mexFunction.map -Wl,--no-undefined -o hdf5_read_strings.mexa64 hdf5_read_strings.o mexversion.o -lmwblas -L/usr/lib64 /opt/matlab/matlab_default/bin/glnxa64/libhdf5.so.0 -Wl,-rpath-link,/opt/matlab/matlab_default/bin/glnxa64 -L/opt/matlab/matlab_default/bin/glnxa64 -lmx -lmex -lmat -lm -lstdc++
this solution should work on all my various target machines and at least until I upgrade to matlab r2009a, which I believe uses hdf5-1.8. thanks for all the help, sorry for being so dense with this--I think I was overly-committed to using the packaged version of hdf5, rather than a local set of header files.
Note this would all have been trivial if Mathworks had provided a set of the header files with the Matlab distribution...