Cython not finding shared library

Question

My issue started identical to this one: Python executable not finding libpython shared library

I updated .bashrc with export LD_LIBRARY_PATH=$HOME/local/lib/python/2.7.6/lib and things were fine. Python works, and I installed pip. But now, I'm running into something similar when installing cython with pip. I get this error message when I execute pip install cython:

gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip_build/cython/Cython/Plex/Scanners.o -L. -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Plex/Scanners.so

/usr/bin/ld: cannot find -lpython2.7

collect2: ld returned 1 exit status

error: command 'gcc' failed with exit status 1

I cannot add $HOME/local/lib/python/2.7.6/lib to /etc/ld.so.conf and run ldconfig as I do not have root. I was under the impression that setting the LD_LIBRARY_PATH was the way around this, but this appears to not be true for compilation. Is there a way to get the compiler to see this local library without running root commands?

Answer 1

Update:

The LD_LIBRARY_PATH is only used by the dynamic loader at runtime, not at build time, so that is not the issue. The issue is that you forgot to put the -L/path/to/pylib before the -l. I've never had to use LIBRARY_PATH because a build requires path extension that is specific to a given build, so you never set LIBRARY_PATH you just use -L. You would only set if if you are going to regularly do builds that use a specific library, and even then I find it better to use -L because sooner or later this will cause linker to find the wrong lib and by then you will have forgotten that it's because LIBRARY_PATH is set permanently.

There are many ways to set -L values in a build: if you run the compiler from command line you don't need that env var, you just specify as many -L as required as part of the command; if you use a makefile, you edit whatever make variable you are using, such as CFLAGS or other, different platforms have different conventions. So whereas setting -L directly will always work, setting CFLAGS will only work if that is the variable used by the makefile.

Now this is a python installation so where to set this may not be obvious, but I am sure there is another way than setting LIBRARY_PATH. In principle any python package you install, if it involves compilation of C++ modules, could require edit of the setup.py to set library paths. For example

Extension(...,
          library_dirs=['/usr/X11R6/lib'],
          ...)

Since you mention nympy, another place to set this might be in site.cfg (see Supplying NumPy site.cfg arguments to pip).

Old (wrong) answer:

Set your LD_LIBRARY_PATH in your bash console. If this doesn't work then it's because you have the wrong path: check by echoing the environment var.

Once you get that to work, edit your .bashrc or .profile then exit your shell and restart it. Echo the env var to verify that contains the part you added.

Also, ensure that you are appending to the path rather overwriting it:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/...
export LD_LIBRARY_PATH

Because python lib might depend on .so in other folders, if the linker can't find them it may appear as though it is the python lib that was not found. This is not explained on the page you linked to in your question.

Answer 2

OK after some more digging I found this: LD_LIBRARY_PATH vs LIBRARY_PATH

Setting LIBRARY_PATH to the same path as LD_LIBRARY_PATH made the compiler aware of the python lib. cython/numpy/scipy all built and installed no problem afterwords.