Installing pygit2/libgit2 in Python virtualenv on a Mac

Question

I think I'm about to lose my mind. I'm trying to simply install pygit2 in my python virtualenv on my macbook air so I can get to work. However this is not budging and I have been trying to set it up all day.

I'm supposed follow these instructions :

http://www.pygit2.org/install.html#how-to-install

I have tried countless amounts of things, messing around with the rpath because apparently on macs this is implemented differently :

How to set the runtime path (-rpath) of an executable with gcc under Mac OSX?

... and a countless amount of other variations, I'm just guessing at this point, however each time I try to build pygit2 it always leads to this result :

(testenv)emil ~/sites/env/testenv/pygit2 > which python
/Users/emil/Sites/env/testenv/bin/python
(testenv)emil ~/sites/env/testenv/pygit2 > python -c 'import pygit2'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "pygit2/__init__.py", line 32, in <module>
    import _pygit2
ImportError: dlopen(/Users/emil/sites/env/testenv/lib/python2.7/site-packages/_pygit2.so, 2): Symbol not found: _git_remote_fetchspec
  Referenced from: /Users/emil/sites/env/testenv/lib/python2.7/site-packages/_pygit2.so
  Expected in: flat namespace
 in /Users/emil/sites/env/testenv/lib/python2.7/site-packages/_pygit2.so

It doesn't seem to be linking the libraries properly :

(testenv)emil ~/sites/env/testenv/pygit2 > nm /Users/emil/sites/env/testenv/lib/python2.7/site-packages/_pygit2.so | grep _git | less

000000000000626c T _Repository_git_object_lookup_prefix
0000000000011288 d _Repository_git_object_lookup_prefix__doc__
                 U _git_blob_create_frombuffer
                 U _git_blob_create_fromdisk
                 U _git_blob_create_fromworkdir
                 U _git_blob_rawsize
                 U _git_checkout_head
                 U _git_checkout_index
                 U _git_checkout_tree
                 U _git_commit_author
                 U _git_commit_committer
                 U _git_commit_create
                 U _git_commit_free
                 U _git_commit_lookup
                 U _git_commit_lookup_prefix
                 ...

When I try to use pip it says :

(testenv)emil ~/sites/env/testenv/pygit2 > pip install pygit2
Requirement already satisfied (use --upgrade to upgrade): pygit2 in /Users/emil/sites/env/testenv/lib/python2.7/site-packages
Cleaning up...
(testenv)emil ~/sites/env/testenv/pygit2 > pip install pygit2 --upgrade
Requirement already up-to-date: pygit2 in /Users/emil/sites/env/testenv/lib/python2.7/site-packages
Cleaning up...

If someone has set this up before I would appreciate any help, otherwise I will use this to document the issue and organize my thoughts and hopefully record a solution.

Answer 1

I managed to get this setup up, in order to do it I had to read up on OSX's implementation of @rpath :

• https://blogs.oracle.com/dipol/entry/dynamic_libraries_rpath_and_mac
• http://www.mikeash.com/pyblog/friday-qa-2009-11-06-linking-and-install-names.html
• http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/install_name_tool.1.html

After reading these I proceeded with a clean install, like so :

libgit2

emil ~/Sites/env/testenv > export LIBGIT2=`pwd`
emil ~/Sites/env/testenv > git clone git://github.com/libgit2/libgit2.git -b v0.18.0

NOTE: Make sure you checkout the tag compatible with the current pygit2 version, at the moment you can use -b v0.18.0.

(testenv)emil ~/sites/env/testenv/libgit2 > mkdir build && cd build
(testenv)emil ~/sites/env/testenv/libgit2/build > cmake .. -DCMAKE_INSTALL_PREFIX=$LIBGIT2
(testenv)emil ~/sites/env/testenv/libgit2/build > cmake --build . --target install

NOTE: If you don't have cmake install it with Homebrew, with brew install cmake

Now we need to set the install name on the libgit2 library so that it can be found in a rpath.

(testenv)emil ~/sites/env/testenv/lib > otool -D libgit2.0.18.0.dylib 
libgit2.0.18.0.dylib:
libgit2.0.dylib
(testenv)emil ~/sites/env/testenv/lib > install_name_tool -id "@rpath/libgit2.0.18.0.dylib" libgit2.0.18.0.dylib 
(testenv)emil ~/sites/env/testenv/lib > otool -D libgit2.0.18.0.dylib 
libgit2.0.18.0.dylib:
@rpath/libgit2.0.18.0.dylib

pygit2

(testenv)emil ~/sites/env/testenv > git clone git://github.com/libgit2/pygit2.git
(testenv)emil ~/sites/env/testenv > cd pygit2/
(testenv)emil ~/sites/env/testenv/pygit2 > python setup.py build

NOTE: The OSX linker doesn't have some of the LDFLAGS recommended in the pygit2 tutorial so skip using them for now, you will set the rpath later.

(testenv)emil ~/sites/env/testenv/pygit2 > python setup.py install

Ok, now you will get an error like this because the rpath is not set up properly on the pygit2 library :

(testenv)emil ~/sites/env/testenv/pygit2 > python -c 'import pygit2'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "pygit2/__init__.py", line 32, in <module>
    import _pygit2
ImportError: dlopen(/Users/emil/sites/env/testenv/lib/python2.7/site-packages/_pygit2.so, 2): Library not loaded: @rpath/libgit2.0.18.0.dylib
  Referenced from: /Users/emil/sites/env/testenv/lib/python2.7/site-packages/_pygit2.so
  Reason: image not found

So let's set it :

(testenv)emil ~/sites/env/testenv/pygit2 > cd /Users/emil/sites/env/testenv/lib/python2.7/site-packages/
(testenv)emil ~/sites/env/testenv/lib/python2.7/site-packages > install_name_tool -add_rpath "@loader_path/../../" _pygit2.so 

Now test it, if there is no output everything went fine :

(testenv)emil ~/sites/env/testenv/lib/python2.7/site-packages > python -c 'import pygit2'

:)

Answer 2

The instructions on pygit2's site seem a bit misleading. Their default branch (master) binds targets libgit2's master branch (i.e. latest release, at this time 0.18.0), but libgit2's default branch is development. The meaning of "latest" on the site isn't clear.

The function the linker can't find has been removed since the last release, so it looks like you're simply building against the wrong libgit2 version. Use libgit2's master branch or its v0.18.0 tag and you should be fine.

Answer 3

I came accross a problem when installing libgit2 and pygit2 libraries for saltstack. The original installation followed the instructions on pygit2 doc, but under python 2.7.10, which promoted the errors like "no member named 'parent' in 'git_cert_hostkey'".

After googled a lot, Carlos Martín Nieto's words gave me a hint, and finally got libgit2 and pygit2 installed on OS X 10.11.4.

Install libgit2 with homebrew, pay attention the version you installed.

brew install libgit2

In my case, libgit2 0.23.4 was installed.

Build pygit2 with source code. Check the pygit2 version in changelog that matches the libgit2 you installed. Here I clone the source code with tag v0.23.3.

git clone https://github.com/libgit2/pygit2.git -b v0.23.3
cd pygit2
python setup.py build
python setup.py install

Finally, libgit2 and pygit2 are installed on you mac. Hope this may help.

Answer 4

Emil Davtyan's answer helped me. I was on Mac OS and am trying to use python3 with a later version of libgit, so had to modify it to the following

brew install python3
libgit_version = '0.24.0'
mkvirutalenv --python=`which python3` py3libgit
workon py3libgit
cd $VIRTUAL_ENV
export LIBGIT2=`pwd`
git clone git://github.com/libgit2/libgit2.git -b "v${libgit_version}"
cd libgit2 &&  mkdir build && cd build
cmake --build . --target install
cd $LIBGIT2/lib
install_name_tool -id "@rpath/libgit2.${libgit_version}.dylib" libgit2.${libgit_version}.dylib
otool -D libgit2.${libgit_version}.dylib

Install pygit

pip install pygit2==0.24.0

Note, ensure that python --version gives a value of Python 3.5.x for e.g Python 3.5.1. Assuming we have 3.5, the following applies

cd $VIRTUAL_ENV/lib/python3.5/site-packages
install_name_tool -add_rpath "@loader_path/../../" _pygit2.cpython-35m-darwin.so
otool -L _pygit2.cpython-35m-darwin.so
python -c 'import pygit2'