Should python-dev be required to install pip

Question

I find that many people have troubles installing python packages with pip because python-dev is not installed. Mainly, the error:

fatal error: Python.h: No such file or directory

So the question: Should python-dev be a required dependency of pip? Or is this only an issue for certain packages being installed with pip, and if so, are there certain steps to take to ensure users do not encounter the error when installing your modules?

Answer 1

I don't think this actually belongs on StackOverflow, but in case I'm wrong…

First, python-dev is not a Python thing, it's an Ubuntu or Fedora or some other distro thing. If you download, build, and install Python, or run any binary installer from python.org, you get Python.h installed in an appropriate place. Many linux distros like to split packages into subpackages, moving stuff you only need for building into a -dev or similar package, and there's absolutely nothing wrong with doing that to Python, but it's still something the distro is doing.

Second, Python.h is not needed for building all packages, only those that include C extension modules. Since many packages are pure-Python and have no such extension modules, it makes sense that a distro's pip package wouldn't require its python-dev package. (In the same way that a distro's pip package probably wouldn't require a C compiler.)

Third, most distros that give you a python-pip or similar package also give you packages for popular packages. If you install them that way, either you won't need python-dev (and a C compiler), because they're binary packages, or you will need them but they'll be pulled in as a dependency (rpm, deb, etc. all have a way to specify separate "build" and "run" dependencies).

But if you go behind your package manager's back and try to install packages with pip (which is a reasonable thing to do), the package manager can't tell you which packages need what dependencies, while pip can only tell you about Python package dependencies, so there's nothing to enforce this.

Answer 2

I checked how popular PyPi packages handle missing Python.h. My first problem was to find popular PyPi packages that build a C extension, which I solved by checking zmq/pyzmq, grpcio, and python-qpid-proton, which I know are three packages doing this.

The zmq package prints a generic catch-all error message. The grpcio package prints a specific error message which tries to be maximally helpful. The python-qpid-proton package prints the compiler error only.

ZMQ

    build/temp.linux-x86_64-3.7/scratch/vers.c:4:10: fatal error: zmq.h: No such file or directory                                                                                                                                           
        4 | #include "zmq.h"                                                                                                                                                                                                                 
          |          ^~~~~~~                                                                                                                                                                                                                 
    compilation terminated.                                                                                                                                                                                                                  

    error: command 'gcc' failed with exit status 1                                                                                                                                                                                           

    ************************************************                                                                                                                                                                                         
    Warning: Couldn't find an acceptable libzmq on the system.                                                                                                                                                                               

    If you expected pyzmq to link against an installed libzmq, please check to make sure:                                                                                                                                                    

        * You have a C compiler installed                                                                                                                                                                                                    
        * A development version of Python is installed (including headers)
        * A development version of ZMQ >= 3.2 is installed (including headers)
        * If ZMQ is not in a default location, supply the argument --zmq=<path>
        * If you did recently install ZMQ to a default location,
          try rebuilding the ld cache with `sudo ldconfig`
          or specify zmq's location with `--zmq=/usr/local`

https://gist.github.com/jiridanek/0fcd09f27d379ae984b84f174986093d

gRPC.io

    commands.CommandError: Diagnostics found a compilation environment issue:

    Could not find <Python.h>. This could mean the following:
      * You're on Ubuntu and haven't run `apt-get install python-dev`.
      * You're on RHEL/Fedora and haven't run `yum install python-devel` or
        `dnf install python-devel` (make sure you also have redhat-rpm-config
        installed)
      * You're on Mac OS X and the usual Python framework was somehow corrupted
        (check your environment variables or try re-installing?)
      * You're on Windows and your Python installation was somehow corrupted
        (check your environment variables or try re-installing?)

https://gist.github.com/jiridanek/9bff05ae79b0eb5207821fc524acf12a

Python Qpid Proton

cproton_wrap.c:150:21: fatal error: Python.h: No such file or directory
      # include <Python.h>
                          ^
compilation terminated.     
error: command 'gcc' failed with exit status 1