Best way to go about including C/C++ dependencies in Python packages?

Question

I would like to include fastparquet as a dependency in a Python library I am working on but it requires Microsoft Visual C++ to build. The goal here is for the end user to be able to easily install my library via PyPi or with the pip install git+(url) one-liner. I found pip wheels for pre-compiled fastparquet binaries here, but obviously that would only be a solution for users with the same CPU architecture.

I'm basically asking if there's a good way to include C++ dependent extensions without the end-user needing to have build tools in order to run my library?

Answer 1

Not possible. By including C++ dependencies in your Python module, you are trading platform independence for performance. You have to decide whether that tradeoff is worth it.

If so, you really only have the choice between two Python packaging formats:

• source packages that the end user must compile themselves, but are potentially portable
• precompiled wheels that are specific to a platform (Python version, OS, snd CPU architecture)

There is no magical in-between format that combines the advantages of both (unless you count currently highly unusual techniques such as compiling native code to WebAssembly, but Python does not provide a suitable platform for this).

What you can do is using hosted CI services to easily generate a large matrix of packages in order to account for different OSes and Python versions.

Wheels only work when installed directly, e.g. via a package index (doesn't have to be the public PyPI though) or when directly installing a .whl file. Installing a package from git will always need a compile because it's effectively a source package, though you could move the native parts into separate packages.

By the way, if your software is Windows-specific (why else would Visual Studio be a hard dependency) then platform independence doesn't matter, because Windows effectively only runs on x86-64 processors.