Issues converting a python script to a linux binary that uses lxml's etree module

Question

I'm essentially having the same issue as this person in Sept of 2013.

Related issues: Cx_freeze with lxml.html TypeError (someone found a fix by editing the cx-freeze code, but did not explain it in depth), cx-freeze doesn't find all dependencies (not sure how to open eggs in Linux)

I'm not allowed to comment on it until I have 50 reputation so I posted this thread. I'm using 4.3.1 of cx_freeze to freeze the following modules:

import sys
import getopt
from time import gmtime, strftime, time
from os.path import exists
from lxml import etree

My setup.py script:

import sys
from cx_Freeze import setup, Executable

#build_exe_options = {"packages": ["sys","getopt","time","os","lxml"]}

build_exe_options = {"packages": ["sys","getopt","time","os","lxml","BeautifulSoup","lxml.html.soupparser","lxml.html.html5parser","lxml.html.diff","lxml.ElementInclude"]}

setup(
    name = "cmpxml",
    version = "r13",
    description = "",
    options = {"build_exe": build_exe_options},
    executables = [ Executable("cmpxml.py", base = None) ] )

I run

python setup.py build_exe

or

python setup.py build

and receive the following messages:

? cjkcodecs.aliases imported from BeautifulSoup
? html.entities imported from lxml.html.soupparser
? html5lib imported from lxml.html.html5parser
? iconv_codec imported from BeautifulSoup
? lxml.html.escape imported from lxml.html.diff
? urllib.parse imported from lxml.ElementInclude
? urllib.request imported from lxml.ElementInclude

When running my compiled program, it fails when trying to import lxml so I felt that the above messages are vital in debugging the compilation.

Am I doing something wrong or is this an issue with cx-freeze? My LXML module was installed using pip (so there's no need for PYTHONPATH) and I'm using python 2.7 on Ubuntu 13.10.

---

Nuitka 0.4.7:

Other issue but related: Coincidentally, I'm also having a similar issue with Nuitka.

/usr/share/nuitka/bin/nuitka cmpxml.py --exe --recurse-all --execute

Will compile correctly but when running ./cmpxml it will fail to find the lxml module.

---

PyInstaller 2.1:

Also tried using pyinstaller with a hooks directory containing hook-lxml.etree.py, hook-lxml.objectify.py, hook-xml.etree.cElementTree.py and running

pyinstaller --additional-hooks-dir=hooks/ cmpxml.py

It still cannot detect lxml when running the binary in dist/cmpxml/cmpxml. The warning file in the build directory contained these missing modules and downloading all the hooks from the pyinstaller git did not solve the issue.

$ cat build/cmpxml_noetree/warncmpxml_noetree.txt
W: no module named msvcrt (conditional import by subprocess)
W: no module named msvcrt (delayed import by getpass)
W: no module named _dummy_threading (top-level import by dummy_threading)
W: no module named cl (delayed, conditional import by aifc)
W: no module named rourl2path (conditional import by urllib)
W: no module named nt (conditional import by __main__)
W: no module named nt (conditional import by pyi_os_path)
W: no module named _subprocess (conditional import by subprocess)
W: no module named cl (delayed import by aifc)
W: no module named _scproxy (conditional import by urllib)
W: no module named org (top-level import by pickle)
W: no module named EasyDialogs (conditional import by getpass)
W: no module named SOCKS (top-level import by ftplib)
W: no module named _winreg (delayed import by urllib)
W: no module named nt (top-level import by ntpath)
W: no module named msvcrt (conditional import by getpass)
W: no module named _winreg (top-level import by mimetypes)
W: no module named org (top-level import by copy)
W: no module named _emx_link (conditional import by os)

The weird thing is is that lxml isn't even mentioned in this warning file. For some reason, it's just not included.

---

Any help would be appreciated. Nuitka, pyinstaller, freeze, cx-freeze, all seem to be great programs but don't seem to work for me.

Answer 1

A friend of mine showed me that the issue was due to the lack of importation of the inspect module.

The correct command for cx-freeze to compile the above:

cxfreeze cmpxml.py --target-dir cmpxml --include-modules=sys,getopt,time,os,inspect,lxml.etree,lxml._elementpath,gzip,encodings.utf_8,encodings.ascii

Answer 2

In addition for those who googled this question. OP used Nuitka 4.3.1 but in Nuitka 0.5.1 the problem is seems fixed http://nuitka.net/posts/nuitka-release-051.html