Script to unpack python 2.7 at bootstrap on amazon EMR node

Question

I've got python scripts that require version 2.7. Installing python 2.7 at bootstrap time on EMR using a bash script is easy enough but is taking too long.

AWS support suggested I compile Python 2.7 locally, tar the installation and unpack it at bootstrap time (bootstrapping can only run for a limited amount of time).

Sounds straightforward enough. The first challenge is how to tar a functional python 2.7 installation and the second one is how to unpack it properly on EMR slave nodes (not via the apt-get install or tgz & configure/make/make install route).

Before reinventing the wheel (the debugging cycle on EMR is long): are there bash script examples available for these two steps? Thanks.

Answer 1

Sounds like you want to have your application use its own Python rather than the one provided by the operating system. These steps have worked for us:

1) tar the Python directory you want to use. /usr/lib64/python2.7 or equivalent. 2) untar the directory into whatever directory works for you. 3) Prepend that directory onto PATH, or use absolute path when invoking the interpreter 4) Set PYTHONHOME to point to that directory.

Assuming /myapp/python is where you want to place Python, your bash script looks like:

tar xvf mypython.tar /myapp/python/
export PATH=/myapp/python/:$PATH
export PYTHONHOME=/myapp/python