How to bundle Python dependancies in IronWorker?

Question

I'm writing a simple IronWorker in Python to do some work with the AWS API.

To do so I want to use the boto library which is distributed via PiPy. The boto library is not installed by default in the IronWorker runtime environment.

How can I bundle the boto library dependancy with my IronWorker code?

Ideally I'm hoping I can use something like the gem dependancy bundling available for Ruby IronWorkers - i.e in myRuby.worker specify

gemfile '../Gemfile', 'common', 'worker' # merges gems from common and worker groups

In the Python Loggly sample, I see that the hoover library is used:

#here we have to include hoover library with worker.
hoover_dir = os.path.dirname(hoover.__file__)
shutil.copytree(hoover_dir, worker_dir + '/loggly') #copy it to worker directory

However, I can't see where/how you specify which hoover library version you want, or where to download it from.

What is the official/correct way to use 3rd party libraries in Python IronWorkers?

Answer 1

[edit]We've worked on our toolset a bit since this answer was written and accepted. The answer from my colleague below is the recommended course moving forward.[/edit]

I wrote the Python client library for IronWorker. I'm also employed by Iron.io.

If you're using the Python client library, the easiest (and recommended) way to do this is to just copy over the library's installed folder, and include it when uploading the package. That's what the Python Loggly sample is doing above. As you said, that doesn't specify a version or where to download the library from, because it doesn't care. It just takes the one installed on your system and uses it. Whatever you get when you enter "import boto" on your local machine is what would be uploaded.

The other option is using our CLI to upload your worker, with a .worker file.

To do this, here's what you'd need to do:

Create a botoworker.worker file:

runtime "binary"
build 'pip install --install-option="--prefix=`pwd`/pips" boto'
file 'botoworker.py'
exec "botoworker.sh"

That second line is the pip command that will be run to install the dependency. You can modify it like you would any pip command run from the command line. It's going to execute that command on the worker during the "build" phase, so it's only executed once instead of every time you run a task.

The third line should be changed to the Python file you want to run--it's your Python worker file. Here's the one we used to test this:

import boto

If you save that as botoworker.py, the above should work without any modification. :)

The fourth line is a shell script that's going to actually run your worker. I've included the one we used below. Just save it as botoworker.sh, and you won't have to worry about modifying the .worker file above.

PYTHONPATH="$HOME/pips/lib/python2.7/site-packages:$PYTHONPATH" python botoworker.py "$@"

You'll notice it refers to your Python file--if you don't name your Python file botoworker.py, remember to change it here, too. All this does is set your PYTHONPATH to include the installed library, and then runs your Python file.

To upload this, just make sure you have the CLI installed (gem install iron_worker_ng, making sure your Ruby version is 1.9.3 or higher) and then run "iron_worker upload botoworker" in your shell, from the same directory your botoworker.worker file is in.

Hope this helps!

Answer 2

Newer iron_worker version has native support of pip command. So, you need:

runtime "python"
exec "something.py"

pip "boto"
pip "someotherpip"

full_remote_build true