Threading with Hadoop Streaming

Question

I am making use of Hadoop streaming to write a python based HTML grabber. I find that running a single threaded python script is slow. I want to modify it to a multithreaded version. Does anyone know what would be a good number to set the number of threads in the mapper to. I am not sure of the specs of each node of the cluster but I assume that it would support atleast two threads.

Answer 1

I tried to use threading with python, there were issues with the Global Interpreter Lock. Ported code to use the multiprocessing module, internally hadoop assigns as many mappers as there are cores in the cluster, hence multiprocessing is not the way to go if you need speed up. Multithreading if performed right might give some speedup

Answer 2

I have not use hadoop streaming for html grabber but here is a post that talking about how urllib2 work s using multiple thread (not multipleprocessing package, just simple multi thread).

Hope can be helpful.