Python multi-threading missing jobs

Question

If I run the script step by step works perfectly, but when I'm using threading misses 50-60%. I'm using Python + mechanize module

#setting up the browser
mySite = 'http://example.com/managament.php?'
postData = {'UserID' : '', 'Action':'Delete'}
job_tab1_user1 = [1,2,3]
job_tab2_user1 = [4,5,6]
job_tab1_user2 = [7,8,9]
job_tab2_user2 = [10,12,13]
.... till user1000
#i want to point out that the lists are 100% different
def user1_jobs:
    for i in job_tab1_user1:
        browser.open("http://example.com/jobs.php?actions="+i) 
        browser.open(mySite, Post_data)
    for i in job_tab2_user1:
        browser.open("http://example.com/jobs.php?actions="+i) 
        browser.open(mySite, Post_data)
def user2_jobs:
    for i in job_tab1_user2:
        browser.open("http://example.com/jobs.php?actions="+i) 
        browser.open(mySite, Post_data)
    for i in job_tab2_user2:
        browser.open("http://example.com/jobs.php?actions="+i) 
        browser.open(mySite, Post_data)
... and so on till user 1000

And I call them in the end like this:

t_user1 = threading.Thread(target=user1_jobs, args=[])
t_user1.start()
t_user2 = threading.Thread(target=user2_jobs, args=[])
t_user2.start()

I have a similar script that sends like 200 request per second and all of them are processed. I also tried using time.sleep(2), but again is missing a lot. Another question besides what is wrong with my script is if its way to compact this code, because I'm using 1000 users and the script reaches thousands of lines. Thank you in advance.

Answer 1

from threading import *

submits = [[1,2,3], [3,4,5], [6,7,8]]    

class worker(Thread):
    def __init__(self, site, postdata, data):
        Thread.__init__(self)
        self.data = data
        self.site = site
        self.postdata = postdata
        self.start()
    def run(self):
        for i in self.data:
            browser.open("http://example.com/jobs.php?actions="+str(i))
            browser.open(self.site, self.postdata)
for obj in submits:
    worker('http://example.com/managament.php?', {'UserID' : '', 'Action':'Delete'}, submits)

Since the OP asked for it, here's a condensed/compressed version of the code.

or:

for index in range(0,1000):
    worker('http://example.com/managament.php?', {'UserID' : '', 'Action':'Delete'}, [i for i in range(1,4)])

If the data you want to send actually is a sequence of 3 integers (1,2,3) that inclines in a perfect order.

Answer 2

Here is a full script that you can easily modify by changing the initial variables. It creates a list dynamically and uses a generator to create the functions for each thread. Currently it creates 1000 users, each with 2 tabs and 3 jobs.

# define your variables here
NUM_USERS = 1000
NUM_JOBS_PER_USER = 3
NUM_TABS_PER_USER = 2
URL_PART = "http://example.com/jobs.php?actions="

# populate our list of jobs
# the structure is like this: jobs[user][tab][job]

jobs = [[[0 for y in range(NUM_JOBS_PER_USER)] \
            for x in range(NUM_TABS_PER_USER)] \
            for x in range(NUM_USERS)]
p = 1
for i in range(NUM_USERS):
    for j in range(NUM_TABS_PER_USER):
        for k in range(NUM_JOBS_PER_USER):
            jobs[i][j][k] = p
            p += 1


# create a generator that builds our thread functions
def generateFunctions(jobs):
    for user in jobs:
        for tab in user:
            for job in tab:
                def f():
                    browser.open(URL_PART + str(job))
                    browser.open(mySite, Post_data)
                yield f

# create and start threads, add them to a list
# if we need to preserve handlers for later use
threads = []
for f in generateFunctions(jobs):
    thr = threading.Thread(target = f, args=[])
    thr.start()
    threads.append(thr)