The key to understanding what is going on is the comment you made
No. When the threads are all completed then I do...but not during thread execution for all the files they have to upload.
and this line of code:
q.join()
Contrary to what you are probably expecting, a control-C does NOT cause it to stop waiting for the queue - it doesn't accept the control-C until after this call has returned. So what is happening is that all of your threads have done their jobs and emptied the queue, and then are waiting on the line
item = q.get()
Only after the last thread calls q.task_done
does the main thread return and then process the control-C. However, at that point all the threads are stuck waiting for more items from the queue (which they aren't going to get), so they will never exit the loop.
There might be more going on here than this, but to see if this is the problem try a busy wait for the queue to be empty:
while not q.empty():
time.sleep(0.1)
q.join()
You need the join afterward because the queue being empty means the last upload has been pulled from the queue, not that it has been finished.
One other thing you can add is an item to the queue that signals the thread should finish, such as None
. For example,
def worker():
global kill_received
while True and not kill_received[0]:
print kill_received[0]
item = q.get()
if item is None:
q.task_done()
break
upload_file(item, file_quantity, retry_list, authenticate)
q.task_done()
for i in range(value):
t = Thread(target=worker)
t.setDaemon(True)
t.start()
for item in file_list:
q.put(item)
for i in range(value):
q.put(None)
Of course, this assumes that None
is not a valid value to upload. This won't help with the control-C issue, but it is something you might find helpful to make sure that the threads exit when the program finishes normally.
As a general help, when you testing things with threads it can be helpful to have a way to print out stack traces for all threads. This SO question talks about how to do that.