Generating multiple observers with Python watchdog

Question

I currently have a basic functioning script that monitors a single directory and all sub-directories below for changes and passes the output to the LoggingEventHandler.

I now want to expand my script to monitor 3 separate locations but I simply can't grasp how to generate multiple Observers to watch each of my designated paths.

I attempted something along the lines of the following:

import time
import thread
import threading
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

event_handler = LoggingEventHandler()
observer = Observer()

paths = ["C:\dir1", "C:\dir2", "C:\dir3"]

for i in paths:
    targetPath = str(i)
    observer.schedule(event_handler, targetPath, recursive=True)
    observer.start_new_thread()

Unfortunately I received an error indicating that observer doesn't have an attribute 'start_new_thread'

There is no example within the documentation that shows more than a single observer monitoring a directory. I have no experience of dealing with threads and I'm not even sure if I am on the right track.

Should I be creating a new instance of the observer class for each path instead? Or is there some method of feeding a single instance of the Observer class, multiple paths?

Apologies if there is a glaringly obvious answer. I am sure this is all completely wrong now, I am just too tired to understand.

ADDITIONAL:

Thanks to @FogleBird I have corrected the thread start issue but I am still stuck with only a single instance rather than three separate observers watching the different paths. My altered code now looks like:

threads = []

for i in paths:
    targetPath = str(i)
    observer.schedule(event_handler, targetPath, recursive=True)
    threads.append(observer)

observer.start()
print threads

This returns three ObservedWatch objects but they all have the same details:

[<Observer(Thread-1, started daemon 1548)>, <Observer(Thread-1, started daemon 1548)>, <Observer(Thread-1, started daemon 1548)>]

Still looks completely wrong, any more help would be great. I am struggling to grasp this concept.

ADDITIONAL 2:

I have continued mucking about with the code and I now have something that appears to be functional:

event_handler = LoggingEventHandler()
N2watch = Observer()
threads = []

for i in paths:
    targetPath = str(i)
    N2watch.schedule(event_handler, targetPath, recursive=True)
    threads.append(N2watch)

N2watch.start()

try:
    while True:
            time.sleep(1)
except KeyboardInterrupt:
    N2watch.stop()
N2watch.join()

From what I can gather from an initial run the output appeared to pickup changes in all three of the path names specified in my list but I will need to write some test code to check.

I am still not certain how this is behaving so any further comments would be great.

Cheers.

ADDITIONAL 3:

I marked FogleBird's answer as the best because it was simply the only one and did highlight issues with my initial code.

My previous edit included fully working code to monitor multiple locations and appears to be functioning correctly at present.

Answer 1

The example code here shows a function called start, not start_new_thread. Have you tried that?

https://pypi.python.org/pypi/watchdog

Also, you should probably call start just once, after the for loop, not inside of it.

Answer 2

Great question. This thread is older but I found it while looking up the exact thing and I expanded on your work and added the ability to pass in a file with a list of directories to watch. By default I do not look recursively, I leave that to someone else to test. Hopefully this helps anyone looking up the same topic. Great work!

Run using python watcher.py filename.

Where watcher.py is what I called my script and filename is the name of the file with my paths.

I list the full paths in the file and these are separated by newlines.

i.e.:

C:\path1
C:\Path2\subpath1
C:\PATH3

watcher.py

import logging
import sys
import time
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

# Attach a logging event AKA FileSystemEventHandler
event_handler = LoggingEventHandler()

# Create Observer to watch directories
observer = Observer()

# Take in list of paths. If none given, watch CWD
paths = open(sys.argv[1], 'r') if len(sys.argv) > 1 else '.'

# Empty list of observers
observers = []

# Base logging configuration
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

# Iterate through paths and attach observers
for line in paths:

    # Convert line into string and strip newline character
    targetPath = str(line).rstrip()

    # Schedules watching of a given path
    observer.schedule(event_handler, targetPath)

    # Add observable to list of observers
    observers.append(observer)

# Start observer
observer.start()

try:
    while True:

        # Poll every second
        time.sleep(1)

except KeyboardInterrupt:
    for o in observers:
        o.unschedule_all()

        # Stop observer if interrupted
        o.stop()

for o in observers:

    # Wait until the thread terminates before exit
    o.join()

Answer 3

Just wanna add some notes:

The threading lib and threads list in the code can be a little bit confusing for people who just start using watchdog (including myself). They are actually not necessary in the solution. A simple way to explain it is just:

• create one observer
• schedule multiple "watching events"
• and start the observer.

That's it.

Answer 4

Here's the code I use to watch multiple directories.

import sys
import time
import logging
from watchdog.observers.polling import PollingObserver as Observer
from watchdog.events import LoggingEventHandler

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')
    event_handler = LoggingEventHandler()
    observer = Observer()
    if len(sys.argv) > 1:
        for i in range(1, len(sys.argv)):
            observer.schedule(event_handler, sys.argv[i], recursive=True)
    else:
        observer.schedule(event_handler, '.', recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()