It is also possible to implement tail from the python side, which is basically a continuous reading of it. The code snippet to make this work can be found here:
http://code.activestate.com/recipes/157035-tail-f-in-python/
Additionally, if you use the append mode of file writing instead of the write method you can continuously output.
Scrapy also uses the concept of pipelines which allow for a lot of the same functionality. Here's an example of some scrapy code you might use to do the same thing:
class JsonWriterPipeline(object):
def __init__(self):
self.file = (open(filepath, 'a'))
def process_item(self, item, spider):
self.file.write(json.dumps(dict(item)) + '\n')
return item