Save Streaming Twitter Data to CSV

Question

I'm new to Python, and I'm attempting to save data from the streaming Twitter API to a CSV file. I can successfully print content to my console, but I cannot get it to save.

I've done a search on stack and I've found several examples which come very close to answering my question, but none which I've found very adaptable due to my very limited skillset.

My code to print to console is as follows:

import sys
import tweepy

#pass security information to variables
consumer_key=""
consumer_secret=""
access_key = ""
access_secret = ""


#use variables to access twitter
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)

#create an object called 'customStreamListener'

class CustomStreamListener(tweepy.StreamListener):

    def on_status(self, status):
        print status.author.screen_name, status.created_at, status.text


    def on_error(self, status_code):
        print >> sys.stderr, 'Encountered error with status code:', status_code
        return True # Don't kill the stream

    def on_timeout(self):
        print >> sys.stderr, 'Timeout...'
        return True # Don't kill the stream


streamingAPI = tweepy.streaming.Stream(auth, CustomStreamListener())
streamingAPI.filter(track=['russia'])

Answer 1

This works for me, but let me know if you have trouble in a comment.

import csv
def on_status(self, status): 
    with open('file.txt', 'w') as f: 
        f.write('Author,Date,Text')
        writer = csv.writer(f)
        writer.writerow([status.author.screen_name, status.created_at, status.text])

Answer 2

You'll need to import csv to get it to save to a file if you want to do this (as your question says write to csv.

First you'll have to:

import csv

Then you'll have to open a file to write to and create a writer:

handle=csv.writer(open('file.csv','wb'))

I'd change

def on_status(self, status):
    print status.author.screen_name, status.created_at, status.text

to something like this:

def on_status(self, status):
    print status.author.screen_name, status.created_at, status.text
    handle.writerow(status.author.screen_name, status.created_at, status.text)

Of course you'd have to send the handle to the class or have some method to get the handle from the main to that method.

I'd also check this stack question as well.

You should be careful with the data that is coming in from Twitter, it can have commas in it as well. Looking at this question they explain how to escape any comma that may exist in a string. Or course you'd then have to escape every variable or at least status.text.

Answer 3

When you say "save," is there a specific format you had in mind?

The first solution I can think of would be to just write whatever you print to a text file:

my_file = open("file.txt","w") 
def on_status(self, status):
    a = status.author.screen_name 
    b = status.created_at 
    c = status.text
    my_file.write(a,b,c) 
my_file.close()

I'm somewhat new to Python as well, so this may not be 100% correct, but it's worth a shot.