Adding timeline while Extracting Twitter data in R

Question

I am trying to extract twitter data for a keyword using the following code:

cred<- OAuthFactory$new(consumerKey='XXXX', consumerSecret='XXXX',
                        requestURL='https://api.twitter.com/oauth/request_token',
                        accessURL='https://api.twitter.com/oauth/access_token',
                        authURL='https://api.twitter.com/oauth/authorize')

cred$handshake(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))

To enable the connection, please direct your web browser to: https://api.twitter.com/oauth/authorize?oauth_token=Cwr7GgWIdjh9pZCmaJcLq6CG1zIqk4JsID8Q7v1s When complete, record the PIN given to you and provide it here: 8387466

registerTwitterOAuth(cred)
search=searchTwitter('facebook',cainfo="cacert.pem",n=1000)

But evenwith n=1000, the function returns a list of only 99 tweets where it should more than that. I also tried the same function with a specific timeline:

search=searchTwitter('facebook',cainfo="cacert.pem",n=1000,since='2013-01-01',until='2014-04-01')

But this function returns a empty list.

Can anyone help me out, with the correct set of additional queries so that I can extract data from a specific timeline and without any restriction on the number of tweets? Does it have to do anything with the amount of data fetched by the API?

Thanks in advance

Answer 1

It looks like Twitter API restricts number of returned tweets. You should check this out in the API documentation. Keeping the restriction in mind, you can use the since and sinceID arguments of searchTwitter() within a loop, something like:

for (i in 1:20) {
  if (i==1) search = searchTwitter('facebook',cainfo="cacert.pem",n=2, since='2014-04-15')
  else search = searchTwitter('facebook',cainfo="cacert.pem",n=2, since='2014-04-15', sinceID=search[[1]]$id)
  print(search)
  Sys.sleep(10)
}

You may need to adjust the Sys.sleep(10) portion if you hit API restrictions.