Access data of a wiki scraper and store it in a local database

Question

Hi i have written a scraper on scraper-wiki which scrape a web page and store the data in scraper-wiki database.Now i want to write a program in python which will go to scraper-wiki API and fetch the data and stores in a sq lite database of local machine.

Answer 1

Hey guys luckily i got the answer On scraper wiki there is an API here https://scraperwiki.com/docs/api#sqlite which will generate the URI of the table of a particular wiki scraper u want access of then u can open it as a url with the help of urllib module of python

import urllib

def access_data():

f= urllib.urlopen('https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=csv&name=example&query=select%20*%20from%20%60table%60')

data=f.read()
s = open('example.csv','w')
s.write(data)

Answer 2

First, you need to query the data you want. Here is the documentation: 1 2

Then, you need to store it using the sqlite library which its documentation can be found at python's official documentation site.

Answer 3

There's also a download link on your scraper's overview page where you can download your data as a sqlite file, no code required. Then you can work with the data locally.