Question

everyone! I'm trying to scrape data from a website and append any entries that have not already been copied to a .csv file, but I can't seem to get it to write properly.

url = 'www.website.com'

def getInfo():
    global oldMaxValue, oldMaxRow, newInfo
    newInfo = 0
    with open("file.csv", "a") as f:
        data = requests.get(url)
        text = data.text

        newRows = [line.split(',') for line in text.split("\n") if line]
        newMaxValue = max(row[0] for row in newRows)

        for i in newRows:
            if int(i[0]) > int(oldMaxValue):
                f.write(str(i))

        oldMaxValue = newMaxValue

Any guidance would be greatly appreciated. Thanks very much! :)

Was it helpful?

Solution

Use the csv module. This is an example of appending a single row to an existing file:

import csv

f = file("file.csv", 'a')
csv_writer = csv.writer(f)

my_record = ["This", "is", "a", "row", "of", "data"]

csv_writer.writerow(my_record)
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top