Why is DictWriter not Writing all rows in my Dictreader instance?

Question

I'm new to coding and by default new to Python, so please excuse my ignorance...I'm working on it.

I am trying to write some code (Python 2.7) to take specific headers from multiple CSV files and export them as a single file. Here is my code:

import csv, os

path = 'C:/Test/'

for fn in os.listdir(path):
    if ".csv" in fn:
        with open(fn, 'rb') as f:
            with open('C:/Test/fun/output.csv', 'wb') as fou:
                reader = csv.DictReader(f, delimiter=",", quotechar="|")
                writer = csv.DictWriter(fou, delimiter=",", quotechar="|", fieldnames=    ['sku', 'stock.qty', 'stock.is_in_stock'], extrasaction='ignore')
                headers = {} 
                for n in writer.fieldnames:
                    headers[n] = n
                writer.writerow(headers)
                for row in reader:
                    print row
                    writer.writerow(row)



    elif ".csv" not in fn:
        break

The print request for the reader instance seems to print all rows from multiple files. I am testing on 3 files with known rows. However, the DictWriter output file only has the rows from the last of the files read. It just doesn't make sense to me how I can print row and writerow and get different results. Obviously my DictWriter is incorrectly written but I do not see where. Probably obvious to most but I am puzzled.

Answer 1

You are opening your target CSV file and clearing it for each matching CSV file you read. Opening the file in 'wb' mode clears the file each time.

Moreover, you break out of the loop as soon as you find a filename that is not a CSV file; you probably didn't want to do that at all; remove the else branch there.

Open the file just once, and continue to use it while looping over the directory, instead:

with open('C:/Test/fun/output.csv', 'wb') as fou:
    writer = csv.DictWriter(fou, delimiter=",", quotechar="|", fieldnames=    ['sku', 'stock.qty', 'stock.is_in_stock'], extrasaction='ignore')
    writer.writeheader()

    for fn in os.listdir(path):
        if ".csv" in fn:
            with open(fn, 'rb') as f:
                reader = csv.DictReader(f, delimiter=",", quotechar="|")
                for row in reader:
                    print row
                    writer.writerow(row)

I used the DictWriter.writeheader() method to write your fieldnames to the output file as an initial header.