Python3: writing info from a text file to a csv file

Question

I have multiple directories each with multiple files in:

Ce  
+---top.txt
+---X0.0        
|     |  
|     +---Y0.0  
|     |     |
|     |     +---X0.0Y0.0Z0.0.dat
|     |     +---X0.0Y0.0Z0.0.out
|     |     +---X0.0Y0.0Z0.05.dat
|     |     +---X0.0Y0.0Z0.05.out   
|     +---Y0.05
|           |
|           +---X0.0Y0.05Z0.0.dat
|     |     +---X0.0Y0.0Z0.05.out
|           +---X0.0Y0.05Z0.05.dat
|           +---X0.0Y0.05Z0.05.out
+---X0.05
      |  
      +---Y0.0  
      |     |
      |     +---X0.0Y0.0Z0.0.dat
      |     +---X0.0Y0.0Z0.0.out
      |     +---X0.0Y0.0Z0.05.dat
      |     +---X0.0Y0.0Z0.05.out   
      +---Y0.05
            |
            +---X0.0Y0.05Z0.0.dat
            +---X0.0Y0.05Z0.0.out
            +---X0.0Y0.05Z0.05.dat
            +---X0.0Y0.05Z0.05.out

I am attempting to extract the relevant information from the '.out' files and write it to a csv file. For this I have devised the following code:

import os
import csv

with open('results.csv', 'a', newline='') as f:
    writer=csv.writer(f)
    writer.writerow(['Filename', 'Optimisation Achieved?', 'Final Energy', 'Number of Defects', 'Defect Charge', 'M-L Centre X', 'M-L Centre Y', 'M-L Centre Z', 'Defect X', 'Defect Y', 'Defect Z', 'Defect Energy']) 
    for root, dirs, files in os.walk('.'):
        for filename in files:
            if filename.endswith('.out'):
                file = os.path.join(root, filename)
                with open(file, 'r') as reader:
                    opt_cnt=0
                    for line in reader:
                        s=line.strip()
                        if s=='**** Optimisation achieved ****':
                            opt_cnt+=1
                            if opt_cnt==1:
                                opt1='Y' + str(opt_cnt)
                            if opt_cnt==2:
                                opt2='Y' + str(opt_cnt)
                            else:
                                opt2='N'
                        elif s.startswith('Final energy='):
                            Final=s[18:31]
                        elif s.startswith('Total Number of Defects'):
                            Number=s[31]
                        elif s.startswith('Total charge on defect'):
                            Q=s[-4:]
                        elif s.startswith('Defect centre'):
                            centrex=s[21:28]
                            centrey=s[31:37]
                            centrez=s[40:46]
                        elif s.startswith('Atom/Fractional'):
                            coordx=s[40:49]
                            coordy=s[51:60]
                            coordz=s[62:71]
                        elif s.startswith('Final defect energy'):
                            Defect_Energy=s[-13:]
                writer.writerow([filename, opt1, Final, Number, centrex, centrey, centrez, coordx, coordy, coordz, Defect_Energy])

However, this produces the following error:

Traceback (most recent call last):
File "C:\Users\Rebecca\Dropbox\Brannerite\Published Potentials\interstitials\Grid\Ce\results.py", line 39, in
writer.writerow([filename, opt1, Final, Number, centrex, centrey, centrez, coordx, coordy, coordz, Defect_Energy])
NameError: name 'Final' is not defined

[Altering the amount of indentation of the the last line in my code alters which of the variables comes up in the nameError: as the code is there are 16 indents (the same is NameError is given with 20 indents); altering this to 8, 12 or 24 gives 'NameError: name 'opt1' is not defined]

I should say that much of this code [remove the lines relating to os.walk] has been used successfully for a different set of files (that were all in one directory).

Can anyone suggest where the error is coming from in the above code and how it can be corrected?

Answer 1

If there is an out file that doesn't have line that starts with Final energy=, Final stays undefined. You may want to define default values for Final, Number etc variables that are defined in the loop, for example set them all to undefined string:

Final = Number = ... = 'undefined'
file = os.path.join(root, filename)
with open(file, 'r') as reader:
    opt_cnt=0
    for line in reader:
        ...

Answer 2

I think you want to have 2 vars Final and opt1 defined. Which is not the case in your code

writer.writerow([filename, opt1, Final, Number, centrex, centrey, centrez, coordx, coordy,    coordz, Defect_Energy])

as your code allows to define only 1 of them or opt1 or Final due to if/elif construct:

if s=='**** Optimisation achieved ****':
    opt_cnt+=1
    if opt_cnt==1:
        opt1='Y' + str(opt_cnt)
elif s.startswith('Final energy='):
    Final=s[18:31]

Maybe try defining default values upfront or change the logic applied