Python, os.walk(), pass information back up?

Question

I'm currently attempting to write a simple python program that loops through a bunch of subdirectories finding java files and printing some information regarding the number of times certain keywords are used. I've managed to get this working for the most part. The problem I'm having is printing overall information regarding the higher directories, for example, my current output is as follows:

testcases/part1/testcase2/root_dir:
    0   bytes     0   public     0   private     0   try     0   catch
testcases/part1/testcase2/root_dir/folder1:
    12586   bytes     19   public     7   private     8   try     22   catch
testcases/part1/testcase2/root_dir/folder1/folder5:
    7609   bytes     9   public     2   private     7   try     11   catch
testcases/part1/testcase2/root_dir/folder4:
    0   bytes     0   public     0   private     0   try     0   catch
testcases/part1/testcase2/root_dir/folder4/folder2:
    7211   bytes     9   public     2   private     4   try     9   catch
testcases/part1/testcase2/root_dir/folder4/folder3:
    0   bytes     0   public     0   private     0   try     0   catch

and I want the output to be:

testcases/part1/testcase2/root_dir :
    27406  bytes    37  public    11  private    19  try    42  catch
testcases/part1/testcase2/root_dir/folder1 :
    20195  bytes    28  public     9  private    15  try     33  catch
testcases/part1/testcase2/root_dir/folder1/folder5 :
    7609  bytes     9  public     2  private     7  try      11  catch
testcases/part1/testcase2/root_dir/folder4 :
    7211  bytes     9  public     2  private     4  try     9  catch
testcases/part1/testcase2/root_dir/folder4/folder2 :
    7211  bytes     9  public     2  private     4  try     9  catch
testcases/part1/testcase2/root_dir/folder4/folder3 :
    0  bytes        0  public     0  private     0  try     0  catch

As you can see the lower subdirectories directly provide the information to the higher subdirectories. This is the problem I'm running into. How to efficiently implement this. I have considered storing each print as a string in a list and then printing everything at the very end, but I don't think that would work for multiple subdirectories such as the example provided. This is my code so far:

def lsJava(path):

print()

for dirname, dirnames, filenames in os.walk(path):

    size = 0
    public = 0
    private = 0
    tryCount = 0
    catch = 0

    #Get stats by current directory.
    tempStats = os.stat(dirname)

    #Print current directory information

    print(dirname + ":")

    #Print files of directory.
    for filename in filenames:
        if(filename.endswith(".java")):
            fileTempStats = os.stat(dirname + "/" + filename)
            size += fileTempStats[6]
            tempFile = open(dirname + "/" + filename)
            tempString = tempFile.read()
            tempString = removeComments(tempString)
            public += tempString.count("public", 0, len(tempString))
            private += tempString.count("private", 0, len(tempString))
            tryCount += tempString.count("try", 0, len(tempString))
            catch += tempString.count("catch", 0, len(tempString))

    print("       ", size, "  bytes    ", public, "  public    ",
        private, "  private    ", tryCount, "  try    ", catch,
        "  catch")

The removeComments function simply removes all comments from the java files using a regular expression pattern. Thank you for any help in advance.

EDIT:

The following code was added at the beginning of the for loop:

    current_dirpath = dirname

    if( dirname != current_dirpath):
        size = 0
        public = 0
        private = 0
        tryCount = 0
        catch = 0

The output is now as follows:

testcases/part1/testcase2/root_dir/folder1/folder5:
    7609   bytes     9   public     2   private     7   try     11   catch
testcases/part1/testcase2/root_dir/folder1:
    20195   bytes     28   public     9   private     15   try     33   catch
testcases/part1/testcase2/root_dir/folder4/folder2:
    27406   bytes     37   public     11   private     19   try     42   catch
testcases/part1/testcase2/root_dir/folder4/folder3:
    27406   bytes     37   public     11   private     19   try     42   catch
testcases/part1/testcase2/root_dir/folder4:
    27406   bytes     37   public     11   private     19   try     42   catch
testcases/part1/testcase2/root_dir:
    27406   bytes     37   public     11   private     19   try     42   catch

Answer 1

I don't believe you can do this with a single counter, even bottom-up: If a directory A has subdirectories B and C, when you're done with B you need to zero the counter before you descend into C; but when it's time to do A, you need to add the sizes of B and C (but B's count is long gone).

Instead of maintaining a single counter, build up a dictionary mapping each directory (key) to the associated counts (in a tuple or whatever). As you iterate (bottom-up), whenever you are ready to print output for a directory, you can look up all its subdirectories (from the dirname argument returned by os.walk()) and add their counts together.

Since you don't discard the data, this approach can be extended to maintain separate deep and shallow counts, so that at the end of the scan you can sort your directories by shallow count, report the 10 largest counts, etc.

Answer 2

os.walk() takes an optional topdown argument. If you use os.walk(path, topdown=False) it will instead traverse directories bottom-up.

When you first start the loop save off the first element of the tuple (dirpath) as a variable like current_dirpath. As you continue through the loop you can keep a running total of the file sizes in that directory. Then just add a check like if dirpath != current_dirpath, at which point you know you've gone up a directory level, and can reset the totals.