Implementing a recursive copy in Python with special case handling

Question

I'd like to implement a version of cp -r in Python that handles certain directories in a special way. If you do mycp.py -r indir outdir, I'd like indir and all of its files/subdirectories to be copied exactly into outdir, except for certain filenames. What is the most portable way to do this in Python?

Example: I have this directory structure:

dir1/
  file1
  dir2/
    dir3/
  specialdir/
    myfile.bar

file1 is a file and specialdir is a directory containing a file myfile.bar. I'd like to copy dir1 and all of its contents, but handling specially directories that have *.bar files in them. In this case only specialdir meets the criteria. I'd like to mycopy to copy all of dir1 but replace any special directories with a zipped version of themselves. In the above example that would mean copying dir1 as is, but replacing specialdir with specialdir.zip that might contain a processed version of myfile.bar.

I tried to follow the suggestion below but I'm not sure how to deal with the copying:

import os
import shutil

SPECIAL_DIRS = []

def is_special_dir(path, dirnames):
    """directories are special if they have .bar files"""
    special_dirs = []
    for d in dirnames:
        d = os.path.join(path, d)
        if os.path.isdir(d):
            files_in_d = os.listdir(d)
            for f in files_in_d:
                if f.endswith(".bar"):
                    # directory is special if it contains
                    # .bar files
                    special_dirs.append(d)
    SPECIAL_DIRS.extend(special_dirs)
    return special_dirs

def my_copy(indir, outdir):
    shutil.copytree(indir, outdir, ignore=is_special_dir)
    print "Found special dirs: ", SPECIAL_DIRS

# make a copy of dir1 but handle special directories
# differently
my_copy("dir1", "copy_dir1")

If I try it it correctly detects the special directories:

$ copy_test.py
Found special dirs:  ['dir1/dir2/specialdir']

how can I make make it insert specialdir in the right corresponding place in copy_dir1? I'd like copy_dir1 (the destination dir) to have exactly the same structure as dir1 (the source dir) except with special handling of directories containing the .bar files.

Answer 1

It sounds like you want shutil.copytree utilizing the ignore parameter:

If ignore is given, it must be a callable that will receive as its arguments the directory being visited by copytree(), and a list of its contents, as returned by os.listdir(). Since copytree() is called recursively, the ignore callable will be called once for each directory that is copied. The callable must return a sequence of directory and file names relative to the current directory (i.e. a subset of the items in its second argument); these names will then be ignored in the copy process. 'ignore_patterns()' can be used to create such a callable that ignores names based on glob-style patterns.

So something like this should work:

def what_to_ignore(path,names):
    if is_special(path):
        # process names here returning any or all to ignore

shutil.copytree(indir,outdir,ignore=what_to_ignore)

Edit for expanded question and example

Here's an example. The simplified ignore function still creates an empty special directory, but it is easy to delete before doing the special zip copy. I also nested the special function so my_copy could be used more than once without relying on using a global variable. Doing the zip is an exercise for the user:

import fnmatch
import shutil
import os

def my_copy(indir, outdir):

    special = []

    def is_special_dir(path, names):
        """directories are special if they have .bar files"""
        if fnmatch.filter(names,'*.bar'):
            special.append(path)
            return names
        return []    

    shutil.copytree(indir, outdir, ignore=is_special_dir)
    print('Found special dirs:',special)

    for src in special:
        rel = os.path.relpath(src,indir)
        dst = os.path.join(outdir,rel)
        os.rmdir(dst)
        print('Zip "{}" to "{}.zip"'.format(src,dst))

my_copy('dir1','dira')
my_copy('dir1','dirb')

Output

Found special dirs: ['dir1\\specialdir']
Zip "dir1\specialdir" to "dira\specialdir.zip"
Found special dirs: ['dir1\\specialdir']
Zip "dir1\specialdir" to "dirb\specialdir.zip"