How to organize a Python GIS-project with multiple analysis steps?

Question

I just started to use ArcPy to analyse geo-data with ArcGIS. The analysis has different steps, which are to be executed one after the other.

Here is some pseudo-code:

import arcpy

# create a masking variable
mask1 = "mask.shp"    

# create a list of raster files
files_to_process = ["raster1.tif", "raster2.tif", "raster3.tif"]

# step 1 (e.g. clipping of each raster to study extent)
for index, item in enumerate(files_to_process):
        raster_i = "temp/ras_tem_" + str(index) + ".tif"
        arcpy.Clip_management(item, '#', raster_i, mask1)

# step 2 (e.g. change projection of raster files)
...

# step 3 (e.g. calculate some statistics for each raster)
...

etc.

This code works amazingly well so far. However, the raster files are big and some steps take quite long to execute (5-60 minutes). Therefore, I would like to execute those steps only if the input raster data changes. From the GIS-workflow point of view, this shouldn't be a problem, because each step saves a physical result on the hard disk which is then used as input by the next step.

I guess if I want to temporarily disable e.g. step 1, I could simply put a # in front of every line of this step. However, in the real analysis, each step might have a lot of lines of code, and I would therefore prefer to outsource the code of each step into a separate file (e.g. "step1.py", "step2.py",...), and then execute each file.

I experimented with execfile(step1.py), but received the error NameError: global name 'files_to_process' is not defined. It seems that the variables defined in the main script are not automatically passed to scripts called by execfile.

I also tried this, but I received the same error as above.

I'm a total Python newbie (as you might have figured out by the misuse of any Python-related expressions), and I would be very thankful for any advice on how to organize such a GIS project.

Answer 1

I think what you want to do is build each step into a function. These functions can be stored in the same script file or in their own module that gets loaded with the import statement (just like arcpy). The pseudo code would be something like this:

#file 1: steps.py
def step1(input_files):
  # step 1 code goes here
  print 'step 1 complete'
  return

def step2(input_files):
  # step 2 code goes here
    print 'step 2 complete'
    return output # optionally return a derivative here

#...and so on

Then in a second file in the same directory, you can import and call the functions passing the rasters as your inputs.

#file 2: analyze.py
import steps
files_to_process = ["raster1.tif", "raster2.tif", "raster3.tif"]

steps.step1(files_to_process)

#steps.step2(files_to_process) # uncomment this when you're ready for step 2

Now you can selectively call different steps of your code and it only requires commenting/excluding one line instead of a whle chunk of code. Hopefully I understood your question correctly.