Custom SCons Builder reports dependencies as current even when files have been changed

Question

I have a custom builder that uses a tool to build documentation using source code and text files with markdown documentation.

The tool takes a configuration file that specifies all the input files and output options.

when run it produces documentation in a folder labeled html.

my builder has a scanner to find all the input files

and an emitter to set the output directory.

the scanner and emitter find all of the files need. However when I rebuild it does not detect input file changes.

I have produced a builder that reproduces the problem put the following in a single directory:

gen_doc.py

import SCons.Builder
import os
import ConfigParser

def _doc_build_function(target, source, env):
    #print '***** Builder *****'
    config = ConfigParser.SafeConfigParser()
    try:
        fp = open(str(source[0]), 'r')
        config.readfp(fp)
    finally:
        fp.close()
    output_dir = ''
    if config.has_option('output_options', 'output_dir'):
        output_dir = config.get('output_options', 'output_dir')
    input_files = []
    if config.has_option('input_options', 'input'):
         input_files = config.get('input_options', 'input').split()
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    with open(output_dir + os.sep + 'index.html', 'wb') as out_file:
        for file in input_files:
            try:
                in_file = open(file, 'r')
                out_file.write(in_file.read())
            finally:
                in_file.close()


def _doc_scanner(node, env, path):
    source = []
    config = ConfigParser.SafeConfigParser()
    try:
        fp = open(str(node), 'r')
        config.readfp(fp)
    finally:
        fp.close()
    if config.has_option('input_options', 'input'):
        for i in config.get('input_options', 'input').split():
            source.append(os.path.abspath(i))
    return source

def _doc_emitter(target, source, env):
    target = []
    config = ConfigParser.SafeConfigParser()
    try:
        fp = open(str(source[0]), 'r')
        config.readfp(fp)
    finally:
        fp.close()
    if config.has_option('output_options', 'output_dir'):
        target.append(env.Dir(os.path.abspath(config.get('output_options', 'output_dir'))))
        env.Clean(source, env.Dir(os.path.abspath(config.get('output_options', 'output_dir'))))

    return target, source


def generate(env):
    doc_scanner = env.Scanner(function = _doc_scanner)

    doc_builder = SCons.Builder.Builder(
        action = _doc_build_function,
        emitter = _doc_emitter,
        source_scanner = doc_scanner,
        single_source = 1
    )

    env.Append(BUILDERS = {
        'gen_doc': doc_builder,
    })

def exists(env):
    '''Using internal builder'''
    return True

SConstruct

env = Environment()
env.Tool('gen_doc', toolpath=['.'])
env.gen_doc('config_doc')

config_doc

[input_options]
input = a.md b.md

[output_options]
output_dir = html

a.md

Hello

b.md

 world

when I run this it produces the correct output

a file in the folder html with a file called 'index.html'

with the words Hello world

when I run

scons -n tree=status html

I get the following

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `html' is up to date.
 E         = exists
  R        = exists in repository only
   b       = implicit builder
   B       = explicit builder
    S      = side effect
     P     = precious
      A    = always build
       C   = current
        N  = no clean
         H = no cache

[E B   C  ]+-html
[E     C  ]  +-config_doc
[E     C  ]  +-a.md
[E     C  ]  +-b.md
scons: done building targets.

I go in and modify the b.md file and re-run

scons -n tree=status html

the output is the same it still reports b.md as current so the documentation is not rebuilt.

Is there a way to make scons see the changes to the source files seen by the scanner and rebuild when the files are changed?

Update

I did a little playing around I created a dummy Decider to see if I could find out why these files were not being added.

def foo(dependency, target, prev_ni):
    print 'dependency = %s' % (dependency)
    print 'target = %s' % (target)
    return True

in the 'generate(env)' I add the line 'env.Decider(foo)'

The files added to the tree by _doc_scanner are not calling the Decider function so a MD5 hash is never being calculated.

What can I do so these files do call the Decider?

update2:

forgot to add the return for the emitter when making the post.

Update 3

Modified the code so that it no longer calls an external builder. It now calls an internal builder function that emulates the builder. This is only emulating the behavior of an external builder. original builder action was action = 'cd ${SOURCE.dir} && gen_docs ${SOURCE.file}

Answer 1

This is caused by what I would consider a design bug in SCons: directory nodes are always considered up-to-date if the directory exists.

Relevant section from the SCons FAQ:

Why is my directory only updated the first time?

Like every other build system, SCons considers a directory used as a target as up-to-date if it exists. The first time you built, the directory wasn't there, so SCons ran the update command. Each time after that, the directory already existed, so SCons considered it up-to-date.

You can work around this, thought it's a bit of a pain. For every directory that you want to participate in the dependency graph, you need to create a dummy file that "represents" that directory. Write to the file whenever you generate the directory. Depend on the file instead of on the directory.

Your code can be updated to do this, thusly:

import SCons.Builder
import os
import ConfigParser
import datetime

def _manifest(target):
    return os.path.join('.manifest', str(target))

def _touch(path):
    dirname = os.path.dirname(path)
    if not os.path.exists(dirname):
        os.makedirs(dirname)
    with open(path, 'wt') as f:
        f.write(str(datetime.datetime.now()))

def _doc_build_function(target, source, env):
    #print '***** Builder *****'
    config = ConfigParser.SafeConfigParser()
    try:
        fp = open(str(source[0]), 'r')
        config.readfp(fp)
    finally:
        fp.close()
    output_dir = ''
    if config.has_option('output_options', 'output_dir'):
        output_dir = config.get('output_options', 'output_dir')
    input_files = []
    if config.has_option('input_options', 'input'):
         input_files = config.get('input_options', 'input').split()
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    with open(output_dir + os.sep + 'index.html', 'wb') as out_file:
        for file in input_files:
            try:
                in_file = open(file, 'r')
                out_file.write(in_file.read())
            finally:
                in_file.close()

    for t in target:
        _touch(_manifest(t))


def _doc_scanner(node, env, path):
    source = []
    config = ConfigParser.SafeConfigParser()
    try:
        fp = open(str(node), 'r')
        config.readfp(fp)
    finally:
        fp.close()
    if config.has_option('input_options', 'input'):
        for i in config.get('input_options', 'input').split():
            source.append(os.path.abspath(i))
    return source

def _doc_emitter(target, source, env):
    target = []
    config = ConfigParser.SafeConfigParser()
    try:
        fp = open(str(source[0]), 'r')
        config.readfp(fp)
    finally:
        fp.close()
    if config.has_option('output_options', 'output_dir'):
        target.append(env.Dir(os.path.abspath(config.get('output_options', 'output_dir'))))
        env.Clean(source, env.Dir(os.path.abspath(config.get('output_options', 'output_dir'))))

    target.extend(map(_manifest, target))

    return target, source


def generate(env):
    doc_scanner = env.Scanner(function = _doc_scanner)

    doc_builder = SCons.Builder.Builder(
        action = _doc_build_function,
        emitter = _doc_emitter,
        source_scanner = doc_scanner,
        single_source = 1
    )

    env.Append(BUILDERS = {
        'gen_doc': doc_builder,
    })

def exists(env):
    '''Using internal builder'''
    return True

Answer 2

Your emitter is not returning the modified target, source lists.

See http://www.scons.org/doc/HTML/scons-user/x3798.html for more info:

The emitter function should return the modified lists of targets that should be built and sources from which the targets will be built.