Testing IPython Notebooks

Question

I'm starting to use IPython notebooks to document some of my code with interactive usage examples. In order to avoid having the documentation get too far out of date from the code, I'd like the code in the notebook to get executed on a regular basis to catch any changes in output and to flag runtime errors.

I use nosetests to run regression tests and was wondering if there is a way to have it execute IPython Notebooks for this purpose. Note that I'm not trying to run nosetests from within the IPython notebook (as is done in ipython_nose). Something more along the lines of the doctest plugin. Does such a plugin exist?

Answer 1

I don't know of an actual nose plugin to do it automatically, but here is a script showing the basic pieces that would be necessary for such a thing. Others have since forked it to add some functionality.

The gist is that you create the same Kernel object that the notebook uses, and run through the same execution that 'Run All' would do, and compare the resulting output. It has some primitive sanitization, which could probably largely be replaced by the right doctest functions, but it's not super complicated.

Answer 2

I had recently written a script that does something similar and most of it was based on this blog on 'Testing Jupyter Notebooks'

Here's the slightly modified version from the one on the blog:

from glob import glob

import nbformat
from nbconvert.preprocessors import ExecutePreprocessor
from nbconvert.preprocessors.execute import CellExecutionError

def _notebook_run(path):
  """
  Execute a notebook via nbconvert and collect output.
   :returns (parsed nb object, execution errors)
  """
  kernel_name = 'python%d' % sys.version_info[0]
  this_file_directory = os.path.dirname(__file__)
  errors = []


  with open(path) as f:
    nb = nbformat.read(f, as_version=4)
    nb.metadata.get('kernelspec', {})['name'] = kernel_name
    ep = ExecutePreprocessor(kernel_name=kernel_name, timeout=10) #, allow_errors=True

    try:
      ep.preprocess(nb, {'metadata': {'path': this_file_directory}})

    except CellExecutionError as e: 
      if "SKIP" in e.traceback:
        print(str(e.traceback).split("\n")[-2])
      else:
        raise e

  return nb, errors

You could now use this function as:

def test_notebooks():
  for notebook in glob("./*.ipynb"):
    nb, errors = _notebook_run(notebook)
    assert errors == []

Answer 3

nosebook may suit your purposes. I built it to handle just such cases: it requires no special markup in the notebook, and does some of the sanitization mentioned by @minrk.

Answer 4

Recently Andrea Zonca published pytest-ipynb. I have not tested it yet but it seems to fit your requirements (well maybe is not targeted for nose, but it has neat features like cell indication on failures). I will definitely use it to test assignments and material for students :)