VTK Examples Python testing

I’m putting together some plans for automating the testing the Python examples in the vtk-examples repository. All comments are welcome, especially from @amaclean since you have already done a lot of great work with these examples.

The goal is to make the python examples introspectable, so that we don’t have to add each example to the CMakeLists.txt scripts by hand. Much of this is already done, because each example has a main() method, uses argparse where arguments are required, and clearly defines which input data file must be passed as an argument, e.g.:

parser.add_argument('filename', help='FullHead.mhd.')

Hence, running an example with the --help option will indicate the needed input file. However, I want to change the examples so that the input file can be introspected from another Python program before the example is run, by importing the example as a module.

import MyExample

try:
    # the example's 'inputfiles' attribute, if present, will be a list of filenames.
    inputfiles = [os.path.join(TestingData, filename) for filename in MyExample.inputfiles]
except AttributeError:
    inputfiles = []

# the example's main() will take an argv-style list of arguments
MyExample.main(['MyExample'] + inputfiles)

Hence it would be possible to make a single Python front-end for all the Python examples. Such a front end could also generate any cmake bits that are needed to integrate the examples with ctest.

The two changes that I hope to make first are as follows:

  1. add an inputfiles attribute to each example that needs an input data file
  2. change main() to main(argv) in all examples (plus related argparse changes)

I already have a working script to automatically apply change (2) to each example, and am adding (1) to the same script. Hopefully I will be able to commit something for review this weekend.

1 Like

@dgobbi +1! Looking forward to it. It sounds really good.

I have a rough merge request for the vtk-examples repository. Currently it finds every “.py” example with a corresponding “.md” file and generates an add_test() in the CMakeLists.txt. So it definitely tests more than it should. Also it doesn’t have a good pass/fail check.

Other than the add_test(), the most important component is a test driver script:

python RunPythonTest.py -D Testing/Data Python/VisualizationAlgorithms/HeadBone.py

The test driver can automatically add any needed arguments for the examples (e.g. data files). When the test driver is complete, it will also handle regression testing, probably by overriding the interactor so that it automatically generates and checks a regression image.

Just had a look at the code, it is looking good.
I have a question about:

Is the “md” file that you refer to here, the file that appears alongside the example? If so, not all examples have a corresponding “md” file e.g. src/Python/GeometricObjects/Pyramid.py

I’ve modified the CMakeLists.txt so that it grabs all .py examples, not just the ones with a matching .md.

Also I’ve modified the test driver so that the interactor will render the scene and then exit. Currently “ctest -R Python” will run most of the examples. Still a fair bit of work to do, though.