# Do we need a testing framework example?
**URL:** https://discourse.vtk.org/t/do-we-need-a-testing-framework-example/16399
**Category:** Development
**Tags:** proposal
**Created:** [April 28, 2026, 6:21am UTC](https://discourse.vtk.org/t/do-we-need-a-testing-framework-example/16399 "2026-04-28T06:21:08Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@amaclean](https://discourse.vtk.org/u/amaclean)
#### Post date: [April 28, 2026, 6:21am UTC](https://discourse.vtk.org/t/do-we-need-a-testing-framework-example/16399/1 "2026-04-28T06:21:09Z")
The background to this is that I have created a MR: [Moving to ssim and fixing tests](https://gitlab.kitware.com/vtk/vtk-examples/-/merge_requests/459) which seems to be working in that, for Linux, only two fail out of 1023 C++ tests and, for Windows, 18 tests fail out of 1021. So I will probably merge soon.
However I came across a problem in that the documentation for SSIM tests is sparse and geared to VTK builds and testing.
I wanted a “simple” way of testing a program that can be readily moved to another machine. For instance, my laptop consistently produces images that are around 30px shorter in height yet, if I export the image directly from the code, it is the correct size. I wanted something compact and fast to build and run tests with.
The “simple” way of testing a program was more complex than I thought and I subsequently ended up creating a folder called **TestingFramework** that is portable and easy to dump code in to test:
[TestFramework.tar.gz](https://discourse.vtk.org/uploads/short-url/c0ikOfJRYkkA47JBJ6CCz5VEhia.gz) (815.5 KB)
In the Documentation subfolder there are usage instructions and and I collated the useful stuff that I got from searches into the document **ImageTestingFramework.md**. The framework works in Linux and Windows.
One thing missing in there is a Python testing framework - I have no idea how to do it, However I have put a couple of Python programs in there to tempt someone!
So, if it is of general use to people, where could it go?
Would people like to make it better?
---
### Author:  [@spyridon97](https://discourse.vtk.org/u/spyridon97)
#### Post date: [April 28, 2026, 5:19pm UTC](https://discourse.vtk.org/t/do-we-need-a-testing-framework-example/16399/2 "2026-04-28T17:19:46Z")
If you set `DEFAULT_USE_SSIM_IMAGE_COMP` in cmake as true, you enable the ssim testing framework, and the default threshold should become 0.05.
In rtImageTest.py, as you can see it falls back to using the vtkTesting.cxx, and has the 0.05 threshold already set.
For tests that compare text that may differ between platform, use LOOSE\_VALID as the test’s cmake parameter. You can easilly find an example in VTK of how it’s used.
Let me know if that helps.
---
### Author:  [@amaclean](https://discourse.vtk.org/u/amaclean)
#### Post date: [May 2, 2026, 6:01am UTC](https://discourse.vtk.org/t/do-we-need-a-testing-framework-example/16399/3 "2026-05-02T06:01:41Z")
All done and merged, SSIM works OK. Can you change the threhold?
I have a pending announcement that hasn’t been approved yet, so here is an extract:
## Linux
99% tests passed, 2 tests failed out of 1023.
- [IndividualVRML](https://examples.vtk.org/site/Cxx/IO/IndividualVRML/) now cannot run `teapot.wrl` as a test, it crashes for some reason, so I created `utah_teapot.wrl` from a recent version of `utah_teapot.obj` for testing purposes.
All tests pass except:
- Images-ImageContinuousDilate3D (Subprocess aborted)
- Images-ImageContinuousDilate3D (Subprocess aborted)
The crash happens at line 79 in vtkDataArrayValueRange\_AOS.h when the PNG file is used.
## Windows
98% tests passed, 16 tests failed out of 1021.
Nine failures are due to the usage of the `vtkBYUReader` and one due to the use of the `vtkSLCReader` please see: [Bug in vtkBYUReader and vtkSLCReader](https://gitlab.kitware.com/vtk/vtk/-/work_items/20012)
As in Linux, these two tests also fail:
- Images-ImageContinuousDilate3D (Subprocess aborted)
- Images-ImageContinuousDilate3D (Subprocess aborted)
The crash happens at line 79 in vtkDataArrayValueRange\_AOS.h when the PNG file is used.
Points-SignedDistance needs `VTK_TESTING_IMAGE_COMPARE_METHOD=LOOSE_VALID` in order to pass.
InfoVis-XGMLReader fails in windows even when `VTK_TESTING_IMAGE_COMPARE_METHOD=LEGACY_VALID` is set. `TIGHT_VALID` is Ok in Linux.
---
### Author:  [@spyridon97](https://discourse.vtk.org/u/spyridon97)
#### Post date: [May 4, 2026, 2:20pm UTC](https://discourse.vtk.org/t/do-we-need-a-testing-framework-example/16399/4 "2026-05-04T14:20:25Z")
Could you create an issue in VTK with links to the cdash failures? Ideally also copy paste the images please.
---
### Author:  [@amaclean](https://discourse.vtk.org/u/amaclean)
#### Post date: [May 5, 2026, 5:39am UTC](https://discourse.vtk.org/t/do-we-need-a-testing-framework-example/16399/5 "2026-05-05T05:39:07Z")
I have created these VTK issues regarding the failures.
- [Assertion failure in vtkDataArrayValueRange\_AOS.h](https://gitlab.kitware.com/vtk/vtk/-/work_items/20040) this applies to both Linux and Windows
- [IndividualVRML failure](https://gitlab.kitware.com/vtk/vtk/-/work_items/20039)
- [XGMLReader rotated image in windows](https://gitlab.kitware.com/vtk/vtk/-/work_items/20041)
- [Bug in vtkBYUReader and vtkSLCReader](https://gitlab.kitware.com/vtk/vtk/-/work_items/20012)
Regarding the vtkBYUReader and vtkSLCReader only the Windows examples fail.
For vtkBYUReader:
```plaintext
ReadAllPolyDataTypesDemo, CapClip, HighlightSelection, CellCentersDemo, PolyData-CellsInsideObject, WalkCowA, WalkCowB, BozoShaderDemo, MarbleShaderDemo, Motor
```
For vtkSLCReader:
```plaintext
ReadSLC, IntermixedUnstructuredGrid
```