The background to this is that I have created a MR: Moving to ssim and fixing tests which seems to be working in that, for Linux, only two fail out of 1023 C++ tests and, for Windows, 18 tests fail out of 1021. So I will probably merge soon.
However I came across a problem in that the documentation for SSIM tests is sparse and geared to VTK builds and testing.
I wanted a “simple” way of testing a program that can be readily moved to another machine. For instance, my laptop consistently produces images that are around 30px shorter in height yet, if I export the image directly from the code, it is the correct size. I wanted something compact and fast to build and run tests with.
The “simple” way of testing a program was more complex than I thought and I subsequently ended up creating a folder called TestingFramework that is portable and easy to dump code in to test: TestFramework.tar.gz (815.5 KB)
In the Documentation subfolder there are usage instructions and and I collated the useful stuff that I got from searches into the document ImageTestingFramework.md. The framework works in Linux and Windows.
One thing missing in there is a Python testing framework - I have no idea how to do it, However I have put a couple of Python programs in there to tempt someone!
So, if it is of general use to people, where could it go?
Would people like to make it better?
If you set DEFAULT_USE_SSIM_IMAGE_COMP in cmake as true, you enable the ssim testing framework, and the default threshold should become 0.05.
In rtImageTest.py, as you can see it falls back to using the vtkTesting.cxx, and has the 0.05 threshold already set.
For tests that compare text that may differ between platform, use LOOSE_VALID as the test’s cmake parameter. You can easilly find an example in VTK of how it’s used.
All done and merged, SSIM works OK. Can you change the threhold?
I have a pending announcement that hasn’t been approved yet, so here is an extract:
Linux
99% tests passed, 2 tests failed out of 1023.
IndividualVRML now cannot run teapot.wrl as a test, it crashes for some reason, so I created utah_teapot.wrl from a recent version of utah_teapot.obj for testing purposes.