vtkImageDifference does not detect different images

Hi all,

In the CI of Themys, our ParaView Branded Application, we use the module Testing.py which makes use of the vtkImageDifference algorithm.

It seems that with default configuration this algorithm does not detect that the two images below are different.

After reading vtkImageDifference does not detect image differences - #6 by mwestphal i tried to add idiff.SetThreshold(0) in Testing.py and with this modification the images are correctfully detected as different.

I do not know if it is a known corner case of the vtkImageDifference algorithm or a bug.
By the way should i use a custom python script with idiff.SetThreshold?

Thanks in advance


It is a known issue. We are currently in the process of changing the metric used for image comparison in regression testing (see here). This will be quite impactful for VTK and projects depending on it so it will be opt in at first. The current metric is too forgiving and does not capture enough information in the image to be satisfactory.

Ok. Thanks for this quick answer @Yohann_Bearzi

TBH making the treshold default to zero and improving the doc would be a very simple test, especially considering that the new class will not replace the old class for compat reasons.

@mwestphal @Yohann_Bearzi If i understand and read correctly the source code, i can have the strictest options for comparison by typing:

    # Here we want to be as strict as possible so do not set threshold
    # Do not compare images by allowing a shift of two pixels between images
    # Do not compare averaged 3x3 data between the images

Am i right?

Indeed, but expect the result to almost never be zero.

Ok. Thanks @mwestphal