Bad regression testing

I know we’ve discussed this one in the past, but I can’t remember what the conclusion was and hence I am posting this again.

I am trying to add a test for ParaView for this issue and am failing to get the test framework to fail. vtkImageDifference tells me the following images are same irrespective of whether I toggle Averaging or AllowShift flags on the filter.

base test

Using compare however, I get a better result.

> compare -verbose -metric PSNR   base.png test.png diff.png
base.png PNG 300x300 300x300+0+0 8-bit sRGB 45.3KB 0.010u 0:00.000
test.png PNG 300x300 300x300+0+0 8-bit sRGB 49.7KB 0.000u 0:00.010
Image: base.png
  Channel distortion: PSNR
    red: 29.2478
    green: 29.2108
    blue: 25.6012
    all: 27.66
base.png=>diff.png PNG 300x300 300x300+0+0 8-bit sRGB 24.2KB 0.190u 0:00.030

Here’s the difference image generated by compare.

Is it time to revisit the vtkImageDifference implementation or add an alternate filter that is more in lines with tools like ImageMagick?


I definitely agree, our image comparison tools sometimes produce false negative.