Bad regression testing

Development
1

I know we’ve discussed this one in the past, but I can’t remember what the conclusion was and hence I am posting this again.

I am trying to add a test for ParaView for this issue and am failing to get the test framework to fail. vtkImageDifference tells me the following images are same irrespective of whether I toggle Averaging or AllowShift flags on the filter.

base test

Using compare however, I get a better result.

> compare -verbose -metric PSNR   base.png test.png diff.png
base.png PNG 300x300 300x300+0+0 8-bit sRGB 45.3KB 0.010u 0:00.000
test.png PNG 300x300 300x300+0+0 8-bit sRGB 49.7KB 0.000u 0:00.010
Image: base.png
  Channel distortion: PSNR
    red: 29.2478
    green: 29.2108
    blue: 25.6012
    all: 27.66
base.png=>diff.png PNG 300x300 300x300+0+0 8-bit sRGB 24.2KB 0.190u 0:00.030

Here’s the difference image generated by compare.
diff

Is it time to revisit the vtkImageDifference implementation or add an alternate filter that is more in lines with tools like ImageMagick?

3 Likes
2

I definitely agree, our image comparison tools sometimes produce false negative.