# Bad regression testing

**URL:** https://discourse.vtk.org/t/bad-regression-testing/1120
**Category:** Development
**Created:** [June 12, 2019, 6:00pm UTC](https://discourse.vtk.org/t/bad-regression-testing/1120 "2019-06-12T18:00:38Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![utkarshayachit](https://discourse.vtk.org/user_avatar/discourse.vtk.org/utkarshayachit/32/379_2.png) [@utkarshayachit](https://discourse.vtk.org/u/utkarshayachit)
#### Post date: [June 12, 2019, 6:00pm UTC](https://discourse.vtk.org/t/bad-regression-testing/1120/1 "2019-06-12T18:00:38Z")

</div>

I know we’ve discussed this one in the past, but I can’t remember what the conclusion was and hence I am posting this again.

I am trying to add a test for ParaView for this [issue](https://gitlab.kitware.com/paraview/paraview/issues/19081) and am failing to get the test framework to fail. `vtkImageDifference` tells me the following images are same irrespective of whether I toggle `Averaging` or `AllowShift` flags on the filter.

![base](https://discourse.vtk.org/uploads/default/original/1X/08aa487da5b02ca4ddae5151e1856f7dc4840011.png) ![test](https://discourse.vtk.org/uploads/default/original/1X/8c02eb2e633e7dae7ce75729f38408ad6ce8f36e.png)

Using [`compare`](https://askubuntu.com/questions/209517/does-diff-exist-for-images) however, I get a better result.

```bash
> compare -verbose -metric PSNR base.png test.png diff.png
base.png PNG 300x300 300x300+0+0 8-bit sRGB 45.3KB 0.010u 0:00.000
test.png PNG 300x300 300x300+0+0 8-bit sRGB 49.7KB 0.000u 0:00.010
Image: base.png
  Channel distortion: PSNR
    red: 29.2478
    green: 29.2108
    blue: 25.6012
    all: 27.66
base.png=>diff.png PNG 300x300 300x300+0+0 8-bit sRGB 24.2KB 0.190u 0:00.030

```

Here’s the difference image generated by `compare`.  
 ![diff](https://discourse.vtk.org/uploads/default/original/1X/7bcb8e2cebeb9c7db611c009cbd39a4c04e91b68.png)

Is it time to revisit the vtkImageDifference implementation or add an alternate filter that is more in lines with tools like **ImageMagick**?

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.vtk.org/user_avatar/discourse.vtk.org/mwestphal/32/19_2.png) [@mwestphal](https://discourse.vtk.org/u/mwestphal)
#### Post date: [June 13, 2019, 7:45am UTC](https://discourse.vtk.org/t/bad-regression-testing/1120/2 "2019-06-13T07:45:54Z")

</div>

I definitely agree, our image comparison tools sometimes produce false negative.