Slightly translating the two meshes (that originally touched each other at that matching cut surface) to make the geometry better visible:
This explains what the problem is. vtkDistancePolyDataFilter uses vtkImplicitPolyDataDistance internally, which computes distance sign as “the sign of the dot product between the angle-weighted pseudonormal at the nearest surface point and the vector x - p”. This is unstable if you give such an ill-posed problem (where the vector connecting the closest points is nearly parallel with the surface normal).
If you really absolutely need signed distance then you may try to use vtkImplicitPolyDataDistance and adjust tolerance, etc. You may also add more heuristics (e.g., sign computation becomes unstable when you get far from the other model, so force sign of points that are above a certain threshold). Or, you can also make the computation fully robust by computing distance map over an image (rasterize the model and compute signed distance on the image). However, I think the proper solution would be to reformulate the question to something more meaningful (for example, ask for absolute distance when you measure distance between two non-intersecting meshes).