vtkDistancePolyDataFilter gives incorrect distance sign

For the attached stl files, vtkDistancePolyDataFilter with signed distance ON produces incorrect signs on certain points. I could see the magnitude of the distance is correct, only the sign is flipped wrong.

Distance of meshB.stl on meshA.stl

Slightly translating the two meshes (that originally touched each other at that matching cut surface) to make the geometry better visible:

This explains what the problem is. vtkDistancePolyDataFilter uses vtkImplicitPolyDataDistance internally, which computes distance sign as “the sign of the dot product between the angle-weighted pseudonormal at the nearest surface point and the vector x - p”. This is unstable if you give such an ill-posed problem (where the vector connecting the closest points is nearly parallel with the surface normal).

If you really absolutely need signed distance then you may try to use vtkImplicitPolyDataDistance and adjust tolerance, etc. You may also add more heuristics (e.g., sign computation becomes unstable when you get far from the other model, so force sign of points that are above a certain threshold). Or, you can also make the computation fully robust by computing distance map over an image (rasterize the model and compute signed distance on the image). However, I think the proper solution would be to reformulate the question to something more meaningful (for example, ask for absolute distance when you measure distance between two non-intersecting meshes).

Thanks for the reply!
The mesh I have will always intersect, how does asking absolute distance helps?
This sign computation also gets unstable on points that are not so far from other model.
The model I shared looks easy to force sign above certain threshold, there are models which blends together (like a woven rope).
Just curious, If this is unstable how does the Boolean operations succeed? (https://www.vtkjournal.org/browse/publication/797)

Unfortunately, Boolean operations in VTK do not work reliably - they fail too often for relatively simple, completely valid inputs.