Sorry for bringing this up again.
I have made the classes below for registrations of vtkPolyData. Using a point-to-plane metric (easy addition) makes the convergence about 10 times faster. Also, it is important to have a MaxDistance to threshold the correspondences found.
I am using vtkSMPTools for parallelizing the closest points found using vtkImplicitPolyDataDistance, I will off course need to use the dispatcher and change the array to vtkDataArray.
Next step is to solve the problem using Gauss-Newton. Right now, correspondences are found in VTK and the registration is made using Eigen.
It is a work-in-progress.
Article with the algorithms: https://www.robots.ox.ac.uk/~vgg/publications/2001/Fitzgibbon01c/fitzgibbon01c.pdf
In order for this to fix into VTK, I would probably need to use PCA for for finding normals in case either source or target has no normals.
The reason for the interface vtkPolyDataRegistrationTransform is because I am working with multiple registration algorithms and all of them need correspondences and it is convenient to have the calculations of those in a base class.
An alternative would be to introduce a filter with two inputs, a method for setting the current transform and this should output two point clouds with normals if it is configured for this. We could call this vtkPolyDataCorrespondenceFilter.
What are you thoughts
I have started a small open-source project demonstrating the idea, where also the point-to-plane registration is implemented.