Thanks for your confirmation.
Issue filled: https://gitlab.kitware.com/vtk/vtk/-/issues/19706
To document a bit more: seems negative numbers behave a bit better. Result is numerical, but still erroneous.
normals.InsertNextTuple3(1, 0, 0);
normals.InsertNextTuple3(1, 0, 0);
normals.InsertNextTuple3(1, 0, 0);
vectors.InsertNextTuple3(-1, 1, 1); // negative value here
vectors.InsertNextTuple3(.707, .707, 0);
vectors.InsertNextTuple3(0, 1, 0);
vtkVectorDot vectorDot = new vtkVectorDot();
vectorDot.MapScalarsOn();
vectorDot.SetScalarRange(0, 1.0);
vectorDot.Update();
returns
Value 0 : 1.0
Value 1 : 1.0
Value 2 : 1.0