vtkSmartPointer and vtkNew in VTK's API. Best practices?

I think it is fine, it is a relatively rare case of API backward incompatibility that should be manageable by proper documentation/migration guide.

To me these look quite big changes and would bring limited benefits. If we decide to completely eliminate raw pointers from all interfaces then it may worth to do it, but as a partial solution, introduced gradually could be quite painful. I would say developers tolerate sweeping API changes in every 5 years or so. For me, the last one was switching to the OpenGL2 backend. Maybe the next one (in 3-4 years) can include getting rid of most raw pointer usage.

They key information that must be documented is not reference counting (that’s an implementation detail), but ownership. Ownership is a simple concept in very specific cases, such as factory methods, but in general they may be very complicated - you cannot document them by using one of 2-3 pointer types. For example, you may use a smart or weak pointer to keep a reference to an object - how would you indicate that in the API? Or, a method might not store the pointer inside its own class but pass the pointer to several other objects that store it in smart pointers - how would you document this using pointer types?

We could add a new macro - vtkSetObjectReferenceMacro or something like that.