Yes, vtkSmartPointer<>::operator=
is much slower than a pointer copy, but it is still extremely fast: vtkSmartPointer copy takes 300ns, while a regular pointer copy takes 2ns (in debug mode; in release mode, vtkSmartPointer copy takes 45ns, while the simple pointer copy test was removed by compiler optimization; measured by copying then setting a pointer to nullptr 1 million times).
Making std::move release reference in the old pointer is the right thing to do, but performance improvement is expected to be negligible (unless you copy millions of VTK object pointers per second).