ITK does not come close to what VTK can do in some aspects that are essential interactive visualization.
For example, ITK cuts off displacement to 0 outside the boundary, which prevents inverse transform computation near the boundary. VTK provides meaningful displacements in the entire spatial domain. ITK can only invert a displacement field non-linear transform and it may take minutes. If you want to transform an image and mesh using a transform that you adjust interactively then it’s just not feasible with ITK. In contrast, VTK can compute inverse of a transform in real-time (and any combination of any transforms, even if they are in a hierarchical pipeline).
Overall, if you just do bulk computation then ITK’s transform infrastructure is sufficient but for real-time interactive visualization involving non-linear transforms, then you need VTK.