If you use any multi-threaded filter then you may benefit from using a non-sequential backend, because creation of hundreds of threads per second (this is what the sequential backend does) may have an enormous overhead on its own and it may also interfere with NVidia’s threaded optimization heuristics.
If you use VTK in a product where you control all the hardware, drivers, operating system settings, and other concurrently running applications then you can just test in that environment to see if benefits of using TBB are significant. If you develop a product that the user can install on his own computer then your best bet is to use TBB to prevent any surprises.
There are also a few filters that are specifically designed for parallel backends (flying edge filter, etc.) that are very slow if you use the sequential backend.