I usually report errors using vtkErrorMacro and capture them in tests using a custom vtkTestingOutputWindow.
The main difference compared to using an explicit error observer (as shown in @lorensen’s example above) is that using a custom output window you capture all errors from all filters and you do not need to add any extra code to your tests (unless you want to check for expected errors) - you just need to add a few extra lines into your CMakeLists.txt.
I originally tried the error observer method, and it works ok. I could catch errors in my filter but it still showed messages for Pipeline failure. The TestingOutputWindow approach seems to do what I want though.