The 2000+ tests in VTK should answer your questions. If you need guidance on application level testing, you might have a look at the 1000+ test of ParaView or 600+ tests of 3D Slicer, which include testing of GUI widgets, complex workflows, event recording/replay based tests, etc.
In general, it is indeed very hard to verify that information displayed on the GUI is correct, so application testing often happens at lower levels: for example, you query GUI widgets about what they display instead of trying to do OCR or image comparison.
Since large part of VTK is about rendering, results must be checked visually. VTK does this pretty well, with its image-based regression testing infrastructure.
Probably not. If you spend too much time with developing and maintaining automatic tests then you will not have enough time for everything else.
I usually determine the minimum level of testing by asking myself: Would I be embarrassed if this was broken? If yes, then I add an automatic test for it.