The vtkRenderWindow contains a collection of vtkRenderers, and parameters that control rendering features such as stereo, anti-aliasing, motion blur and focal depth.
You can enable FXAA by flipping a switch in the renderer: UseFXAAOn(). It is simple and fast but it just a 2D image post-processing operation, therefore it removes all kinds of details from your rendered image. There are many parameters that you can tune though, so there is a good chance that the image quality will be overall better than without using anti-aliasing at all.
It is all just 2D software post-processing, so it is quite limited in what it can do, but it is fully hardware-independent and does not interfere with 3D rendering passes.
FXAA is fast and does a decent job, but some people don’t like its visual results. Skyrim, a game from Bethesda, uses FXAA and is criticized for its blurry looks by some of its users. I let this up to the user (choose FXAA, MSAA or none). I personally find MSAA better. Just try each and choose the one that you feel right.
FXAA can be good for artistic visualization, where accurate colors and sizes are not crucial. I would consider risky to use FXAA for scientific visualization, as it alters the information: it blurs objects as if they had equal importance, while in correct anti-aliasing what is in front should never be altered by what is behind (unless it is semi-transparent).
For example, see discoloration in the purple line as it goes above green and blue background:
Discourse scales down the image in the post, so you can only see the difference in the zoomed-in red circles, or if you click on the image, download the full resolution image, and view it on your screen 1:1 (without any rescaling).
You can do true antialiasing, with multisampling (MSAA) - see vtkRenderWindow::SetMultiSamples(). If you use a GUI toolkit then you may need to enable MSAA in the application and/or embedding rendering widget, too.
This is exactly what is hard with hardware MSAA. It depends on the hardware, operating system, type of render window, etc. Do you embed the render window into a GUI toolkit (Qt, wxWidgets, …) or it is a plain vtkRenderWindow? How many samples did you set? You may get some hints about what goes wrong if you run your code step by step in a debugger.