FXAA is bad at handling thin lines, but this looks especially wrong. Are you configuring the render window without an alpha channel? The “steps” of the anti-aliased line edge should be dropping off in opacity to give a smooth appearance, but they seem to be maintaining uniform opacity. What is the output of renWin->Print(std::cout); (or print(renWin) on python)?
If MSAA and FXAA aren’t working, there’s also vtkSSAAPass that does super-sampled anti-aliasing. It’s an expensive option, but might work better for you.