Rendering is expected to be very slow if you have tens of thousands of actors. You would normally display this many objects using a single vtkGlyph3DMapper.
Currently, there is no efficient interaction/picking mechanism in VTK that would allow you to interact with tens of thousands of points. We reimplemented widgets and interactions in 3D Slicer from almost the ground up (keeping performance and multi-view synchronization in mind) and managed to get interactive performance with tens of thousands of points. See more information in this post: picking glyphs vtkGlyph3DMapper.