Slow rendering when adding points to glyph

Hello,

We are building an application where we need to trace paths by placing spheres along said path. The spheres are added to a single vtkGlyph3D, i.e. using a single actor.

Here is a quick GIF of the process, to show what we mean by tracing a path:
Tracing path with Glpyh

We get very good performance with many spheres (>10.000) when interacting with the scene, moving/rotating the view and selecting. However, we experience very long rendering times while we are adding new spheres.
E.g. turning off the tracing allows us to interact with high fps, but turning the tracing back on, so that spheres are added continuously, brings the fps down to as low as 5-10.

The following code snippet shows how we set up the pipeline for the vtkGlyph3D with a sphere source. This is done once at the start of the program. The glyph actor is added to the renderer right away.

vtkSmartPointer<vtkPolyData> sphere_positions_polydata = vtkSmartPointer<vtkPolyData>::New();
sphere_positions_polydata->SetPoints(glyph_component.positions);
sphere_positions_polydata->GetPointData()->SetScalars(glyph_component.colors);

vtkSmartPointer<vtkSphereSource> sphere_source = vtkSmartPointer<vtkSphereSource>::New();
sphere_source ->SetRadius(sphere_size);
sphere_source ->SetPhiResolution(phi_resolution);
sphere_source ->SetThetaResolution(theta_resolution);

sphere_source->GenerateNormalsOff(); // Tried these to increase performance
sphere_source->SetOutputPointsPrecision(vtkAlgorithm::SINGLE_PRECISION); // Tried these to increase performance

vtkSmartPointer<vtkGlyph3D> glyph = vtkSmartPointer<vtkGlyph3D>::New();
glyph->SetSourceConnection(sphere_source->GetOutputPort());
glyph->SetInputData(sphere_positions_polydata);

glyph->SetColorModeToColorByScalar();
glyph->SetScaleModeToDataScalingOff(); 
glyph->GeneratePointIdsOn();
glyph->Update();

vtkSmartPointer<vtkPolyDataMapper> glpyh_mapper = vtkSmartPointer<vtkPolyDataMapper>::New();
glpyh_mapper->SetInputConnection(glyph->GetOutputPort());
glpyh_mapper->Update();

actor_component.actor->SetMapper(glpyh_mapper);
actor_component.actor->GetProperty()->SetPointSize(10.0);

And the next snippet shows how we add more spheres. As the pipeline is connected, all we need to do is update the input polydata object with a new position and color:

// Glyph component is an alias for sphere_positions_polydata from above
glyph_component.positions->InsertNextPoint(pos[0], pos[1], pos[2]);
glyph_component.positions->Modified();

We’ve done some profiling to analyse the problem and we’ve found that the time it takes to call vtkRenderWindow::Render(), while the tracing is enabled, increases as more spheres are present. The time spent inside vtkRenderWindow::Render() can get up to around 300ms. And then as soon as the tracing is disabled (all the spheres are still present, but no more are added) the rendering time goes down to about 5-6ms, which is a perfect duration for us.

So, our question is: How can we increase the performance while adding spheres to the glyph?