For larger volumes, transferring the voxel data from CPU memory to GPU memory may significantly slow down the rendering, but you depending on the data size and your hardware you can reach few ten fps.
You may consider using the OpenIGTLink standard, which is developed for fast streaming of imaging, tool positions, meshes for image-guided therapy. It is well supported in the VTK-based 3D Slicer application. So first you don’t need to develop anything for image visualization, just implement a very simple imaging component that dumps imaging data in OpenIGTLink messages and use 3D Slicer to visualize the data.
See for example how 3D Slicer displays 4D (3D+t) ultrasound streamed live from a Philips ultrasound system:
If you reconstruct 3D volume from live tracked 2D frames then you don’t have to send full 3D reconstructed volumes but you can just stream the 2D frames and 3D Slicer can do the live volume reconstruction (optionally with GPU acceleration):
