itk-wasm is necessary if you want to read NRRD, MHA, NIFTI, etc. vtk.js only supports reading VTI image otherwise.
One other alternative is to send the pixel buffer plus the image metadata (dimensions, origin, spacing, orientation/direction) and then reconstruct it on the client. In fact, if you grep through the vtk.js examples, you will see examples that use LIDC2.vti, which is broken down into a metadata file and a gzipped pixel buffer.
You can almost express a volume with VTI. Unfortunately, VTI has no fields for specifically storing image orientation, which is a crucial bit of info for DICOM images. Hence you will need to transmit just the orientation information in a separate manner if you opt to use VTI.
DICOM has a lot of different metadata information, so yes there is metadata loss. If you just want 3D + 3 slice view, all you should need is the direction matrix.