Boolean mesh implementations are really hard to implement correctly and they often fail (even in very expensive commercial software). Unfortunately, VTK’s implementation is quite fragile and it often generates incorrect meshes for complex inputs (and sometimes even for quite simple inputs, too). You may try to slightly move around the meshes until there are no errors in the output mesh.
If you need a definitive solution that always produces correct results then one option is to convert the mesh to binary labelmap, perform Boolean operations on the labelmap, then convert the labelmap back to surface mesh. Disadvantage of this approach that original model points are not preserved and you may need a lot of memory if you need preserve mesh geometry very accurately. If you want to give it a try without implementing anything, then you can perform these steps in 3D Slicer’s Segment Editor module (load models, import them to segmentation, use Logical operators effect, export resulting segment to model).
If you use Python, then another option is to use Boolean mesh operations of Blender, which is reported to be quite robust and very fast. You can import Blender Python package and process VTK meshes as described here.