Unfortunately, vtkBooleanOperationPolyDataFilter is very unreliable. It can give incorrect output for simple, completely valid input or can randomly crash. Classes in vtkbool work much better and its maintainer is usually willing to look into problems and fix them.