I believe that you have found a bug in vtkOBBTree, it is not checking the trivial rejection case. The code is also in need of performance and other clean up.
Having said that, if your intent is to determine whether one polygon intersects with another, there are several other ways to do this which would be much faster. Probably the easiest is to use a combination of a cell locator (e.g., vtkCellLocator or vtkStaticCellLocator), bounding box (e.g., vtkBoundingBox), and polygon intersection code (vtkPolygon::IntersectPolygonWithPolygon()). That is, you want to use the cell locator to find candidate intersection polygons quickly (e.g., FindCellsWithinBounds()).