Another idea to have a compact storage would be to add a “Parent” 2d array that for each face will store at first the cell owner and at second the cell neighbor. For the cell owner the face will be correctly oriented while for the neighbor it will be needed to flip it.
If a Face is a boundary it will not have a neighbor (thus value will be -1 for neighbor).
This is like a FaceToCell relationship reverse to the current CellToFace storage.
This is done by solvers like openfoam to store polyMesh.