vtkCellArray: Support offsets/connectivity as vtkDataArray | vtkUnstructuredGrid: Support CellTypes as vtkDataArray

A large Merge request is about to land in VTK master that changes several core parts of VTK quite a bit. Any feedback would be more than welcome.

vtkCellArray: Support offsets/connectivity as vtkDataArray including vtkAffineArray for Offsets

The vtkCellArray used to be able to support storing connectivity and offsets that are either vtkTypedInt32Array or
vtkTypedInt64Array. This was limiting because it did not allow using other vtkDataArray subclasses, therefore,
many times a copy to the supported types was needed.

A user can now store the connectivity and offsets as any vtkDataArray subclass using
vtkCellArray::SetData(vtkDataArray* offsets, vtkDataArray* conn). This is useful because it allows
storing any user given array without worrying whether VTK will deep copy the data arrays or not.

vtkCellArray used to have 2 storage types but now has the following 5 :

  1. Int64 (Old): Offsets vtkTypeInt64Array, Connectivity vtkTypeInt64Array
  2. Int32 (Old): Offsets vtkTypeInt32Array, Connectivity vtkTypeInt32Array
  3. FixedSizeInt32 (New): Offsets vtkAffineArray<vtkTypeInt32>, Connectivity vtkTypeInt32Array
  4. FixedSizeInt64 (New): Offsets vtkAffineArray<vtkTypeInt64>, Connectivity vtkTypeInt64Array
  5. Generic (New): Offsets vtkDataArray, Connectivity vtkDataArray

vtkAffineArray has become a first class citizen in vtkCellArray to support creating a cell array with cells
of constant size without needing to store the offsets explicitly, and therefore saving memory.

The following functions can be used to interact with the new storage types:

  1. bool IsStorage64Bit()
  2. bool IsStorage32Bit()
  3. bool IsStorageFixedSize64Bit()
  4. bool IsStorageFixedSize32Bit()
  5. bool IsStorageFixedSize()
  6. bool IsStorageGeneric()
  7. void Use64BitStorage()
  8. void Use32BitStorage()
  9. void UseFixedSize32BitStorage(int size)
  10. void UseFixedSize64BitStorage(int size)
  11. bool CanConvertTo64BitStorage()
  12. bool CanConvertTo32BitStorage()
  13. bool CanConvertToDefaultStorage()
  14. bool CanConvertToFixedSize32BitStorage()
  15. bool CanConvertToFixedSize64BitStorage()
  16. bool CanConvertToFixedSizeStorage()
  17. void ConvertTo64BitStorage()
  18. void ConvertTo32BitStorage()
  19. void ConvertToDefaultStorage()
  20. void ConvertToFixedSize64BitStorage()
  21. void ConvertToFixedSize32BitStorage()
  22. void ConvertToFixedSizeStorage()
  23. bool ConvertToStorageType(int type)
  24. vtkTypeInt64Array* GetOffsetsArray64()
  25. vtkTypeInt32Array* GetOffsetsArray32()
  26. vtkAffineArray<vtkTypeInt32>* GetOffsetsAffineArray32()
  27. vtkAffineArray<vtkTypeInt64>* GetOffsetsAffineArray64()
  28. vtkDataArray* GetOffsetsArray()
  29. vtkTypeInt64Array* GetConnectivityArray64()
  30. vtkTypeInt32Array* GetConnectivityArray32()
  31. vtkDataArray* GetConnectivityArray()
  32. int GetStorageType()

When the storage type is Int64, Int32, FixedSizeInt64 or FixedSizeInt32 then direct access to the array is
available through the Visit functor. When the storage type is Generic, then the Visit functor will provide the
arrays as vtkDataArray, therefore the user can no longer assume the arrays will be a subclass of
vtkAOSDataArrayTemplate. To solve this problem, the array access calls shown in the following Visit functor:

struct MyVisitor
{
  template <typename CellStateT>
  void operator()(CellStateT& state)
  {
    using ArrayType = typename CellStateT::ArrayType;
    using ValueType = typename CellStateT::ValueType;
    auto* conn = state.GetConnectivity();
    auto connRange = vtk::DataArrayValueRange<1>(conn);
    ValueType connValue = conn->GetValue(0);
    conn->SetValue(0, connValue);
    ValueType* connPtr = conn->GetPointer(0);
    conn->InsertNextValue(connValue);
    vtkIdType numCells = state.GetNumberOfCells();
    vtkIdType beginOffset = state.GetBeginOffset(0);
    vtkIdType endOffset = state.GetEndOffset(0);
    vtkIdType cellSize = state.GetCellSize(0);
    auto cellRange = state.GetCellRange(0);
  }
};
cells->Visit(MyVisitor{});

should be replaced with:

struct MyVisitor : public vtkCellArray::DispatchUtilities
{
  template <class OffsetsT, class ConnectivityT>
  void operator()(OffsetsT* offsets, ConnectivityT* conn)
  {
    using AccessorType = vtkDataArrayAccessor<ConnectivityT>;
    using ValueType = GetAPIType<ConnectivityT>; // The result of GetAPIType<OffsetsT> is the same
    auto connRange = GetRange(conn);             // or vtk::DataArrayValueRange<1, vtkIdType>(conn)
    ValueType connValue = connRange[0];
    connRange[0] = connValue;
    auto connPtr = connRange.begin();
    AccessorType accessor(conn);
    accessor.InsertNext(connValue);
    vtkIdType numCells = GetNumberOfCells(offsets);
    vtkIdType beginOffset = GetBeginOffset(offsets, 0);
    vtkIdType endOffset = GetEndOffset(offsets, 0);
    vtkIdType cellSize = GetCellSize(offsets, 0);
    auto cellRange = GetCellRange(offsets, conn, 0);
  }
};
cells->Dispatch(MyVisitor{});
// or also
if (!vtkArrayDispatch::Dispatch2ByArrayWithSameValueType<vtkCellArray::StorageOffsetsArrays,
      vtkCellArray::StorageConnectivityArrays>::Execute(cells->GetOffsetsArray(),
      cells->GetConnectivityArray(), MyVisitor{}))
{
  MyVisitor{}(cells->GetOffsetsArray(), cells->GetConnectivityArray());
}

Due to this new requirement, all Visit functors in VTK have been updated to use the new Dispatch API.

It should be noted that due to this change, offsets and connectivity arrays generated by a VTKm filter can now also
be stored in vtkCellArray as vtkmDataArray without needing to copy the data to vtkTypeInt32Array or
vtkTypeInt64Array.

Additionally, the majority of VTK filters that were generating constant size cells have been updated to use
FixedSizeInt32/64 storage type to save memory.

Finally, the following methods that are associated with the legacy format of vtkCellArray have been deprecated:

  1. vtkTypeBool Allocate(vtkIdType sz, vtkIdType vtkNotUsed(ext) = 1000)
  2. virtual void SetNumberOfCells(vtkIdType)
  3. vtkIdType EstimateSize(vtkIdType numCells, int maxPtsPerCell)
  4. vtkIdType GetSize()
  5. vtkIdType GetNumberOfConnectivityEntries()
  6. void GetCell(vtkIdType loc, vtkIdType& npts, const vtkIdType*& pts)
  7. void GetCell(vtkIdType loc, vtkIdList* pts)
  8. vtkIdType GetInsertLocation(int npts)
  9. vtkIdType GetTraversalLocation()
  10. vtkIdType GetTraversalLocation(vtkIdType npts)
  11. void SetTraversalLocation(vtkIdType loc)
  12. void ReverseCell(vtkIdType loc)
  13. void ReplaceCell(vtkIdType loc, int npts, const vtkIdType pts[])
  14. void SetCells(vtkIdType ncells, vtkIdTypeArray* cells)
  15. vtkIdTypeArray* GetData()

vtkDataSet: GetCellTypes updates

vtkDataSet’s method void GetCellTypes(vtkCellTypes* types), has been deprecated in favor of

void GetDistinctCellTypes(vtkCellTypes* types).

vtkCartesianGrid subclasses (vtkImageData, vtkRectilinearGrid) and vtkStructuredGrid’s method

vtkConstantArray<int>* GetCellTypesArray() has been deprecated in favor of vtkConstantUnsignedCharArray GetCellTypes()

vtkUnstructuredGrid’s method vtkUnsignedCharArray* GetCellTypesArray() has been deprecated in favor of

vtkDataArray* GetCellTypes(), which is templated to allow returning other arrays, such as

vtkUnsignedCharArray* GetCellTypes<vtkUnsignedCharArray>() or

vtkConstantUnsignedCharArray* GetCellTypes<vtkConstantUnsignedCharArray>().

This was done to enable storing the cell types of meshes with only 1 cell type as vtkConstantUnsignedCharArray instead

of vtkUnsignedCharArray, which saves memory.

It should be noted that due to this change, a cell types array generated by a VTKm filter can now also be stored in

vtkUnstructuredGrid as vtkmDataArray without needing to copy the data to vtkUnsignedCharArray.

Additionally, the majority of VTK filters that were generating single cell type cells have been updated to use vtkConstantUnsignedCharArray to save memory.

2 Likes

You posted this 2025-10-02 10:52 and merged the MR on 2025-10-03 18:06.

If you wanted feedback, maybe giving only 31 hours notice was too little?

Future MRs before the 9.6.0 release are always an option, so everything posted here can potentially change, if needed If you have any idea/comment feel free add it here!

I originally asked a question related to this here How to get full cell types array with new VTK 9.6 API, but it seems more appropriate to post here…

Can we keep GetCellTypes(vtkCellTypes* types)? GetDistinctCellTypes is not a suitable replacement since it is useful in many cases to keep the full cell types array (such as filter/extract cells by type).

E.g. downstream we are trying to make the pyvista API compatible with the latest VTK dev wheels, but this change (i.e. the deprecation of a full cell types array) has broken a lot of features in pyvista and it seems there is no suitable alternative (see Fix latest vtk incompatibilty by edabor · Pull Request #8003 · pyvista/pyvista · GitHub).

Or maybe there is another way to keep the full array with the new API that I missed?

I will take a look.

Fix is in https://gitlab.kitware.com/vtk/vtk/-/merge_requests/12549.

1 Like

Rather than opening a new topic, would like to find some clarification for the (possibly) related changes to vtkUnstructuredGrid::SetPolyhedralCells(), in older VTK versions the polyhedral cells were simply added into SetCells(), in which case the faces were a face stream and the faceLocations were the begin locations of the faceswith -1 for primitive cells (and everything just using vtkIndTypeArray).

In SetPolyhedralCells() the faces have now changed to be a vtkCellArray, which I infer to mean that the face stream is now properly split as connectivity/offsets internally (makes sense). But what do faceLocations now represent?

According to the docs:

  /**
   * Provide cell information to define the dataset.
   *
   * Cells like vtkPolyhedron require points plus a list of faces. To handle
   * vtkPolyhedron, use SetPolyhedralCells()
   * SetPolyhedralCells requires a faces vtkCellArray that will describe
   * the faces use by the polyhedral cells.
   * SetPolyhedralCells also requires a faceLocations vtkCellArray to fully describe a polyhedron
   * cell The faceLocations is a collection of face ids pointing to the faces vtkCellArray.
   */

From this description, there isn’t much of a clue. Are they begin/end offsets for each cell (like vtkhdf PolyhedronOffsets), or something else entirely??

EDIT: seems that digging through to vtkUnstructuredGrid.cxx Line 2050 is where the work is being done. Here the cellArray parameter is faces and the facesArray parameter is faceLocations with the outside caller having faceLocations empty for primitives. still digging…

I think that Polyhedral cells: storing faces as vtkCellArray which first discussed this change, should answer all your questions.

But a high level answer is this:

The vtkCellArray Faces includes:

  1. Connectivity: which defines the point ids of each global face in the mesh (like before duplicates are allowed, so that ieach polyhedron has correct face point orientation)
  2. Offsets: which define the offsets for each face in the connectivity so that we can get them with O(1) and also know how many points each face has rather than storing before the actual point ids.

The vtkCellArray FaceLocations includes:

  1. Connectivity: the global face ids of each cell (polyhedron)
  2. Offests: which define the offests to the global face ids of each cell. If the offset of one cell is the same as its next one, it means that it basically had 0 explicitly defined faces, which should be the case for all non-polyhedral cells.

Hi @spyridon97 - thanks for the quick followup. Interestingly enough I see that I was actually already involved in that initial discussion, but obviously lost track of what has happened in the years since.

So indeed the revised version means that the internals are almost identical to what we need to target for the VTKHDF output (makes sense). I was hoping to upgrade our code to avoid the deprecation warnings on SetCells(), but it since the change to vtkCellArray only came in very recently it looks like I’ll have to wait for a bit (not really easy to rewrite code that will works with either/both interfaces).

Thanks,

/mark

1 Like

You are welcome! Feel free to let me know if you need any help with anything.

P.S. you can convert you code to the new format and use vtkCellArray::ImportLegacyFormat to handle the old one.

This might be a plan, but we unfortunately still have a fair bit of nasty code for these routines. For this so-called “internal” VTK format we allocate the vtkIdTypeArray which could potentially be 32 or 64 bit, wrap them with our own type of span-view and then pass that into a backend algorithm to actually fill in the streams, connectivity, whatever. Will need to widen/extend that first.

Cheers,

/mark

Would it help to have a void ImportLegacyFormat/AppendLegacyFormat that takes either a 32 bit array or 64 bit array?

Assuming you have a strong stomach, this is what we are doing in the vtu adaptor :

  • our own routines to walk the mesh to be converted and determine the sizing. This makes sense, since we can easily navigate our mesh structures ourself.
  • allocate corresponding space as vtkSmartPointers, which makes sense since we want to hand things off to VTK in the end and thus place it in charge of memory handling
  • wrap the allocated space and then pass back to our own routines to fill out the values. Again this is the easiest, most consistent way.

The obvious problem with this is that the “filling things in” part uses general (non-vtk) integral types only and the logic for what should be filled in needs adjustment. If we change the adaptor code to use the newer connectivity/offsets for the faces, I suspect that this would fall apart the moment we try to compile with a somewhat older VTK (which probably doesn’t have the the importLegacyFormat bits).

Gotcha… i will leave this up to you then.

Yes you need new VTK for the new vtkCellARray.

1 Like