Optimizing vtkOverlappingAMR: Reducing .vti File Count

Support
1

Hi VTK community,

I’m working with AMR data output from the RAMSES astrophysical simulation code. I’m using Python with VTK to convert this data into the vtkOverlappingAMR format for visualization.

Currently, each AMR cell ends up generating a separate .vti file, which results in thousands of tiny .vti files — one for every data point. This makes file I/O and visualization inefficient.

I’m trying to reduce the number of .vti files by:

  • Grouping nearby AMR cells into larger uniform grids wherever possible
  • Creating smaller grids only for leftover or isolated points
  • Still maintaining the level-based AMR hierarchy required by vtkOverlappingAMR

Has anyone tackled this kind of problem before? Any suggestions on:

  • Efficient spatial binning strategies that align with VTK’s AMR requirements
  • Possible existing tools or methods in VTK or elsewhere that could help optimize this process
  • Best practices for aggregating AMR blocks while preserving correctness

I’ve provided a simplified version of my current Python script that builds the vtkOverlappingAMR structure using Osyris + VTK. If you have any feedback on how I could restructure or optimize the code to reduce the number of .vti files, it would be very helpful.

I’m using:

  • VTK version: 9.4.2
  • Python version: 3.11.9
  • Platform: Linux

Any support, pointers, or example approaches would be greatly appreciated!

import vtk
import osyris
import numpy as np
import pandas as pd

# Load RAMSES AMR data using Osyris
data = osyris.RamsesDataset(1).load()
mesh_data = data["mesh"]

# Initialize arrays to store AMR cell information
x, y, z, length, spacing, density, level = [], [], [], [], [], [], []
corner_coords = []

# Loop through each cell to extract spatial and scalar info
for i in range(len(mesh_data["density"].values)):
    px = mesh_data["position"][i].x.values
    py = mesh_data["position"][i].y.values
    pz = mesh_data["position"][i].z.values

    l = float(mesh_data["dx"][i].values)
    lev = mesh_data["level"][i].values
    d = float(mesh_data["density"][i].values)

    space = [l, l, l]
    spacing.append(space)
    x.append(float(px))
    y.append(float(py))
    z.append(float(pz))
    length.append(l)
    density.append(d)
    level.append(lev)

    # Compute lower corner of the cell
    half_l = l / 2
    point1 = [float(px)-half_l, float(py)-half_l, float(pz)-half_l]
    corner_coords.append(point1)

# Organize cell data into a DataFrame
df = pd.DataFrame({
    "level": level,
    "corner_coords": corner_coords,
    "spacing": spacing
})

# Split DataFrame by AMR level
df_list = []
unique_levels = sorted(df["level"].unique())
for lev in unique_levels:
    df_level = df[df["level"] == lev].reset_index(drop=True)
    df_list.append(df_level)

# Initialize vtkOverlappingAMR with correct structure
block_values = [len(df_level) for df_level in df_list]
amr = vtk.vtkOverlappingAMR()
amr.Initialize(len(unique_levels), block_values)
amr.SetOrigin([0.0, 0.0, 0.0])

# Set level-wise spacing (assumes uniform spacing per level)
unique_spaces = [df[df["level"] == lev]["spacing"].iloc[0] for lev in unique_levels]
for i, space in enumerate(unique_spaces):
    amr.SetSpacing(i, space)

# Create grids for each AMR block
for i, df_level in enumerate(df_list):
    for j, row in df_level.iterrows():
        grid = vtk.vtkUniformGrid()
        grid.SetDimensions(11, 11, 11)
        grid.SetOrigin(row["corner_coords"])
        grid.SetSpacing(row["spacing"])

        # Calculate AMRBox dimensions (indices in global grid)
        var = row["corner_coords"]
        space = row["spacing"]
        i_min, j_min, k_min = int(var[0]/space[0]), int(var[1]/space[1]), int(var[2]/space[2])
        dx, dy, dz = 9, 9, 9  # Number of cells per dimension
        i_max, j_max, k_max = i_min + dx, j_min + dy, k_min + dz

        box = vtk.vtkAMRBox()
        box.SetDimensions([i_min, i_max, j_min, j_max, k_min, k_max])

        amr.SetAMRBox(i, j, box)
        amr.SetDataSet(i, j, grid)

# Write the AMR data to a .vthb file
writer = vtk.vtkXMLUniformGridAMRWriter()
writer.SetFileName("AMR_4.vthb")
writer.SetInputData(amr)
writer.Write()

Thanks.

2

@Francois_Mazen or @nicolas.vuaille do you have some inputs regarding this topic ?

3

Moving to VTKHDF file format would solve the I/O performance problem:

1 Like
4

And VTKHDF supports saving a single file instead of one file by image.

5

Thanks for your suggestion. Actually, I implemented this and found that overlapping AMR does not have HDF5 writer support. So, is it possible to keep intact the overlapping AMR structure, its levels, and all the scalar fields while making it time and space efficient?

6

Indeed currently the vtkHDFWriter doesn’t support the OverlappingAMR, which is why I didn’t consider that.

As Mathieu and François said, VTKHDF could be nice because we design it for such use case. For now, an alternative could be to write, in your python script with h5py, a valid overlapping amr following the file format documentation and this example : https://gitlab.kitware.com/keu-public/vtkhdf/vtkhdf-scripts/-/blob/main/scripts/generate_overlappingAMR.py?ref_type=heads

it will require more work for you and depending on the size of your data it could be slow of course but there is no other alternative for now until someone funds the support of overlapping amr in the vtkHDFWriter.

7

Hello,

I am providing the zip folder of the RAMSES output which I am using in the provided code as input. You can check the original structure which vtkOverlappingAMR is creating. I visualized this using ParaView.

Ramses Output