vtkHyperTreeGridEvaluateCoarse: data race in vtkThreadedTaskQueue produces short output array

I am running into an issue of non-deterministic output when using vtkHyperTreeGridEvaluateCourse

when I run my vtkhdf with a few hundred time steps, in about 1-5% of the times steps the resulting cell arrays have off by one array sizes after EvaluateCoarse, but which time steps have the off by 1 differ on repeat runs

I extracted 40 time steps more likely to trigger the issue into the vtkhdf file attached for testing.

from vtkmodules.vtkIOHDF import vtkHDFReader
from vtkmodules.vtkFiltersHyperTree import vtkHyperTreeGridEvaluateCoarse
from vtkmodules.vtkCommonExecutionModel import vtkStreamingDemandDrivenPipeline
from vtkmodules.vtkCommonCore import vtkSMPTools

vtkhdf = "evaluatecoarse_reproducer.vtkhdf"
num_runs =  5

print(f"File: {vtkhdf}")
print(f"SMP backend: {vtkSMPTools.GetBackend()}")
print(f"Runs: {num_runs}")
print()

for run in range(1, num_runs + 1):
    reader = vtkHDFReader()
    reader.SetFileName(vtkhdf)
    reader.Update()
    info = reader.GetOutputInformation(0)
    sddp = vtkStreamingDemandDrivenPipeline
    n_steps = info.Length(sddp.TIME_STEPS()) if info.Has(sddp.TIME_STEPS()) else 1

    mismatches = []
    for step in range(n_steps):
        reader.SetStep(step)
        reader.Update()
        htg = reader.GetOutput()

        ec = vtkHyperTreeGridEvaluateCoarse()
        ec.SetInputData(htg)
        ec.SetOperator(vtkHyperTreeGridEvaluateCoarse.OPERATOR_AVERAGE)
        ec.Update()
        out = ec.GetOutput()

        expected = out.GetNumberOfCells()
        cd = out.GetCellData()
        for i in range(cd.GetNumberOfArrays()):
            arr = cd.GetArray(i)
            if arr.GetNumberOfTuples() != expected:
                mismatches.append(
                    f"  step {step}: {arr.GetName()} "
                    f"{arr.GetNumberOfTuples()} vs {expected} "
                    f"(diff={expected - arr.GetNumberOfTuples()})"
                )
        del ec

    if mismatches:
        print(f"Run {run}: {len(mismatches)} mismatches")
        for m in mismatches:
            print(m)
    else:
        print(f"Run {run}: OK")

    del reader

Environment:

  • VTK 9.6.0 (pip wheel), SMP backend: Sequential
  • Python 3.13.9, Linux 4.18.0 (RHEL 8), x86_64
  • Source file: Filters/HyperTree/vtkHyperTreeGridEvaluateCoarse.cxx (v9.6.0)
    evaluatecoarse_reproducer.vtkhdf (6.5 MB)

Hello @omdaniel, thanks for the report, I can reproduce (unreliably) the issue

~/dev/ParaView-6.1.20260319-MPI-Linux-Python3.12-x86_64/bin/pvpython scripts/eval.py

File: evaluatecoarse_reproducer.vtkhdf
SMP backend: TBB
Runs: 5

Run 1: 3 mismatches
  step 36: scl 10937 vs 10940 (diff=3)
  step 36: texp 10937 vs 10940 (diff=3)
  step 36: var 10939 vs 10940 (diff=1)
Run 2: OK
Run 3: OK
Run 4: OK
Run 5: 3 mismatches
  step 13: scl 10919 vs 10920 (diff=1)
  step 13: texp 10919 vs 10920 (diff=1)
  step 13: var 10919 vs 10920 (diff=1)

Intuitively, this means that either the reader or the filter reads/write uninitialized or out-of-bounds memory somewhere, which is not a good sign.

This might be caused by https://gitlab.kitware.com/vtk/vtk/-/commit/4a93b9c5a6ae94c09b5b73b8cbe5db34cc0c8885 , can you try building VTK after reverting the commit and see if it still happens?

I will be on vacation for the next week, but after that I should be able to test that.

Same, keep me updated!

@omdaniel https://gitlab.kitware.com/vtk/vtk/-/merge_requests/13108 should fix the race issue and the bad performance

EDIT: Actually it’s not sufficient, MR needs more work