Motivation: VTK arrays have names, component names, lookup tables and a whole
information object. What they don’t have is units. Is that temperature in
Kelvin or Celsius? Are those coordinates in meters or millimeters? Today the
answer lives in the head of whoever wrote the file (or in a UNITS_LABEL
string that hardly anything reads).
I have been working on a branch (https://gitlab.kitware.com/vtk/vtk/-/merge_requests/14000) that adds physical units to VTK arrays and
carries them through the pipeline. It is not ready for review yet but it is
far enough along that I would like feedback on the design before going
further. This is what it looks like from Python.
>>> from vtkmodules import vtkImagingCore as imaging_core
>>> from vtkmodules import vtkFiltersGeneral as filters_general
>>> w = imaging_core.vtkRTAnalyticSource()()
>>> w.points.data.SetUnit("mm")
>>> w.point_data["RTData"].SetUnit("degC")
>>> gradient = filters_general.vtkGradientFilter(result_array_name="dT")
>>> gradient.SetInputArrayToProcess(0, 0, 0, 0, "RTData")
>>> out = (w >> gradient).update().output
>>> out.point_data["dT"].GetUnit()
'K/mm'
>>> out.points.data.GetUnit()
'mm'
The gradient filter worked out that the derivative of a temperature with
respect to a length is a temperature difference over a length. A difference
of two degC values is in K, hence K/mm. Note that I didn’t do anything to
the output points. Their units came along by themselves. More on how below.
Storage
Each vtkDataArray now has a unit string per component:
array->SetUnit("m/s"); // all components
array->SetUnit(2, "deg/s"); // just one
array->GetUnit(1); // "m/s"
array->HasUnits();
array->HasUniformUnit();
array->CopyUnits(other);
array->ClearUnits();
A few things to note:
- Units are per component. Points are the obvious use case: x and y in
meters and z in seconds is a perfectly reasonable space-time dataset. So
are velocities with a mix of linear and angular components. - Units are plain strings in vtkDataArray. CommonCore does not depend on
any units library. Only the code that does unit math does. - They are on vtkDataArray, not vtkAbstractArray. A string array doesn’t
need a unit. - The existing UNITS_LABEL information key is still there and is not
connected to the new API. I am not sure what to do with it. See the
questions at the end.
The units library
For the actual math (parsing “kg/m^3”, knowing that Pa*m^3 is J, converting
degF to K) I vendored LLNL’s units library
( GitHub - llnl/units: A run-time C++ library for working with units of measurement and conversions between them and with string representations of units and measurements · GitHub ) under ThirdParty/units. It is BSD
licensed, has no dependencies and parses pretty much any unit string you can
throw at it. Its namespace is mangled to vtkunits so it won’t clash with
another copy in the same process. LLNL also ships Python bindings on PyPI
(units-llnl), and those carry their own copy of the library.
Side note for those who care about third party details: LLNL’s
UNITS_NAMESPACE option is actually broken in their latest release (a dozen
or so places hard-code units::). The fix is small and I plan to send it
upstream so we don’t have to carry a patch.
VTK code doesn’t talk to the library directly. It goes through
vtkUnitConversion:
>>> import numpy as np
>>> from vtkmodules.vtkCommonUnits import vtkUnitConversion
>>> from vtkmodules.util.numpy_support import numpy_to_vtk
>>> vtkUnitConversion.GetConversionFactor("mi", "km")
1.609344
>>> vtkUnitConversion.AreCompatible("degF", "K")
True
>>> t = numpy_to_vtk(np.array([0.0, 100.0]), deep=True)
>>> t.SetUnit("degC")
>>> vtkUnitConversion.Convert(t, "K")
True
>>> t.GetUnit(), t.to_numpy()
('K', array([273.15, 373.15]))
Note that the degC to K conversion includes the offset. The first version
of this branch got that wrong: it multiplied by convert(1, degC, K), which
happily turned 100 degC into 27415 K. Ugh.
One thing the library doesn’t do is remember how you wrote a unit. It keeps
a unit as a number times SI base units, so K/mm is 1000 K/m, and when you
ask it to print that you get kK/m (kilokelvin per meter). Correct, but not
what a human would write. A velocity gradient came out as Hz and a velocity
squared as Gy (gray, J/kg). So vtkUnitConversion builds derived units from
the symbols of its inputs instead: K over mm is K/mm, m/s over m is 1/s,
Pa times m^3 is Pa*m^3. It then checks that the library reads the result as
the right unit, and if not, uses the library’s spelling. So a unit may come
out ugly but never wrong.
Moving units around
Units are only useful if they survive the pipeline. There are three
mechanisms:
- Copying. DeepCopy, ShallowCopy, vtkFieldData::CopyStructure,
CopyAllocate/InterpolateAllocate and vtkDataSetAttributesFieldList all
copy units. This covers most filters without touching them: if a filter
creates its output arrays the usual way, the units come along. - Spatial units in the executive. Lots of filters create brand new points.
So, after RequestData(), vtkDemandDrivenPipeline copies the point units
of the input to output points that have none. That’s how the points in
the first example got their units. It only does this when all inputs
(every connection and every block) agree. Appending a dataset in m to
one in cm gives points with no unit, not points in m. - Filters that compute new quantities. These have to know what they are
doing. So far: vtkGradientFilter (gradient, divergence, vorticity,
Q-criterion), vtkIntegrateAttributes (value times length, area or
volume) and vtkArrayCalculator.
The calculator is the interesting one. It walks the bytecode of the
expression with a stack of units, the same way vtkFunctionParser walks it
with a stack of values. Coordinates take the units of the points.
>>> from vtkmodules import vtkFiltersCore as filters_core
>>> calc = filters_core.vtkArrayCalculator(function="x / RTData", result_array_name="ratio")
>>> calc.AddScalarArrayName("RTData")
>>> calc.AddCoordinateScalarVariable("x", 0)
>>> (w >> calc).update().output.point_data["ratio"].GetUnit()
'mm/degC'
When the calculator can’t tell, for example
an operand without a unit, p^x where x isn’t a constant, or L + F with L
in meters and F in feet, the result gets no unit. Never a wrong one. For the
last case it also warns, because the values are wrong too: the calculator
adds the raw numbers.
Files
Units are written and read by the XML formats (Unit0, Unit1, … attributes
on DataArray elements, SpatialUnit0-2 on ImageData), VTKHDF (the same names
as HDF5 attributes) and the legacy format (a UNITS entry in the array
METADATA block). The legacy one matters more than it seems: vtkCommunicator
uses it to send data objects between processes, so without it units
disappear the moment data crosses an MPI boundary.
Python
All the numpy-style array classes (AOS, SOA, constant, affine, strided,
indexed, composite, structured points and partitioned arrays) carry units
through arithmetic and ufuncs:
>>> T = out.point_data["RTData"]
>>> (T * 2).GetUnit(), (T ** 2).GetUnit(), (T - T).GetUnit()
('degC', 'degC^2', 'K')
>>> a = numpy_to_vtk(np.array([1.0, 2.0]), deep=True)
>>> a.SetUnit("m")
>>> b = numpy_to_vtk(np.array([1.0, 2.0]), deep=True)
>>> b.SetUnit("ft")
>>> c = a + b
>>> c.GetUnit(), c.to_numpy()
('m', array([1.3048, 2.6096]))
>>> s = numpy_to_vtk(np.array([1.0, 2.0]), deep=True)
>>> s.SetUnit("s")
>>> (a / s).GetUnit()
'm/s'
And when the units don’t match:
>>> import warnings
>>> with warnings.catch_warnings(record=True) as caught:
... warnings.simplefilter("always")
... bad = a + s
>>> bad.HasUnits()
False
>>> print(caught[0].message)
numpy.add of values in incompatible units 'm' and 's'; the result has no unit.
A few things to note:
a + bconverts b to the unit of a first. 1 m + 1 ft is 1.3048 m. The
same goes for comparisons,maximum,minimumetc.- A difference of two temperatures is in K, not degC (that’s the
T - T
above). Adding a K value to a degC one adds a difference (no offset). - Incompatible units give a UnitWarning and a result without a unit.
sinof an array in degrees converts it to radians first.- Operations that keep an array implicit still do. A constant array times 2
is still a constant array, now with a unit.
All of this lives in one small module that the array classes apply as
decorators to their ufunc and operator methods, so a new array class gets it
with a few lines.
Open questions
This is where I would like input:
- UNITS_LABEL. Should it go away, become an alias for the uniform unit, or
stay as an unrelated label? Some code out there reads and writes it. - The executive heuristic. Copying input point units to outputs is right for
most filters and wrong for the few that change coordinate systems
(geographic projections, cylindrical/spherical transforms, a calculator
writing coordinates in some other unit). Those filters will have to set
their own units, or we need a way for an algorithm to opt out. Thoughts? - Unit spelling. Strings are the interface between VTK and everybody else
(unyt, pint, yt etc.). Most strings mean the same thing everywhere, but
not all. In LLNL’s library, kt is a kilotonne (a knot elsewhere) and yr is
365 days (365.25 in unyt). Also, since derived units keep the symbols of
their inputs, the same unit can come out spelled differently depending on
where it came from (Pa*m^3 from one filter, J from a reader). Is that OK
or should VTK pick a standard form for some of them? - What’s missing. Image data spatial units in VTKHDF, the legacy format and
.pvti files. vtkIntegrateAttributes in parallel when rank 0 has no data.
The old dataset_adapter module (no units there and I don’t plan to add
them). More filters.
I hope that others will chime in, especially those of you who have been
writing UNITS_LABEL or your own per-array unit metadata in readers.