Temporal filters - example using registration

Development
,
1

Hi Developers

I believe that I have something cool that be of use for other developers using VTK for real-time temporal processing.

The changes consists of the following

Common/Transforms:

class VTK_EXPORT vtkLinearDataSetTransform : public vtkLinearTransform
{
  vtkTypeMacro(vtkLinearDataSetTransform, vtkLinearTransform);
  void PrintSelf(ostream& os, vtkIndent indent) override;

  virtual void SetSource(vtkDataSet* polyData) = 0;
  virtual void SetTarget(vtkDataSet* polyData) = 0;

protected:
  vtkLinearDataSetTransform() = default;
  ~vtkLinearDataSetTransform() override = default;

private:
  vtkLinearDataSetTransform(const vtkLinearDataSetTransform&) = delete;
  void operator=(const vtkLinearDataSetTransform&) = delete;
};

Filters/Temporal:

class VTKSPSFILTERSTEMPORAL_EXPORT vtkContinuousPointRegistration : public vtkPolyDataAlgorithm
{
public:
  static vtkContinuousPointRegistration* New();
  vtkTypeMacro(vtkContinuousPointRegistration, vtkPolyDataAlgorithm)
  void PrintSelf(ostream& os, vtkIndent indent) override;

  //@{
  /**
   * Transform the output using @ref
   * vtkTransformPolyDataFilter. Points, Normals and Vectors are
   * transformed (default=ON).
   */
  vtkSetMacro(ExplicitTransform, vtkTypeBool);
  vtkGetMacro(ExplicitTransform, vtkTypeBool);
  vtkBooleanMacro(ExplicitTransform, vtkTypeBool);
  //@}

  //@{
  /**
   * Use temporal registration. The module will continue execution
   * until two successive data are registered (default=ON).
   */
  vtkSetMacro(TemporalRegistration, vtkTypeBool);
  vtkGetMacro(TemporalRegistration, vtkTypeBool);
  vtkBooleanMacro(TemporalRegistration, vtkTypeBool);
  //@}

  ///@{
  /**
   * Set/Get the algorithm used for registration.
   */
  virtual void SetRegistrationAlgorithm(vtkLinearDataSetTransform*);
  vtkGetObjectMacro(RegistrationAlgorithm, vtkLinearDataSetTransform);
  ///@}

  ///@{
  /**
   * Whether or not to deep copy the input. This can be useful if you
   * want to create a copy of a data object. You can then disconnect
   * this filter's input connections and it will act like a source.
   * Defaults to OFF.
   */
  vtkSetMacro(DeepCopyInput, vtkTypeBool);
  vtkGetMacro(DeepCopyInput, vtkTypeBool);
  vtkBooleanMacro(DeepCopyInput, vtkTypeBool);
  ///@}

  virtual void SetLastTarget(vtkPolyData* lastTarget);
  vtkGetObjectMacro(LastTarget, vtkPolyData);

protected:
  vtkContinuousPointRegistration();
  ~vtkContinuousPointRegistration() override;

  void ReleaseLastTarget();

  // RequestDataObject - inherited

  int RequestInformation(vtkInformation* request, vtkInformationVector** inputVector,
    vtkInformationVector* outputVector);

  int RequestUpdateTime(vtkInformation* request, vtkInformationVector** inputVector,
    vtkInformationVector* outputVector);

  int RequestUpdateTimeDependentInformation(vtkInformation* request,
    vtkInformationVector** inputVector, vtkInformationVector* outputVector);

  int RequestUpdateExtent(vtkInformation* request, vtkInformationVector** inputVector,
    vtkInformationVector* outputVector);

  int RequestData(vtkInformation* request, vtkInformationVector** inputVector,
    vtkInformationVector* outputVector);

  vtkLinearDataSetTransform* RegistrationAlgorithm;
  vtkPolyData* LastTarget;
  vtkTypeBool ExplicitTransform;
  vtkTypeBool TemporalRegistration;
  vtkTypeBool DeepCopyInput;

  void CreateDefaultRegistration();

private:
  vtkContinuousPointRegistration(const vtkContinuousPointRegistration&) = delete;
  void operator=(const vtkContinuousPointRegistration&) = delete;
  vtkTransform* UserTransform;
  double* TimeSteps;
  int NTimeSteps;
  int TimeStepIndex;
  double TimeStep;
};

It is continuous registration of surfaces that can be read from a vtkDataSeriesReader available in Paraview or any other class that expose requests for temporal data. I could make the continuous registration more general and simply use vtkDataSet.

I am working on registrations which uses normal vectors and converge in must fewer iterations than the vanilla vtkIterativeClosestPointTransform. Obviously, I should then create a vtkLinearTransformPolyData in Common/Transforms instead.

What do you think? My secret agenda is of course to have a professional review that I have done things correctly with a temporal pipeline.

Thanks in advance
Jens Munk

2

Hi Jens,

Thank you for sharing what you came up with for real-time (or in situ) temporal filters.
I don’t think it has landed in a release yet, but there is a new base algorithm class in VTK master that I believe lets you do what you intend to: vtkTemporalAlgorithm (in Common/ExecutionModel). It is a templated class. In your case, you would have the following inheritance:

class VTKSPSFILTERSTEMPORAL_EXPORT vtkContinuousPointRegistration : public vtkTemporalAlgorithm<vtkPolyDataAlgorithm>;

It allows you to run an unknown number of timesteps (or iterations) before calling UpdateTimeStep(double). The regular pipeline in VTK enforces knowledge of the time extent before running the filter.

In order for the filter to work, you need to implement the methods Initialize, Execute and Finalize.

  • Initialize clears any internal cache and prepare and allocates whatever needs allocated.
  • Execute is called at each timestep (or iteration for your case). It assumes the internal state of the filter is usable (it has been initialized at some point in the past)
  • Finalize converts the internal cache into a usable output.

In order to use the filter in real-time mode, you need to call vtkAlgorithm::SetNoPriorTemporalAccessInformationKey(). It will produce a valid output at each call to UpdateTimeStep(double t). If you do not call this method, then the filter will behave like any other VTK filter.

Here is an example of how the pipeline would execute:

real-time mode: need to request each time step one by one.
You can stop whenever you want.
UpdateTimeStep(0)
  Initialize(...)
  Execute(...)
  Finalize(...)
UpdateTimeStep(1)
  Execute(...)
  Finalize(...)
...

regular mode: Need knowledge of the number of iterations before running.
UpdateTimeStep(t)
  Initialize(...)
  Execute(...)
  Execute(...)
  ...
  Execute(...)
  Finalize(...)

Would this algorithm base class help you? You can find 2 examples of filters using it in VTK: vtkTemporalStatistics and vtkTemporalPathLineFilter.

3

Thank you. This is exactly what I need. I will fetch the master and inherit this interface. Right now, I actually ended up implemented vtkPassInputTypeAlgorithm and implemented the RequestUpdateTime method. I ran into trouble when running in real-time and had to fake the TimeRange but I guess the new key that you reveal will solve my issues.

4

This is great!

If you run in any troubles or find unexpected behaviors, don’t hesitate to come back to this thread.

1 Like
5

Could you elaborate a bit on the intent behind having both
RequestUpdateTime and RequestUpdateTimeDependentInformation. In my examples they are always called only once and after RequestInformation.

6

I think that you can ignore them right now. RequestUpdateTime propagates information downstream (originated from the source / reader going down the pipeline), as opposed to RequestUpdateExtent which propagates information upstream.

RequestUpdateTimeDependentInformation needs implemented when some information keys that get propagated upstream / downstream change over time. This doesn’t happen for a lot of filters.

7

Okay. Right now, I use RequestUpdateTime for propagating time information downstream, so at the end of my pipeline I can query an update for a given time. For real-time applications, I insert data into the upstream pipeline and afterwards query an output from downstream with a temporal cache inserted. This way, I can synchronize input with output (offset by one time step).

8

To clarify the latter: an example of RequestUpdateTimeDependentInformation could be a vtkImageData whose extent changes over time. I am talking about properties that are not supposed to move. Keys like UPDATE_TIME do change over time but it is expected.

9

Makes perfect sense.

10

Sure, this should work. I haven’t used this feature a lot so if things don’t propagate as expected let me know, I’ll look into it.

1 Like
11

I can see that I need to change the vtkTemporalAlgorithm for my purpose. Either I need to be able to access the previous time step and make a request to fetch the previous data object that I can perform registration to (this is what I do right now) or alternatively I could store the input internally (one dataset) and only return when I can register two consecutive datasets, but then I need to overload the MustContinue, which is unfortunately a private method. In my current implementation, I can do both without any problems. With some modification of vtkTemporalAlgorithm, I would be able to use it, but not in its current implementation. Perhaps, I should give it an attempt for making an implementation for which it is possible to overload MustContinue.

12

What about wrapping your filter inside another filter? The main filter doesn’t have to be temporal, and it can internally have a pipeline calling UpdateTimeStep until convergence, and then stop. You would create your subclass of vtkTemporalAlgorithm in the .cxx. You can cache the input in the temporal filter if you shallow copy it.

13

If it doesn’t satisfy your needs we can probably make MustContinue protected. I tend to try to hide the internals to give more flexibility when we want to change how a filter works. But if it is useful it can be changed. If we do that I would appreciate if you could share your code when it is completed, if it is okay with you, so I can add a regression test to ensure we don’t break this protected mechanics in the future.

14

Sure. Later this week, I will have a working solution up and running with a local version of vtkTemporalAlgorithm, a vtkContinuousPointRegistration, which uses the vanilla ICP, vtkIterativeClosestPointTransform. I will just create a branch of master and push it upstream. I believe the most elegant solution is not to expose the internal TimeSteps and in my filter to continue until I have a valid registration.

I have uploaded a simplified version of my code using vtkPassInputTypeAlgorithm to the branch jens.munk.hansen/temporalRegistration. Later this week, I will update it with a version inheriting from vtkTemporalAlgorithm.

15

@Timothee_Couble this may be of interest for LidarView ?

1 Like
16

I am a little held up with other projects. Will be working with my own clone of vtkTermporalAlgorithm until I can fully say that I have a working solution. But it is really nice that the new pipeline supports keys for handling temporal algorithms.

17

vtkForEeach might be of interest for you? Introduce `vtkForEach` filter

18

I have more less done the same as the vtkForEach filter in a small Python prototype using CONTINUE_EXECUTING(). My goal is that if registration fails, I will try to register the data to a larger mesh computed further down the pipeline (this part will probably be operational in spring 24). Right now, I am focusing on the registration algorithms following to strategies. Using VTK locators or FLANN and doing the math using Eigen.

Some of the challenges I will run into with vtkTemporalFilter could be the following.

First Update(): The filter will continue executing and use data for timestep=0 and timestep=1 and I want it to return data for timestep=0 registerered to that of timestep=1.
Second Update(): I want to return data for timestep=1 even if it fails to register data for timestep=1 and timestep=2.

and so forth.

For this, I assume that that I will need to update some temporal information in the vtkRequestUpdateExtent or an update that goes from downstream to upstream.

Do you have any idea of when you will have some examples using the new vtkTemporalAlgorithm?