### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 18, 2023, 5:23pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/1 "2023-09-18T17:23:24Z")
Hi Developers
I believe that I have something cool that be of use for other developers using `VTK` for real-time temporal processing.
The changes consists of the following
Common/Transforms:
```auto
class VTK_EXPORT vtkLinearDataSetTransform : public vtkLinearTransform
{
vtkTypeMacro(vtkLinearDataSetTransform, vtkLinearTransform);
void PrintSelf(ostream& os, vtkIndent indent) override;
virtual void SetSource(vtkDataSet* polyData) = 0;
virtual void SetTarget(vtkDataSet* polyData) = 0;
protected:
vtkLinearDataSetTransform() = default;
~vtkLinearDataSetTransform() override = default;
private:
vtkLinearDataSetTransform(const vtkLinearDataSetTransform&) = delete;
void operator=(const vtkLinearDataSetTransform&) = delete;
};
```
Filters/Temporal:
```auto
class VTKSPSFILTERSTEMPORAL_EXPORT vtkContinuousPointRegistration : public vtkPolyDataAlgorithm
{
public:
static vtkContinuousPointRegistration* New();
vtkTypeMacro(vtkContinuousPointRegistration, vtkPolyDataAlgorithm)
void PrintSelf(ostream& os, vtkIndent indent) override;
//@{
/**
* Transform the output using @ref
* vtkTransformPolyDataFilter. Points, Normals and Vectors are
* transformed (default=ON).
*/
vtkSetMacro(ExplicitTransform, vtkTypeBool);
vtkGetMacro(ExplicitTransform, vtkTypeBool);
vtkBooleanMacro(ExplicitTransform, vtkTypeBool);
//@}
//@{
/**
* Use temporal registration. The module will continue execution
* until two successive data are registered (default=ON).
*/
vtkSetMacro(TemporalRegistration, vtkTypeBool);
vtkGetMacro(TemporalRegistration, vtkTypeBool);
vtkBooleanMacro(TemporalRegistration, vtkTypeBool);
//@}
///@{
/**
* Set/Get the algorithm used for registration.
*/
virtual void SetRegistrationAlgorithm(vtkLinearDataSetTransform*);
vtkGetObjectMacro(RegistrationAlgorithm, vtkLinearDataSetTransform);
///@}
///@{
/**
* Whether or not to deep copy the input. This can be useful if you
* want to create a copy of a data object. You can then disconnect
* this filter's input connections and it will act like a source.
* Defaults to OFF.
*/
vtkSetMacro(DeepCopyInput, vtkTypeBool);
vtkGetMacro(DeepCopyInput, vtkTypeBool);
vtkBooleanMacro(DeepCopyInput, vtkTypeBool);
///@}
virtual void SetLastTarget(vtkPolyData* lastTarget);
vtkGetObjectMacro(LastTarget, vtkPolyData);
protected:
vtkContinuousPointRegistration();
~vtkContinuousPointRegistration() override;
void ReleaseLastTarget();
// RequestDataObject - inherited
int RequestInformation(vtkInformation* request, vtkInformationVector** inputVector,
vtkInformationVector* outputVector);
int RequestUpdateTime(vtkInformation* request, vtkInformationVector** inputVector,
vtkInformationVector* outputVector);
int RequestUpdateTimeDependentInformation(vtkInformation* request,
vtkInformationVector** inputVector, vtkInformationVector* outputVector);
int RequestUpdateExtent(vtkInformation* request, vtkInformationVector** inputVector,
vtkInformationVector* outputVector);
int RequestData(vtkInformation* request, vtkInformationVector** inputVector,
vtkInformationVector* outputVector);
vtkLinearDataSetTransform* RegistrationAlgorithm;
vtkPolyData* LastTarget;
vtkTypeBool ExplicitTransform;
vtkTypeBool TemporalRegistration;
vtkTypeBool DeepCopyInput;
void CreateDefaultRegistration();
private:
vtkContinuousPointRegistration(const vtkContinuousPointRegistration&) = delete;
void operator=(const vtkContinuousPointRegistration&) = delete;
vtkTransform* UserTransform;
double* TimeSteps;
int NTimeSteps;
int TimeStepIndex;
double TimeStep;
};
```
It is continuous registration of surfaces that can be read from a `vtkDataSeriesReader` available in Paraview or any other class that expose requests for temporal data. I could make the continuous registration more general and simply use `vtkDataSet`.
I am working on registrations which uses normal vectors and converge in must fewer iterations than the vanilla `vtkIterativeClosestPointTransform`. Obviously, I should then create a `vtkLinearTransformPolyData` in `Common/Transforms` instead.
What do you think? My secret agenda is of course to have a professional review that I have done things correctly with a temporal pipeline.
Thanks in advance
Jens Munk
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [September 18, 2023, 6:33pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/2 "2023-09-18T18:33:15Z")
Hi Jens,
Thank you for sharing what you came up with for real-time (or in situ) temporal filters.
I don’t think it has landed in a release yet, but there is a new base algorithm class in VTK master that I believe lets you do what you intend to: `vtkTemporalAlgorithm` (in `Common/ExecutionModel`). It is a templated class. In your case, you would have the following inheritance:
```auto
class VTKSPSFILTERSTEMPORAL_EXPORT vtkContinuousPointRegistration : public vtkTemporalAlgorithm;
```
It allows you to run an unknown number of timesteps (or iterations) before calling `UpdateTimeStep(double)`. The regular pipeline in VTK enforces knowledge of the time extent before running the filter.
In order for the filter to work, you need to implement the methods `Initialize`, `Execute` and `Finalize`.
- `Initialize` clears any internal cache and prepare and allocates whatever needs allocated.
- `Execute` is called at each timestep (or iteration for your case). It assumes the internal state of the filter is usable (it has been initialized at some point in the past)
- `Finalize` converts the internal cache into a usable output.
In order to use the filter in `real-time` mode, you need to call `vtkAlgorithm::SetNoPriorTemporalAccessInformationKey()`. It will produce a valid output at each call to `UpdateTimeStep(double t)`. If you do not call this method, then the filter will behave like any other VTK filter.
Here is an example of how the pipeline would execute:
```auto
real-time mode: need to request each time step one by one.
You can stop whenever you want.
UpdateTimeStep(0)
Initialize(...)
Execute(...)
Finalize(...)
UpdateTimeStep(1)
Execute(...)
Finalize(...)
...
regular mode: Need knowledge of the number of iterations before running.
UpdateTimeStep(t)
Initialize(...)
Execute(...)
Execute(...)
...
Execute(...)
Finalize(...)
```
Would this algorithm base class help you? You can find 2 examples of filters using it in VTK: `vtkTemporalStatistics` and `vtkTemporalPathLineFilter`.
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 18, 2023, 7:02pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/3 "2023-09-18T19:02:30Z")
Thank you. This is exactly what I need. I will fetch the master and inherit this interface. Right now, I actually ended up implemented `vtkPassInputTypeAlgorithm` and implemented the `RequestUpdateTime` method. I ran into trouble when running in real-time and had to fake the `TimeRange` but I guess the new key that you reveal will solve my issues.
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [September 18, 2023, 7:03pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/4 "2023-09-18T19:03:56Z")
This is great!
If you run in any troubles or find unexpected behaviors, don’t hesitate to come back to this thread.
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 18, 2023, 7:10pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/5 "2023-09-18T19:10:58Z")
Could you elaborate a bit on the intent behind having both
`RequestUpdateTime` and `RequestUpdateTimeDependentInformation`. In my examples they are always called only once and after `RequestInformation`.
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [September 18, 2023, 7:14pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/6 "2023-09-18T19:14:33Z")
I think that you can ignore them right now. `RequestUpdateTime` propagates information downstream (originated from the source / reader going down the pipeline), as opposed to `RequestUpdateExtent` which propagates information upstream.
`RequestUpdateTimeDependentInformation` needs implemented when some information keys that get propagated upstream / downstream change over time. This doesn’t happen for a lot of filters.
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 18, 2023, 7:18pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/7 "2023-09-18T19:18:50Z")
Okay. Right now, I use `RequestUpdateTime` for propagating time information downstream, so at the end of my pipeline I can query an update for a given time. For real-time applications, I insert data into the upstream pipeline and afterwards query an output from downstream with a temporal cache inserted. This way, I can synchronize input with output (offset by one time step).
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [September 18, 2023, 7:19pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/8 "2023-09-18T19:19:16Z")
To clarify the latter: an example of `RequestUpdateTimeDependentInformation` could be a `vtkImageData` whose extent changes over time. I am talking about properties that are not supposed to move. Keys like UPDATE\_TIME do change over time but it is expected.
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 18, 2023, 7:20pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/9 "2023-09-18T19:20:11Z")
Makes perfect sense.
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [September 18, 2023, 7:20pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/10 "2023-09-18T19:20:28Z")
Sure, this should work. I haven’t used this feature a lot so if things don’t propagate as expected let me know, I’ll look into it.
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 18, 2023, 8:26pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/11 "2023-09-18T20:26:39Z")
I can see that I need to change the `vtkTemporalAlgorithm` for my purpose. Either I need to be able to access the previous time step and make a request to fetch the previous data object that I can perform registration to (this is what I do right now) or alternatively I could store the input internally (one dataset) and only return when I can register two consecutive datasets, but then I need to overload the MustContinue, which is unfortunately a private method. In my current implementation, I can do both without any problems. With some modification of `vtkTemporalAlgorithm`, I would be able to use it, but not in its current implementation. Perhaps, I should give it an attempt for making an implementation for which it is possible to overload `MustContinue`.
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [September 18, 2023, 8:35pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/12 "2023-09-18T20:35:18Z")
What about wrapping your filter inside another filter? The main filter doesn’t have to be temporal, and it can internally have a pipeline calling `UpdateTimeStep` until convergence, and then stop. You would create your subclass of `vtkTemporalAlgorithm` in the `.cxx`. You can cache the input in the temporal filter if you shallow copy it.
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [September 18, 2023, 8:46pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/13 "2023-09-18T20:46:23Z")
If it doesn’t satisfy your needs we can probably make `MustContinue` protected. I tend to try to hide the internals to give more flexibility when we want to change how a filter works. But if it is useful it can be changed. If we do that I would appreciate if you could share your code when it is completed, if it is okay with you, so I can add a regression test to ensure we don’t break this protected mechanics in the future.
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 18, 2023, 9:01pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/14 "2023-09-18T21:01:30Z")
Sure. Later this week, I will have a working solution up and running with a local version of `vtkTemporalAlgorithm`, a `vtkContinuousPointRegistration`, which uses the vanilla ICP, `vtkIterativeClosestPointTransform`. I will just create a branch of master and push it upstream. I believe the most elegant solution is not to expose the internal `TimeSteps` and in my filter to continue until I have a valid registration.
I have uploaded a simplified version of my code using `vtkPassInputTypeAlgorithm` to the branch `jens.munk.hansen/temporalRegistration`. Later this week, I will update it with a version inheriting from `vtkTemporalAlgorithm`.
---
### Author:  [@nicolas.vuaille](https://discourse.vtk.org/u/nicolas.vuaille)
#### Post date: [September 26, 2023, 11:57am UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/15 "2023-09-26T11:57:23Z")
@Timothee_Couble this may be of interest for LidarView ?
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [September 28, 2023, 5:17pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/16 "2023-09-28T17:17:42Z")
I am a little held up with other projects. Will be working with my own clone of `vtkTermporalAlgorithm` until I can fully say that I have a working solution. But it is really nice that the new pipeline supports keys for handling temporal algorithms.
---
### Author:  [@Yohann\_Bearzi](https://discourse.vtk.org/u/Yohann_Bearzi)
#### Post date: [October 9, 2023, 12:52pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/17 "2023-10-09T12:52:43Z")
`vtkForEeach` might be of interest for you? [Introduce `vtkForEach` filter](https://discourse.vtk.org/t/introduce-vtkforeach-filter/12448)
---
### Author:  [@Jens\_Munk\_Hansen](https://discourse.vtk.org/u/Jens_Munk_Hansen)
#### Post date: [October 9, 2023, 5:12pm UTC](https://discourse.vtk.org/t/temporal-filters-example-using-registration/12311/18 "2023-10-09T17:12:01Z")
I have more less done the same as the `vtkForEach` filter in a small Python prototype using `CONTINUE_EXECUTING()`. My goal is that if registration fails, I will try to register the data to a larger mesh computed further down the pipeline (this part will probably be operational in spring 24). Right now, I am focusing on the registration algorithms following to strategies. Using VTK locators or FLANN and doing the math using Eigen.
Some of the challenges I will run into with `vtkTemporalFilter` could be the following.
First Update(): The filter will continue executing and use data for timestep=0 and timestep=1 and I want it to return data for timestep=0 registerered to that of timestep=1.
Second Update(): I want to return data for timestep=1 even if it fails to register data for timestep=1 and timestep=2.
and so forth.
For this, I assume that that I will need to update some temporal information in the `vtkRequestUpdateExtent` or an update that goes from downstream to upstream.
Do you have any idea of when you will have some examples using the new `vtkTemporalAlgorithm`?