My use case involves a unique AppendPolyData, which input is many PolyDataAlgorithm (connected with SetInputConnectionByNumber).
Inputs PolyDataAlgorithm are totally independent each-others, and can be quite long to compute.
My wish is to update each PolyDataAlgorithm in its own thread, and at the end; update the unique AppendPolyData with the updated outputs.
What is the good way to do this ?
I’ve tried to call vtkAlgorithm.Update() on each input in a different thread, but I don’t see much difference with letting VTK Update the whole thing on its own (I see no real speedup).
Maybe there is some locking mechanism that ruins the multi-thread approach ?
Maybe it can be done by leveraging internal VTK mechanisms ?
Did you check the CPUs usage? If they’re nearly full then you have an effective multithreaded processing. Notice that effective does not necessarily mean efficient. Maybe the underlying code is not designed with multithreading in mind and are full of mutexes and such ruining parallel processing like you said. Maybe there’s some detail you’re missing (you didn’t post the code). BTW, what are you using? OpenMP? MPI? I my projects I use C++'s own <thread> API for that. One way to do what you want with std::thread can be:
#include <thread>
(...)
////////////// Execute algorithm in a separate thread. ////////////////////////
void executeAlgorithmThread( vtkPolyDataAlgorithm* algorithm ){
algorithm->Update();
}
///////////////////////////////////////////////////////////////////////////////
(...)
// Init the the number of threads with the number of logical processors.
unsigned int nThreads = std::thread::hardware_concurrency();
//create and run the algorithm threads
std::thread threads[ nThreads ];
for( unsigned int iThread = 0; iThread < nThreads; ++iThread ){
threads[iThread] = std::thread( executeAlgorithmThread,
algorithms[ iThread ] ); //<--- this array contains pointers to vtkPolyDataAlgorithm objects.
}
//lock-wait for the threads to finish their execution contexts.
for( unsigned int iThread = 0; iThread < nThreads; ++iThread )
threads[ iThread ].join();
This is basically what I’ve done.
I’ve compiled VTK with std::thread as parallel framework.
As you said, I don’t see the CPU usage reaching 80/100% but instead it stays near 10-15% that is the sign of a sequential work on my machine.
Moreover, if I use some stopwatch to time the whole process, I can see hardly any difference between the parallel version, and the version in which I let VTK make its own cook.
This is for that very reason that I was asking if there was any mutex in Update() code that ruin parallelism.
Just compiling VTK with <thread> is not enough. Please, take a look at this: https://www.kitware.com/vtk-shared-memory-parallelism-tools-2021-updates/ and make sure you go through all the necessary steps to enable parallelism with the newer multithreading API (SMP tools). Also, I think you need to share more of what you’ve done to increase the probability of success. I mean:
a) What have you enabled in VTK configuration to enable multithreading?
b) What parts of your code are dealing with multithreating?
