Like most of the VTK image filters, this filter is multithreaded by default. It will automatically use the fastest threading method available. So I recommend to not call SetEnableSMP() or SetNumberOfThreads().
Perhaps vtkImageMapToColors will be faster, since its code is less complex. The usage is slightly different:
Also, it’s possible your code is calling this filter in an inefficient manner, for example if you are only displaying one slice, but you are calling ImageMapToColors on the entire volume, that will greatly reduce the speed.