consider a specific float value as a NaN in a vtkColorTransferFunction

Patrick, I think vtkDiscretizableColorTransferFunction does what you need. It is a subclass of vtkColorTransferFunction, so you can use the same methods to build the valid colors intervals. I used it some time ago to work with discrete color tables, but I needed to add some ranges with different colors. Finally I had to turn some other value ranges completely off.
This is part of the code I used. Here, min and max are the minimum and maximum values from the data attribute you are going to render.

vtkSmartPointer table = vtkSmartPointer< vtkDiscretizableColorTransferFunction >::New();
vtkSmartPointer< vtkPiecewiseFunction > opacityFunction;
if (min > 0) { // A min value greater than 0 means that 0 is to be considered as invalid.
opacityFunction = vtkSmartPointer::New();
opacityFunction->AddSegment(0, 0, min, 0); // Range [0, min] will have zero opacity.
opacityFunction->AddSegment(min, 1, max, 1); // Range [min, max] will have full (1) opacity.
}
}
table->EnableOpacityMappingOn();
table->SetScalarOpacityFunction(opacityFunction);

In the final part of your post, you mention the need of transforming all unwanted values into Nan. If I have understood the question correctly, I don’t think it would be necessary using vtkDiscretizableColorTransferFunction.