How can I use ray cast volume rendering customizing my own shader?

Try //VTK::Shading::Impl. For a list of fragment shader replacements for this mapper, see Rendering/VolumeOpenGL2/shaders/raycasterfs.glsl.