Looking at the CMakeLists.txt of VTK, it seems only the StandAlone and Rendering groups are turned on by default, so maybe it would be enough with something like
cmake \
-DVTK_GROUP_ENABLE_StandAlone=DONT_WANT \
-DVTK_GROUP_ENABLE_Rendering=DONT_WANT \
-DVTK_MODULE_ENABLE_<the_module_you_need>=ON
to get the minimal VTK you need.