Image fusion is a built-in Slicer feature, for example you can visualize PET/CT images like this:
Yes, for sure this is possible. However, in the mid/long term, you would miss a lot of opportunities to make your users happy.
For example, if you go with a simple implementation then your users will notice that tilted-gantry CT images, and images with variable slice spacing or orientation, and images with overview image stored with the same series instance UID as regular frames, etc. appear distorted in MPR views. Stock ITK/VTK DICOM implementation cannot load and display these images correctly. Or they may complain that certain images are not loaded at all, or they would like to see DICOM segmentation objects or RT structure sets. With decades of hard work we added all these features because they are necessary. Yes, they make the application bigger and more complex, but general-purpose 3D DICOM viewers cannot ignore these problems.
Also, there is a lot of work keeping all the underlying libraries up-to-date, make sure they all work on old and new hardware, operating system versions, in various software configurations. If you build your application on a platform then others do this work for you for free.
Slicer is not the best solution for everything, but if you want to develop a general-purpose, extensible, and customizable 3D medical image viewer in C++/Python then it is very hard to find anything better.