A scale factor is one possible way to solve the issue, but natively macOS uses pixel conversion functions instead (see !2019, for example).
Hopefully, tk 8.6 provides an API for getting the scale factor. If not, it might be necessary to get the info natively from macOS.
By “help”, I mean that I can provide advice, and if you get something working (on both retina and non-retina displays of course!) then I can help to test it and get it into the VTK git repository and releases.
I’m fond of tkinter because it’s quicker than Qt and it’s also built-in to Python. But I switched to Qt a few years ago because, at the end of the day, Qt provides a more professional-looking app and has many more features.