We have done some more research and testing in 3D Slicer and the conclusion is the following:
Why the problem occurs? If an application does not work correctly with a core profile then it indicates that still some old API or methods are used somewhere. Of course it may also mean that there are some other OpenGL handling mistakes that happen to not cause any problems when a compatibility profile is used. Qt is supposed to fully support core profiles since Qt-5.10, but it is a large project, so maybe there are still some issues. Overall, it is somewhat more likely that VTK has some OpenGL usage related bug, most probably related to initialization (as the application works well even with core profile, depending on what the main application window contains on application startup). We will not spend more time trying to pinpoint the issue, hopefully it will come up and fixed in another scenario (either in VTK or Qt).
What is the solution? We have decided to use compatibility profile for Windows, and core profile on Mac & Linux. Compatibility profiles are not fully supported on Mac, so core profile must be used there. We added an application setting option (not exposed on the application user interface) that allows forcing compatibility/core profile so that we can experiment with it and users can override it as needed.
Any regressions or risks of regressions? Using a compatibility profile on Windows does not seem to have any disadvantages. Performance may be even better than with a core profile.