As part of the validation of the Software Of Unknown Provenance (SOUP) in our medical device we need to review all know issues to see if any pose a safety risk. Using a risk based approach, we filter out issues that, for example, are labeled as affecting a part of VTK that we don’t use. So thank you for all the labels that are already in use like “area:vtkm” or “lang:tcl”.
However I have noticed that there are quite a number of issues closed with a commit to the effect of “not an issue, ask on discourse: https://discourse.vtk.org/” . If those comments could also include a something like “~question” to apply a ‘question’ label that would be really helpful for filtering those issues out.
I’m not sure why closed issues should be considered valid with your system. Anyone with a gitlab account can open an issue. You should only considered triage:confirmed issues (but we should use it more often).
This is a very interesting topic. I was involved in development of a number of medical devices that used VTK, but we never reviewed issues, only commit history, and only in very rare situations (when we backported specific fixes or changes in selected files). Since VTK is open-source, in theory we could have analyzed impact of all changes, but due to the pace of VTK development (for example, last month 253k lines were added, 53k lines were deleted, in 1174 files), this would have been practically impossible.
Do you plan to review VTK changes to reduce the amount of verification and validation work (e.g., for small patches)? Or just to detect if there are changes that would justify updating your VTK version?
How often do you plan to update VTK?
Have you considered treating VTK as a black box, similarly to other software that is modified outside of your control and without having full insight into all the changes (similarly to the operating system, graphics drivers, firmware, etc. updates)?
I’m not sure why closed issues should be considered valid with your system.
We only consider issues that were closed after the release of the current VTK issue that we are using. This happens to be 8.2.0 at the moment.
You should only considered triage:confirmed
We considered this but it didn’t seem to be used reliably enough to ensure that we don’t miss real issues.
Do you plan to review VTK changes to reduce the amount of verification and validation work (e.g., for small patches)? Or just to detect if there are changes that would justify updating your VTK version?
The review of known issues is part of the V&V work. If we find an issue that presents more than a minor risk to the patient, we either upgrade our VTK version to or apply a targeted patch.
How often do you plan to update VTK?
Only as needed to fix important bugs, gain VTK functionality for new features we are developing, or for compatibility with other Off The Shelf (OTS) software upgrades.
Have you considered treating VTK as a black box, similarly to other software that is modified outside of your control and without having full insight into all the changes (similarly to the operating system, graphics drivers, firmware, etc. updates)?
Yes, all of these are OTS that we treat the same way (except for firmware which we only consider for a couple critical components)