As I’m sure many have noticed, macOS CI testing for VTK has been rough lately. We’ve been in the middle of a transition from running directly on the hardware to instead using VMs to run the jobs. Now that VTK has merged this migration into master, the last hardware arm64 runners will be going away by next week. Please rebase onto master to get this update and to get macOS coverage back. I am updating release as well; hopefully that comes back green and it can be merged today or Monday.
isolation (we don’t have to coordinate absolute paths for all projects to not step on each others’ toes)
we now have stable paths for all machines, so they can leverage the shared compiler cache instead of each machine having their own
Xcode comes with the VM instead of maintaining a handful on each machine
We started looking at VMs because we’ve been having resource exhaustion issues for a while now. Basically, the window manager would exhaust the system of Mach ports which are used by userspace to communicate with the kernel. Once they exhausted, the machine was “dead”. ssh failed because the exec couldn’t get resources to do its thing, etc. It could show up on any fork or other syscall, so things just died at random places. AFAICT, the issue is probably a race condition in the window manager with VTK opening and closing windows rapidly (ParaView sometimes saw issues, but rarely comparatived the VTK testers). I observed leaks of around 2k/hour and the machine died around 30k-40k ports by the window manager. I’ve reported it to Apple, but it is in the black hole of their issue tracking if/until it gets fixed. We started seeing it back in 2017 or so with Buildbot, but GitLab-CI really made the pain acute.
We run CI jobs without admin privileges (even sudo), so there’s no way to have them reboot reliably from a CI job.