Thanks for raising this issue Mathieu, I’m sorry I didn’t see it before (I have yet to find how to make @mentions stand out when using discourse mailing list mode.)
Some of you may know, some don’t, but VTK have a few external dashboards that are run nightly. There used to be a lot, but now, only RogueResearch dashboard (managed by @seanm ) are in this section.
Last I checked @dgobbi also submits builds.
But it is indeed a shame that there is no longer a wide and rich variety that there used to be years ago. It would be nice to reinvigourate that IMHO.
This external dashboard process is not documented at all in develop.md
It predates that document of course. Perhaps no one ever thought to write up anything. I could add a few sentences…
The mail is way too spammy, and inform on all broken tests, not only new broken tests, increasing the chance of the mail being ignored (I know I do).
True, those emails could certainly be improved. That would require changes to cdash I guess. I made one proposal here suggesting that cdash use git blame on lines with errors/warnings.
Another solution of course is to fix those tests. 
Another “solution” is that I give up on them ever being fixed and just suppress any test currently failing. If we could get all the Rogue bots green, maybe they could be kept green.
External dashboard is considered not part of the CI of VTK but the responsability of the external buildbot manager.
Certainly part of the responsability falls to the external buildbot manager, and although I don’t have time for VTK everyday, I do think I’m pretty attentative/responsible for my bot submissions.
However, I do think the “you break it, you fix it” policy should also be applied. If one makes a code change to VTK, one must be prepared to do a little cleanup promptly after.
Integrate external dashboards in the CI process fully
I don’t know anything about gitlab CI, but maybe there is some way to better integrate my bots with gitlab? My bots are not dedicated to VTK though, they also build other projects, both internal and external, so they are only available for VTK between certain hours of the day (overnight really); during the day they are mostly busy building my own stuff.
Also, let’s not pretend that it’s only the Nightly submissions on cdash that are messy. The gitlib CI is never green either. I don’t think I’ve ever created an MR that doesn’t have some warning or test failure because of some issue that already exists in master. I’d favour a policy of “no merges unless green”.
Another “easy” thing that would help is to actually turn on some warning flags on the gitlab CI. Currently the gitlab CI uses only compiler default warnings, which is a very low bar.
Is there anything else RogueResearch is doing that I’m glossing over here?
Address Sanitizer, Thread Sanitizer, Rosetta, Universal Binaries, GuardMalloc, pre-release versions of macOS, pre-release versions of clang, various compiler flag permutations (LTO, -fstack-protector-all, -ftrapv, -ftrivial-auto-var-init=pattern, -std=c++11, -std=c++14, -std=c++17, -std=c++20).
Testing is an infinitely large problem space, and pre-merge testing is great and covers a lot, but there’s so much more, and there’s only time to do some of it overnight.