See GitLab availability and preservation of VTK’s community history
The full vm on which gitlab.kitware.com is backed up every 12 hours to our local storage in the server room. These are backed up to a raided storage array with immutable storage. We currently have 2 backups per day going back 10 days. Plus we keep an offsite/offline copy of gitlab.kitware.com which gets updated monthly.
Is there a plan to mirror or archive the full issue, merge request, and discussion history (along with this discourse) somewhere broadly and reliably accessible?
Not currently. Any suggestions? Is this done for cloud hosted projects? Or does people rely on the backup strategy of the cloud host? Or is the main issue the “broadly and reliably accessible” part?
Yesterday I tried to open a normal issue from an email notification. After more than 30 minutes, the page still would not load. Instead, I was repeatedly presented with the Anubis proof-of-work challenge shown in the attached screenshots. If this were a one-off occurrence, I’d think nothing of it, but this has happened to me what feels like every time I’ve tried to access the GitLab for the last 6 months.
Working on getting answer to this from our sysadmin folks.
I got an answer from our sysadmin team. It is a long one so I will give a short summary here. Bottom line is that our infrastructure continues to evolve to fend off denial of service attacks. As an example, cmake.org was recently subjected to a massive DoS attack which ended up bringing down almost our entire server infrastructure. This was over the last few days and we changed how we handle Anubis as a result. So it is an evolving situation and let’s have patience to see if it settles.
As I mentioned before, we always have the option of moving to github while maintaining our current testing infrastructure (which was not available when we moved to gitlab).
Please keep reporting issues with the infrastructure here rather than just being frustrated and making voodoo dolls to curse the Kitware team. We have the same goals so let’s work together towards a solution.
@berk.geveci, thank you for forking the thread over here… that other thread went south quick and I’m regretful my proposal, attitude, or phrasing triggered that… My intention in all of this is to ensure there is a “backup” in place and that we as a community have a plan for any sort of disastrous scenario. My goal is not to see VTK move to GitHub (though I am confused why gitlab.com isn’t getting nearly as much attention in that other thread…)
Indeed this is my primary concern. It makes me incredibly nervous that all of this rich context, history, and community contributions are self-hosted and backed up all in ~one server room… Your note that it was almost entirely brought down by one DoS attack only adds to my nervousness. From the outside here it feels like the posterity of the project is at the whims of bad voodoo or a fault in the HVAC/plumbing systems.
My hot take here is that Kitware isn’t in the business of building data centers and cloud hosting… With millions of people depending on VTK, this doesn’t feel appropriate to self-host.
Someone brought up “the Iron Mountain incident” in the other thread as an example of how cloud providers are also vulnerable… My point here is that we’re all vulnerable. However, there are bigger initiatives and systems of redundancy built into these cloud providers that we can lean on and benefit from with the added benefit of reducing the strain Kitware’s resources.
I’ll explore some backup options and what other open source communities do to ensure posterity and reliable access. Would you please try to get a rough estimate of the size of these backups? I’m assuming its on the order of 1-100 TB but I’d like to make sure we understand the scale of this first.
That’s dangerous dark magic, won’t catch me doing that! I’m only sending good vibes and sunshine from San Diego to you New Yorkers!
…except that isn’t the case. We have backup and off-site backup. It’s third-party backup that’s missing (and isn’t provided by “the cloud”, anyway), so looking into that is potentially worthwhile. Mind, I would really like to see that backup be all of (public) gitlab.kitware, not just VTK. (In particular, I would like to see at least CMake included.)
No need to be nervous. The VTK source code history would be very difficult to destroy completely, as full VTK git repositories are cloned to tens of thousands of users’ computers. Even if entire continents are taken out, VTK would survive. Losing all the issues and pending merge requests and having to reconfigure CI from scratch would be unpleasant but nothing catastrophic.
Thanks for clarifying. That off-site backup either didn’t land with me or it took a while for us to establish that (which is part of the problem I’m outlining around the public committments and transparency).
What I’m looking to see here is whether we can establish an “off-site” backup is actually accessible to the people who’ve contributed to VTK who aren’t employed by or under contract with Kitware.