GitLab availability and preservation of VTK’s community history

banesullivan · September 15, 2026, 5:05pm

The current reliability of gitlab.kitware.com has me concerned for the project’s rich ~30 year history with countless community discussions and contributions.

VTK has become so foundational and impactful that its availability is now critical infrastructure for a broad ecosystem, which is precisely why the current reliability and accessibility of its primary project platform are so concerning.

Yesterday I tried to open a normal issue from an email notification. After more than 30 minutes, the page still would not load. Instead, I was repeatedly presented with the Anubis proof-of-work challenge shown in the attached screenshots. If this were a one-off occurrence, I’d think nothing of it, but this has happened to me what feels like every time I’ve tried to access the GitLab for the last 6 months.

The GitHub repository mirror is useful, but it is not a substitute for the GitLab project. The issues, merge requests, design discussions, reviews, and other project history are essential parts of the open-source ecosystem. The discussions, code reviews, explanation of decisions, etc form a just as vital part to the value of open source projects as the code itself. When the primary service is inaccessible, that entire body of knowledge is effectively unavailable.

This also raises broader concerns about resilience and continuity. What is the disaster-recovery plan for the GitLab instance and its associated metadata? Is Kitware planning to provide an independently accessible mirror or archival export of the issues, merge requests, and discussions? Not just the Git repository?

The VTK community should not have to treat basic access to a ~30 year project history as uncertain. There are well established solutions and cloud providers that make this a solved infrastructure problem (and I’m willing to bet far more cost effective than what I imagine your current resources are allocated to maintaining this).

The current situation gives users and downstream projects a very poor impression of the project’s openness and reliability.

Speaking from the PyVista community, our users require a dependable foundation for the ecosystem deeply embedded in their 3D workflows. If contributors cannot reliably access the upstream project then that puts everyone at risk.

Would Kitware please clarify:

What is causing the recurring availability and verification problems?
Does Kitware have a rationale for not going with GitLab’s cloud hosting?
What reliability and disaster-recovery guarantees exist for Kitware’s GitLab instance?
Is there a plan to mirror or archive the full issue, merge request, and discussion history (along with this discourse) somewhere broadly and reliably accessible?

This situation is not sustainable, and it needs to be treated as a serious community-infrastructure issue.

banesullivan · September 15, 2026, 5:11pm

I also want to note and emphasize that this line on the loading screen has me deeply concerned for the openness of the project.

Specifically, many endpoints now require a valid login.

I’m not sure what endpoints these are and I’m assuming they are things that put real strain on your infrastructure but gating access behind a login to privately hosted platform is not a good precedent to set for a project with such a rich open source history.

jaswantp · September 15, 2026, 5:16pm

It would be sad to see all the discussions in issues/merge requests be lost should any disaster of such magnitude occur. I hope someone with more knowledge than me can respond to your questions.

Matthew_Woehlke · September 15, 2026, 5:30pm

With the disclaimer that I am not speaking on Kitware’s behalf… I have to say that I find some of the questions raised here a little… odd. Do you ask the same questions about any project hosted on gitlab.com or github.com? If not, why not? Why would you expect those entities, which have limited commercial interest in many of the projects they host, to be “more sustainable” and less likely to suddenly decide it is no longer worthwhile for them to provide those services, as compared to an entity who has a significant business interest in supporting VTK?

Related, the answer to your second question is self-evident. Critical infrastructure you control is superior to critical infrastructure you don’t. Ironically, the gist of your complaint is that you (understandably) don’t like feeling that VTK’s infrastructure relies on a “third party”. If you feel that way, is it so strange that Kitware does too?

ben.boeckel · September 15, 2026, 5:32pm

Speaking for myself based on information I have and feel comfortable providing (e.g., explaining too much about mitigations can make them useless).

They are those that are HTML renderings of expensive git operations. All of these are readily obtainable with a local clone of the repository instead of the world asking us to do uncacheable computation on things oodles of times.

The robot has a record of things that are important here.

Largely, AI training scraping. Oodles of requests from single IPs for “unimportant” things.

We need internal hosting anyways for other reasons, but the features we needed when we migrated from Gitosis (gitolite’s predecessor) and Gerrit required functionality not available from gitlab.com hosting (e.g., custom instance-wide CI runners). AFAIK, those are still not possible. AFAIK, custom runners require Developer+ access to the target repo for CI to work for MRs from forks (or manual pipeline creation by a Developer+).

No plans at the moment; we have the event history for the main repositories going back years. This does miss comment edits because GitLab doesn’t send events for those.

berk.geveci · September 15, 2026, 5:40pm

Howdy,

I will answer these questions piecemeal because some require detailed information from our sysadmin team and I didn’t want to wait that long before I responded first.

Summary:

We have backup solutions that are aimed at preserving Gitlab history. Details to follow.
Due to the overwhelming demand arising from the use of AI, all open git sites are groaning under the pressure. gitlab.kitware is no exception. We have all experienced issues with Github also. Personally, I have had many CI and UI response issues with my native Github repos.
gitlab.kitware interacts with a very extensive test infrastructure. In fact, this is the main reason why we started hosting VTK on gitlab.kitware. The github infrastructure did not support such an infrastructure at the time. This testing infrastructure is also groaning under the weight of the enormous amount of testing we perform with each MR and leads to performance issues with gitlab itself.
Our sysadmin team, to which we should all express gratitude, are working hard to keep things functioning well. There is active work happening now to improve the hardware infrastructure.
We are actively considering moving some of the hosting functionality to Github. Such decisions naturally take longer for a project with a 30 year history so please be patient with us.

banesullivan · September 15, 2026, 6:18pm

@Matthew_Woehlke, I think this conflates separate issues. Of course github.com and gitlab.com can have outages. The point is not that large cloud hosted services are infallible… my point is that reliability, operational transparency, and independent preservation are ~solved engineering problems there (and they allocating immense resources to these challenges)

GitHub and GitLab publish public status histories, incident reports, and availability commitments. GitLab documents 24/7 infrastructure monitoring and a 99.9% availability commitment for covered SaaS services, including issues, merge requests, and Git operations (status, availability policy). GitHub likewise publishes incident histories and detailed availability reports (status, availability report).

No? The gist of my complaint is not that its a third party but that third party (Kitware today) has limited transparency or public commitments to ensuring the posterity of the project and is unintenionally making it more difficult for efforts to try to access and preserve the project by gating access.

GitHub and GitLab exist within a much larger preservation ecosystem:

Software Heritage crawls GitHub and GitLab repositories and preserves source code and development history
The Internet Archive provides independent web-page preservation (Wayback Machine))
GH Archive records public GitHub activity, including issues and comments.

As I understand it, Kitware has made any external effort to archive the project very difficult… though I am able to find some issues on the WayBack Machine.

None of this makes those cloud platforms perfect. But it does mean that failure is observable, incidents are communicated, and there are established paths for independent replication. That is materially different from relying on a single organization’s private GitLab instance while offering only a code mirror elsewhere.

What happens if something unthinkable happens like Kitware going out of business? Can the community be confident that the ledger and records are independently accessible? It’s hard to go on faith alone for something like that as there would be zero-to-limited business interest in ensuring the posterity of the project in such a case (though I believe wholeheartedly there would be a ragtag team of Kitware engineers who would stage a heist in the name of open source to ensure everything is preserved).

How does someone outside of Kitware talk to this robot or access this record?

My main complaint/question really lies here: where is this event history? is this backed up somewhere independently hosted or accessible?

Don’t take my complaints here as saying “Please move to GitHub!”… While I’d love to see VTK on GitHub simply to be able to dynamically link PyVista and VTK issues/pull requests, I’m not going to pretend it’s the most reliable or best solution here.

Matthew_Woehlke · September 15, 2026, 6:53pm

…and that is exactly my point. They aren’t. There is no more guarantee that those services will exist tomorrow than there is that Kitware will exist tomorrow. The difference is that Kitware and VTK have significant interests in each other’s continued existence. Even if they’re being paid for hosting, I don’t see GitLab/GitHub having the same level of motivation that Kitware has. The recent Iron Mountain incident is a reminder of the unreliability of cloud providers.

I don’t know offhand about public commitments; you may have a point there. I do know that Kitware has significant interest in VTK; more than any cloud provider you can name is likely to have.

You also mentioned other organizations seeking to preserve GitLab/GitHub repositories. Why do they not also seek to preserve kitware.gitlab? Considering the importance of CMake, especially, it seems like they should. I would guess (again, speaking personally and not on behalf of my employer) that Kitware is more than willing to assist with said preservation.

I don’t know the technical details of which endpoints are affected, but my impression is that they mostly aren’t ones that are critical for archiving. (As per Ben’s note, no legitimate archivist is going to pester the server to generate information that can be obtained from a local clone.) Also, I don’t see why a login requirement is an issue for a legitimate archivist. Again, I would guess that Kitware is willing to work with legitimate archivists to the extent it is even necessary. If you are aware of actual archival difficulties, it would be helpful to share those. Otherwise this feels like FUD.

Anyway, the underlying cause of the service problems is scrapers who don’t care how their actions impact the site’s integrity. Those are the entities who are the proper targets of your ire.

banesullivan · September 15, 2026, 7:02pm

Going to take a moment to pause here because I think I am failing to properly get my point across.

A lot of my concern lies on the assumption that gitlab.kitware.com is self-hosted, on-prem at Kitware HQ’s office? Is that correct or incorrect?