If you read our earlier blog on the technology behind our preservation process, you already know what Archivematica does for your files and why we trust it for file processing. But we also want to highlight the engineering work it took to make that process hold up at Permanent’s scale, and why it matters for the long-term health of your archive.
As more people bring their files to Permanent, keeping every single upload fast and dependable gets harder, not easier. As more members joined Permanent and started uploading years of scanned photos, documents, and family recordings all at once, we needed our preservation process to keep up without slowing down for everyone else on the platform.
Archivematica At Scale
We may run one of the only high-volume, high-availability deployments of Archivematica anywhere, meaning ours has to handle a large number of files without ever going down or falling behind. CERN runs a similarly large deployment on Kubernetes, a system that lets you run software across many machines instead of one, splitting the work up and adding more capacity as needed, but theirs is an internal tool. If processing takes a week there, that’s fine, because they can just tell their internal users to wait. Permanent averages over 6,000 uploads a week, and every one of them needs to process fast enough to feel successful to the person who just hit upload.
Getting there meant two major changes. First, we optimized the SQL queries powering the Archivematica dashboard, essentially making the requests it sends to its database faster, so it doesn’t buckle under high volume. Second, we moved away from running Archivematica on a single server and adapted it to run on Kubernetes, breaking it into independent pieces that scale up or down based on demand.
Since moving to this new architecture, Archivematica has converted over 80% of uploads, even accounting for outages. And because file processing is no longer sitting in the same queue- the same waiting line other actions on the platform depend on- a large upload no longer freezes moving, copying, or deleting files for other members while it works through the backlog. You’ll also see your original file right away instead of waiting on a converted version, so the platform never feels stuck, even when a lot is happening behind the scenes.
The Sustainability Behind the Technology
Solving the scale problem matters, but scale alone isn’t the point. Permanent is a nonprofit, and our sustainability model rests on five pillars: our mission, our nonprofit status, an endowment, an open source technology stack, and the Byte4Byte program. Archivematica sits squarely inside that fourth pillar. Because it’s open source, it’s transparent, community-maintained, and improves with every institution that uses it.
Our engineers have contributed fixes upstream to Archivematica itself, starting with the database query optimizations that let the dashboard hold up under high volume. We’re also in the process of submitting fixes to how Archivematica checks for existing folders in Backblaze, one of the storage providers we use to back up your files, so it no longer slows down on directories with a lot of content. And one of our team members regularly joins the project’s developer meetings and has become a resource other organizations reach out to directly, including engineers at CERN.
What Comes Next
Archivematica is one part of a broader, ongoing program to make Permanent’s infrastructure more complete. A few things already underway:
- Preservation copy access: We are actively working to make preservation copies available for download; a next concrete step in our Archivematica roadmap
- Web archiving: In partnership with Webrecorder, an open-source web archiving toolkit with over 10 years of development, we are building support for preserving websites and web-based content; an area of growing importance as an estimated 38% of webpages from 2013 are no longer accessible
- EDTF date standards: We are completing implementation of the Extended Date/Time Format (EDTF), a Library of Congress standard for capturing dates with archival precision; supporting nuanced date expressions like date ranges, approximate dates, and partial dates that matter for historical records
What This Means for You
You don’t see any of this when you upload a file, and that’s intentional. The work described here all exists so your upload processes reliably no matter how many other members are uploading at the same time.
This isn’t a one-time fix. It’s infrastructure we keep maintaining and improving, because a preservation platform only does its job if it keeps working, year after year, without you having to think about it. That reliability is what lets you trust Permanent with the files you can’t afford to lose.
