In case you didn’t hear — on October 22, 2025, the Internet Archive, who host the Wayback Machine at archive.org, celebrated a milestone: one trillion web pages archived, for posterity.
Founded in 1996 by Brewster Kahle the organization and its facilities grew through the late nineties; in 2001 access to their archive was greatly improved by the introduction of the Wayback Machine. From their own website on Oct 21 2009 they explained their mission and purpose:
Most societies place importance on preserving artifacts of their culture and heritage. Without such artifacts, civilization has no memory and no mechanism to learn from its successes and failures. Our culture now produces more and more artifacts in digital form. The Archive’s mission is to help preserve those artifacts and create an Internet library for researchers, historians, and scholars.
We were curious about the Internet Archive technology. Storing a copy (in fact two copies!) of the internet is no mean feat, so we did some digging to find out how it’s done. The best information available is in this article from 2016: 20,000 Hard Drives on a Mission. They keep two copies of every “item”, which are stored in Linux directories. In 2016 they had over 30 petabytes of content and were ingesting at a rate of 13 to 15 terabytes per day, web, and television being the most voluminous.
In 2016 they had around 20,000 individual disk drives, each housed in specialized computers called “datanodes”. The datanodes have 36 data drives plus two operating system drives per machine. Datanodes are organized into racks of 10 machines, having 360 data drives per rack. These racks are interconnected via high-speed Ethernet to form a storage cluster.
Even though content storage tripled over 2012 to 2016, the count of disk drives stayed about the same; this is because of disk drive technology improvements. Datanodes that were once populated with 36 individual 2 terabyte drives are today filled with 8 terabyte drives, moving single node capacity from 72 terabytes (64.8 T formatted) to 288 terabytes (259.2 T formatted) in the same physical space. The evolution of disk density did not happen in a single step, so there are populations of 2, 3, 4, and 8 T drives in the storage clusters.
We will leave you with the visual styling of Hackaday Beta in 2004, and what an early google.com or amazon.com looked like back in the day. Super big shout out to the Internet Archive, thanks for providing such an invaluable service to our community, and congratulations on this excellent achievement.
If there’s one online resource to donate to, it’s the Archive.
I really hope that one day we’ll see them get funding and manpower for people to go through and organize the general archive.
Its still useful and you can still find things but I feel there is a lot of duplicated content and thing that would benefit from being grouped together.
Use sourced metadata only gets you so far afterall because a user might forget, make mistakes, leave things blank, or might not even have the option for certain important elements.
If we had a team of people cleaning up the archive then there would be magic to be had with linking all sorts of items together.
I wonder about the physical security of the archive. If the San Andreas fault slips again in a BIG way, does the archive lose both copies? It’s fantastic that they have this data, but hard drives are fragile things. Most importantly, it’s almost impossible to store digital data for LONG periods of time. Just look at how much trouble it is to read that old tape from Unix version 4? We need to keep this data for thousands of years. I don’t think anybody’s figured out a good way to keep digital data safe and accessible for more than 20 years (which is equivalent to just a few seconds for an archaeologist).
I think right now the bigger issue the archive needs to tackle is all the different groups who try to strike down archives. From authors to politicians everyone goes after the archive and they lack the funding to properly fight back against these pests.
I’m not sure what politicians you’re referring to, but in the U.S., authors have specific rights under longstanding copyright law.
Copying… excuse me… “archiving”… the copyrighted contents of my website without securing my permission is one such violation. Now, as As their intent is benign and my site, in and of itself, is not a revenue-generator, to me this violation becomes a technicality not worth creating bad feelings over.
On the other hand, when the full text of books I’ve written show up in the archive–books still in print and still a source of income to me—every “free” download from the archive is literally money out of my pocket. You can be sure I will… and have… taken them to task for this, despite their purpose being… again… benign.
I like the internet archive, I use it, and I’ve donated to their operation. But authors still have rights.
tell that to the people running the so called AI’s
This is one of the occasions to talk about Dismaster, a very useful tool to search for vintage files : https://discmaster.textfiles.com/