If there’s one thing that’s guaranteed in the tech world it’s that nothing is guaranteed. From AOL, Netscape, Yahoo, and MySpace, every tech empire seems to eventually fall to ruin. One method to reliably maintain information or online experiences that are lost to whims of computer users and markets is to backup, preserve, and host that information on one’s own computers, and the Kiwix project aims to help make offline backups of favorite websites.
The open-source tool started as a way for users to make their own offline backups of Wikipedia but eventually expanded into having the ability to backup many other sites as well. Wiki-type sites are generally the easiest, but it can also create backups of other sites like StackExchange so that when the Internet goes down or the site vanishes from the world, the information is preserved for use. There are plenty of guides online for setting up Kiwix as well, with it running easily on most Linux systems with or without the help of Docker, as well as versions available for Windows, Android, and Apple platforms.
Although the Kiwix software hosts the website backups, creating the backups needs to be done with another tool called Zimit to create the .zim files Kiwix uses. The Kiwix organization also hosts pre-downloaded .zim files of common websites so each user is not independently crawling Wikipedia on their own, a task that could take months on limited consumer hardware. As for what hardware to ultimately host it on, we might recommend an armageddon-proof machine like this one.
gl & hf mirroring anything in the age of Anubis, Cloudflare, ReCaptcha and others.
As someone who runs a site: every single one of these is trivial to bypass.
My site went from ~300 daily users to ~1.8 million in a span of a few weeks. Then I introduced a captcha it immediately cut down the traffic by 99% from simple crawlers. With the captcha it first fallen back down. But the AI overlord decided that the content on the site is valuable so it directed more sophisticated crawlers my way which actually solved the captchas. Over time (few weeks) the more sophisticated bots started solving it, so it’s back around 2000. I’m fine with it scraping until it does not maxes out the CPU usage on the server.
Non interactive captchas (Annubis, Cloudflare’s 2 out of 3 methods, etc) only deter bots who don’t want to spend a lot of time calculating the proof of work token.
Interactive captchas (hCaptcha, ReCaptcha) are harder, but not impossible. There are AI (Actual Indians) who built an industry around solving captchas.
The only thing that seems to work reliably is fully custom captchas. The AI will not spend minutes/half an hour to bypass it. Yet.
The Kiwix icon reminds me of a famous cartoon about addiction. Is it intentional?
I had never seen the “Nuggets” video until you prompted me to seek it out. Thank you for sharing.
r/DataHoarder if you want to see the connection.
It’s probably most efficient to download the official dump rather than scrape the site.
Maybe a better use of time will be to define a better ‘robots.txt’ that will redirect scrapping to downloading yearly/monthly/daily archives
Modern scrapers are a problem in large part because they don’t do polite things like obey a robots.txt (if they did, they’d also keep the scrape rate to something reasonable and never get noticed). The people running these scrapers are 100% sociopaths and care not one whit for anything beyond ‘gimme, gimme’.
It’s also lack of intelligence and grace, maybe.
The US-based bots used to be very simple (not to say brute) in comparison to bots from, say, eastern Europe.
They downloaded whole websites over and over again instead of just downloading parts that have changed or were new.
There was a time when websites came on CD-ROMs.
Still have the intel one. They’d mail them out to you.
I used Kiwix to keep local copies of Wikipedia and Project Gutenberg for a while, but for a good long while they weren’t delivering updates in anything approximating a timely manner.
I haven’t done anything toward mirroring Wikipedia since, but I’ve cobbled together a script that mirrors Project Gutenberg’s main collection, generates EPUB and MOBI files, and makes them available through a simple (one might say Craigslist-esque :-) ) interface:
https://git.alfter.us/salfter/gutenberg-mirror-browser
For Wikipedia, you can download the entire database directly from Wikimedia. No third-party scraper needed. https://dumps.wikimedia.org/
I downloaded a pile of sites to host locally a while back, just in case the web goes down, so that I can use them for my kid’s home education. Between all of the mirrorable content out there, local AI, and DIY vector/graph databases you can achieve university level tuition support at home, off grid, and on very humble ex-corporate hardware off the secondhand market. The key is to get extra RAM (+32GB) and give the old machines new fast SSD storage, then you can host the likes of Qwen 3.8 locally, and Gemma 4, which when constrained by the RAG will give you pretty solid answers to anything in your reference materials. Most people don’t appreciate local AI’s real super power, the semantic search capabilities are like having your own human librarian, one that is a savant. Then there is what is possible if your AI is agentic and has root on it’s own Linux box to build tools to self improve. Trust me on this, if you are not doing the above you will get left behind, and consumerised by the big AI labs.
Very interesting, would love to know more about this. What all do you run/do with this? Any other functions/tasks/capabilities/etc. that it wields? Wondering what other potential could be had if I were to spin this up myself.
I’ve been taking a similar approach, except for general purposes, not for education. I’m not a “prepper”, but I don’t trust even the most reliable Internet resources to be there forever. I also don’t trust that some of the (breathtakingly amazing) local AI models which are currently available will be there forever.
So I equipped my Linux PC with one but strong GPU (still before the prices went mad) and tons of hard drive space in redundant zfs pool configuration, and I’m regularly downloading Wikipedia and similar resources + the best current local models for reasoning, coding, vision (OCR), speech recognition. Scripts run regularly from cron (e.g. Wikipedia refreshes their bulk downloads every 2 months or so, Project Gutenberg adds books daily).
If the entire Internet went down today, at least I would be able to a) easily translate to/from many languages; b) converse with a local model and have it help me reason/summarize/suggest steps based on locally collected materials (including personal ones); c) program, automate, fix config issues with the help of a coding model.
There are limits to how far you can go with this approach. Whether you want to download detailed OSM maps for your local area or for the entire world depends only on how many hundreds of terabytes of storage you can devote to it! Do you want to be as independent as to maintain the mirror of apt-get sources? Do you need all scientific articles on all topics? All books from Anna? Do you want to have a physical copy of the same computer in a remote location?
trouble is what web page storige ? blogs, web pagest are death
https://www.projectnomad.us/
Project Nomad would be something similar.
what websites are worth dow koading beside wikipedia and project gutenberg?
Reddit
Ok,
I’m just going to leave this right here:
https://www.httrack.com/
Been around for as long as i can remember