Web Archiving Community
Last updated
Last updated
💬 Join us on our new ArchiveBox community chat server: https://Zulip.ArchiveBox.io
🔢 Just getting started and want to learn more about why Web Archiving is important? Check out this article: .
The internet archiving community is surprisingly far-reaching and almost universally friendly!
Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open source tool for your web archiving need, or just want to see where archivists hang out online, this is my attempt at an index of the entire web archiving community.
Community-maintained indexes of web archiving tools and groups by IIPC, COPTR, ArchiveTeam, Wikipedia, & the ASA.
Open source tools and projects in the internet archiving space.
Articles, posts, and blogs relevant to ArchiveBox and web archiving in general.
A collection of the most active internet archiving communities and initiatives.
Indexes of archiving institutions and software maintained by other people. If there's anything archivists love doing, it's making lists.
https://archive.is / https://archive.today
https://ghostarchive.org
https://perma.cc
https://arquivo.pt
https://www.pagefreezer.com
https://www.smarsh.com
https://www.stillio.com
https://archive.st
https://theoldnet.com/
https://timetravel.mementoweb.org/
https://freezepage.com/
https://webcitation.org/archive
https://archiveofourown.org/
https://megalodon.jp/
https://www.webarchive.org.uk/ukwa/
https://github.com/HelloZeroNet/ZeroNet (super cool project)
There are lots more projects listed here too: https://github.com/stars/pirate/lists/internet-archiving
Ones I haven't personally vetted:
Random helpful utilities for web archiving, WARC creation and replay, and more...
https://github.com/TheCakeIsNaOH/xbs-to-archivebox A utility to sync xBrowserSync bookmarks with ArchiveBox
https://github.com/vrtdev/save-page-state A Chrome extension for saving the state of a page in multiple formats
https://github.com/jsvine/waybackpack command-line tool that lets you download the entire Wayback Machine archive for a given URL
https://github.com/hartator/wayback-machine-downloader Download an entire website from the Internet Archive Wayback Machine.
https://github.com/Lifesgood123/prevent-link-rot Replace any broken URLs in some content with Wayback machine URL equivalents
https://en.archivarix.com download an archived page or entire site from the Wayback Machine
https://proofofexistence.com prove that a certain file existed at a given time using the blockchain
https://github.com/chfoo/warcat for merging, extracting, and verifying WARC files
https://github.com/mozilla/readability tool for extracting article contents and text
https://github.com/mholt/timeliner All your digital life on a single timeline, stored locally
https://github.com/wkhtmltopdf/wkhtmltopdf Webkit HTML to PDF archiver/saver
http://squidman.net/squidman/index.html
https://wordpress.org/plugins/broken-link-checker/
https://github.com/ArchiveTeam/wpull
http://freedup.org/
https://en.wikipedia.org/wiki/Furl
https://preservica.com/digital-archive-software-1/active-digital-preservation For-profit company offering a digital preservation software suite
https://github.com/karlicoss/grasp capture webpages from Firefox and Chrome into Org-mode documents
https://github.com/dgtlmoon/changedetection.io Change detection and monitoring of web page content changes
A collection of blog posts and articles about internet archiving, contact me / open an issue if you want to add a link here!
https://blog.archive.org
https://webrecorder.net/blog
https://netpreserveblog.wordpress.com
https://blog.conifer.rhizome.org/
https://ws-dl.blogspot.com
https://siarchives.si.edu/blog
https://parameters.ssrc.org
https://sr.ithaka.org/publications
https://ait.blog.archive.org
https://brewster.kahle.org
https://ianmilligan.ca
https://medium.com/@giovannidamiola
https://items.ssrc.org/parameters/on-the-importance-of-web-archiving/
https://theconversation.com/your-internet-data-is-rotting-115891
https://www.bbc.com/future/story/20190401-why-theres-so-little-left-of-the-early-internet
https://sr.ithaka.org/publications/the-state-of-digital-preservation-in-2018/
https://gizmodo.com/delete-never-the-digital-hoarders-who-collect-tumblrs-1832900423
https://siarchives.si.edu/blog/we-are-not-alone-progress-digital-preservation-community
https://www.gwern.net/Archiving-URLs
http://brewster.kahle.org/2015/08/11/locking-the-web-open-a-call-for-a-distributed-web-2/
https://lwn.net/Articles/766374/
https://en.wikipedia.org/wiki/List_of_Web_archiving_initiatives
https://medium.com/@giovannidamiola/making-the-internet-archives-full-text-search-faster-30fb11574ea9
https://xkcd.com/1909/
https://samsaffron.com/archive/2012/06/07/testing-3-million-hyperlinks-lessons-learned#comment-31366
https://www.gwern.net/docs/linkrot/2011-muflax-backup.pdf
https://thoughtstreams.io/higgins/permalinking-vs-transience/
http://ait.blog.archive.org/files/2014/04/archiveit_life_cycle_model.pdf
https://blog.archive.org/2016/05/26/web-archiving-with-national-libraries/
https://blog.archive.org/2014/10/28/building-libraries-together/
https://ianmilligan.ca/2018/03/27/ethics-and-the-archived-web-presentation-the-ethics-of-studying-geocities/
https://ianmilligan.ca/2018/05/22/new-article-if-these-crawls-could-talk-studying-and-documenting-web-archives-provenance/
https://ws-dl.blogspot.com/2019/02/2019-02-08-google-is-being-shuttered.html
If any of these links are dead, you can find an archived version on https://archive.sweeting.me or https://web.archive.org.
Beware: many of these may be outdated, as ArchiveBox has frequent updates and continual improvement.
"Install ArchiveBox on SaltBox.dev" https://docs.saltbox.dev/sandbox/apps/archivebox/#3-setup
"ArchiveBox is an open-source self-hosted web archiving system for the web and the desktop" https://medevel.com/archivebox/
"Install ArchiveBox on a One-Click Docker Application" https://www.vultr.com/docs/install-archivebox-on-a-oneclick-docker-application/
"ArchiveBox, una solución para crear nuestro propio Archive.org en miniatura y personalizado" https://www.genbeta.com/herramientas/archivebox-solucion-para-crear-nuestro-propio-archive-org-miniatura-personalizado
"网页存档的开源工具ArchiveBox，可以将网页文字、图片、媒体文件等都保存下来，供日后查看。基于Python的开源项目，可搭建私人的网络存档服务。" https://www.bilibili.com/s/video/BV1ib4y1X7SL
"Персональный интернет-архив без боли" https://habr.com/ru/company/vdsina/blog/550180/
"ArchiveBox, una solución para crear nuestro propio Archive.org en miniatura y personalizado" https://www.genbeta.com/herramientas/archivebox-solucion-para-crear-nuestro-propio-archive-org-miniatura-personalizado
"Preserve the Internet With ArchiveBox" https://www.cyberpunks.com/preserve-the-internet-with-archivebox/
"Сам себе архивариус. Изучаем возможности ArchiveBox" https://xakep.ru/2021/02/01/archivebox/
"使用存档盒制作自己的Internet存档" http://www.diglog.com/story/1045192.html
"How to Make Your Own Internet Archive With ArchiveBox" https://nixintel.info/osint-tools/make-your-own-internet-archive-with-archive-box/
"Mit ArchiveBox Webseiten auf der Festplatte archivieren" https://www.linux-community.de/ausgaben/linuxuser/2020/12/mit-archivebox-webseiten-auf-der-festplatte-archivieren/
"ArchiveBox：开源的WEB存档" https://zhen.bushini.de/14738.html / https://www.1fishsauce.com/?p=4206
"两个基于爬虫的项目: Kiwix & ArchiveBox" https://blog.csdn.net/JackLang/article/details/108328791
"如何创建自己的私人自托管即时阅读应用程序" https://www.pcpc.me/tech/self-hosted-read-later-app
"How to install ArchiveBox to preserve websites you care about" https://blog.sleeplessbeastie.eu/2019/06/19/how-to-install-archivebox-to-preserve-websites-you-care-about/
"How to remotely archive websites using ArchiveBox" https://blog.sleeplessbeastie.eu/2019/06/26/how-to-remotely-archive-websites-using-archivebox/
"How to Create Your Own Private Self-Hosted Read-It-Later App" https://www.makeuseof.com/tag/self-hosted-read-later-app/
"How to use CutyCapt inside ArchiveBox" https://blog.sleeplessbeastie.eu/2019/07/10/how-to-use-cutycapt-inside-archivebox/
"Automate ArchiveBox with Google Spreadsheet to Backup your internet" https://manfred.life/archivebox
"【デモ有♪】ConoHaのArchiveBoxアプリケーションを使ってみたよ" https://qiita.com/CloudRemix/items/691caf91efa3ef19a7ad
"WEB-ARCHIV TEIL 8: WALLABAG UND ARCHIVEBOX" http://webermartin.net/blog/web-archiv-teil-8-wallabag-und-archivebox/
https://metaxyntax.neocities.org/entries/7.html
Follow these technological and organizational archiving hubs for the latest archiving news.
Find your local archiving group in the list and see how you can contribute!
You can find more organizations and initiatives on these other lists:
(COPTR)
(IIPC)
(DataTogether)
(ArchiveTeam.org)
(Wikipedia.org)
(American Society of Archivists)
Bookmarking tool that provides an archiving service in their paid version, run by Mozilla
Bookmarking tool that provides archiving in a paid version, run by a single independent developer
Bookmarking tool with archiving in their paid version, run by a company est. 2011
Bookmarking alternative to Pocket/Pinboard (with no archiving)
/ Self-hostable web archiving server that can import via RSS
Self-hostable bookmark tagging, archiving, and sharing service
A paid Pocket/Pinboard alternative that includes article snippet and highlight saving
Another brookmarking/annotation service with archiving as a paid feature
The O.G. Wayback Machine provided publicly by the Internet Archive (Archive.org)
commercial Wayback Machine solution
The king of internet archiving crawlers, powers the Wayback Machine
chrome headless crawler + WARC archiver maintained by Archive.org
WARC proxy recording and playback utility
utilities for dealing with WARCs
An easy preconfigured web crawler designed for backing up websites
A pure python implementation of wget with WARC saving
develops a suite of open source tools, to capture websites and replay them at a later time as accurately as possible. Webrecorder also publishes the .
Fully integrated (self hostable) SaaS web archiving platform
Chrome extension for manual, interactive archiving of websites as you browse the web. Good for capturing high-fidelity complex interactions
Web archive viewer that runs entirely in the browser and doesn't require any server-hosted component to view WARC and WACZ files. Also available as a standalone electron app for local desktop use
Command-line crawling application that powers Browsertrix's core crawling features
aka Python Wayback, the open source toolkit forked from archive.org for self-hosting your own wayback machine among other web archiving tools
Create a WARC file out of a folder full of assets
fast streaming asynchronous WARC reader and writer
An open-source personal archiving server that uses pywb under the hood.
A distributed web archiving solution using pywb with IPFS for storage
tool that pushes urls into all the online archive services like Archive.is and Archive.org
Parse And Create Web ARChive (WARC) files with node.js
Web archiver GUI using Heritrix and OpenWayback
User-scriptable, archival crawler using Chrome
Electron app version of the original for creating and interacting with web archives
a Chrome extension for creating WARCs from any webpage
Archives Unleashed Toolkit for analyzing web archives (formerly WarcBase)
A Rails engine for finding and searching web archives
Open source project developing core Wayback Machine components
Large list of archiving projects and orgs
A Java library for reading and writing WARC files.
Google, Bing, DuckDuckGo, and other
+ + Webrecorder's archiving suite has the highest fidelity, and can flawlessly archive YouTube, X, Facebook, and other complex, JS-heavy SPAs
Web Extension / CLI util for Firefox and Chrome to save a web page as a single HTML file
a beautiful, user-friendly browser extension that archives all history with full-text search, annotation support, and more
a web/pdf/ebook annotation tool that also archives content
extremely similar to ArchiveBox, uses a Django backend + UI and provides auto-tagging and summary features with NLTK
very similar project that archives Firefox, Shaarli, or Delicious bookmarks and all linked media, generating a markdown/HTML index
Python-based self-hosted knowledge base embedded into your filesystem
a desktop application for bookmarking, annotating, and archiving articles offline
Link archival and curation web app, very similar to ArchiveBox
a fast crawler with archiving and asset extraction support
Create high-fidelity WARC/WACZ captures using a playwright browser, with support for signing, media extraction, PDFs, etc. ()
Simple bookmark manager + readability archiver built with Go (like a clone of Pocket)
A command-line tool to turn web pages into beautiful, readable PDF, EPUB, or HTML docs.
A self-hosted bookmark management tool that saves snapshots to archive.org
Self-hosted bookmark manager that is designed be to be minimal, fast, and easy to set up using Docker.
A self-hosted bookmark database with full-text page content search and limited archiving features
Bookmark manager and search tool with limited archiving features
Archiving tool that uses the Chrome debugger protocol to save each page as-loaded in the browser** (aka 22120 by c0fe or i5ik)
Personal web UI based knowledge-base with web clipping and note-taking
Django-based web archiving tool with a focus on collecting text-based content
Browser-independent bookmark manager CLI written in Python3 and SQLite3
A proxying archiver that downloads content from sites and can snapshot multiple versions of sites over time
"Perkeep lets you permanently keep your stuff, for life."
A commercial archiving solution that appears to be very similar to ArchiveBox
Simple self-hosted bookmark management + note-taking app with limited archiving features
web GUI for institutional long-term archiving of web and other content
distributed web crawler built on puppeteer with screenshots
old proxying recorder software similar to ArchiveBox
Super simple CLI utility to bookmark and archive webpages
collect, organize, cite, and share research (mainly for technical/scientific papers & citations)
Non-linear bookmark and note-taking tool with archiving support
Desktop + mobile app for knowledge-base-style info collection and notes (w/ optional plugin for archiving)
A paid web archiving / session recording tool design for OSINT
CLI tool for saving complete web pages as a single HTML file
Go package and CLI tool for saving web page as single HTML file
Social media archiver for Facebook, Instagram and VKontakte accounts.
Archiving in style like ArchiveBox, but with a chat.
https://github.com/karlicoss/promnesia A browser extension that into a hierarchical/graph structure with metadata
project that provides a pretty auto-updating index of your Pocket links (without archiving them)
Post by Christopher Su on his Pocket saving IFTTT recipe
Aggregators: , , , , , , , , ,
Blog Posts & Podcasts: , , , ,
Hacker News threads and comments: , , , ,
Reddit r/DataHoarder, r/SelfHosted, etc. posts and comments: , , , , , , , ,
Twitter: , , , ,
(USA)
(International)
, , (International)
The digital preservation group that works on formerly Webrecorder.io (USA)
(formerly known as Webrecorder.io) is a company led by Ilya Kreymer, that researches and develops web archiving tools, widely used by the community.
(Virginia, USA)
, , (International)
Non-profit working on content archival and long-term preservation (Europe)
& their (UK & Wales)
and (Canada)
(Canada)
(Canada)
(Washington D.C., USA)
(USA)
(USA)
(USA)
(UK)
& their (International)
(USA)
(USA)
(USA)
(USA)
(USA)
(USA)
(USA)
(UK & Ireland)
(UK & Europe)
(International)
, , , , (USA)
(UK & Ireland)
(Sweden)
(Spain)
(Brazil)
(Portugal)
(France)
(Italy)
(Australia)
(Ireland)
(Netherlands)
(China)
(old)
