Archiving Twitter with pywb

Pywb is a web archiving toolkit for playing back web archives. In late 2020, it was selected by the IIPC (International Internet Preservation Coalition) as the best software for web archive playback.¹ With pywb, you can play back and create web archives directly in your browser. It is a good tool for archiving dynamic websites that can be viewed without logging in. Twitter is an example of such a dynamic website where public accounts can be viewed without having an account.

• Python 3.7 to Python 3.11
• not fear of the command line

• runs on Windows, macOS, and Linux
• archives social media in the standard WARC format
• archiving happens directly in the browser
• same software for creating and playing web archives
• extensive documentation

• both software installation and crawl creation are done via the command line
• not automated
• cannot capture social media that requires logging in
• generates limited metadata, only the URL and the capture timestamp
• installation may not work smoothly on the latest Python version, requiring you to install and use an older Python version

Workflow

Step 1: install the software

Install pywb using pip, the Python package manager. Make sure Python is already installed on your computer.

• open a terminal window
• use the command pip install pywb to install pywb

Pywb is now installed.

Step 2: create a collection

Like Browsertrix and Webrecorder, pywb works with collections.

• create a folder for your pywb web archives with a name of your choice. In these manuals, we’ll call it pywb-folder.
• download this configuration file and place it in the pywb-folder. Check the pywb documentation if you want to customize pywb further.

• open a terminal and navigate to the pywb folder using cd path/to/pywb-folder. Replace path/to/pywb-folder with the correct path. If it's on your Desktop:
• for Windows: cd c:\Users\(username)\Desktop\pywb-map (replace (username) with your username)
• for macOS: cd ~/Desktop/pywb-map

• name your collection as you wish. In this manual, we call it my-archive. Use the command wb-manager init my-archive.

• Keep the terminal window open and proceed to the next step

Step 3: start pywb in record mode

Next, pywb needs to be started in record mode so you can begin browsing the social media platform.

• Go back to the terminal window from the previous step
• Start pywb using wayback --record --live --enable-auto-fetch. This enables live site recording. --enable-auto-fetch ensures pywb downloads additional content from responsive sites like individual tweets on the timeline. Embedded media in tweets are downloaded automatically, but to view full-size images in the archive, you must manually open them during archiving as Twitter hosts them on a different domain.

• If everything worked, you’ll see [INFO]: Starting Gevent Server on 8080 in the terminal
• Leave the terminal open

Step 4: navigate the Twitter account

• Go to http://localhost:8080/my-archive/record/https://twitter.com/twitter-user. Replace my-archive with your collection name and twitter-user with the account name.

• Scroll through the site (open all images if desired) until you’ve archived everything you want. As you interact with the page, you’ll see terminal messages indicating archiving activity.

• When done, you can close the browser or tab. You can also close the terminal window or stop the process with Ctrl+C.
• Check your pywb folder. In collections/my-archive/archive you should see a warc.gz file.

Result

You now have a web archive in WARC format that you can view.

• Open a terminal and return to the pywb folder: cd path/to/pywb-folder
• Start pywb with wayback -a. The -a option builds an index of the WARC files in the collection. Pywb uses this to search quickly through the archive.

• You’ll see messages in the terminal. When you see [INFO]: ...Done, your archive is ready for playback.
• Go to http://localhost:8080/my-archive (replace my-archive with your collection’s name).
• You’ll see a search box. Enter the archived URL.

• Click Search. A list of captures and their dates should appear. Normally, you’ll see one capture.

• Click on the crawl and scroll through the archived page.

References