Saving social media accounts as structured text with snscrape.

Snscrape is open-source software for retrieving content from social media in a structured text format.

Snscrape can scrape the following social media:

• Twitter:
• thread;
• search;
• user;
• hashtag;
• posts;
• profile;
• user.
• Telegram channel.
• Weibo user.
• VKontakte user.
• Instagram:
• user;
• hashtag;
• location.
• Reddit:
• user;
• subreddit;
• search.

Only the content of public posts can be retrieved.

• Python 3.8 or higher: you can check if you have Python, and the version, via the command python -V
• git: you can check if you have git, via the command git --version

• works on Windows, macOS and Linux;
• suitable for Twitter and Instagram;
• automated;
• doesn't use APIs, meaning the software isn’t dependent on their limitations.

• the output format JSON Lines is not a standard format, but can be converted to CSV via a script;
• only retrieves textual content, no media;
• since March 2021, Facebook blocks this tool, meaning it cannot be used for this platform;
• comments on posts are not retrieved.

Workflow

Step 1: install snscrape

Snscrape is installed via pip, the Python package manager.

• Open a terminal window (on Windows you can use Windows Terminal, Command Prompt or PowerShell).
• Type in the window: pip3 install snscrape.
• Upgrade snscrape to the latest version: pip3 install --upgrade git+https://github.com/JustAnotherArchivist/snscrape@master.

Snscrape is now properly installed.

Step 2: download the content into a text file

Snscrape saves all the text from a social media account in JSON Lines format.

• Find the name of your account. This is the text that appears behind facebook.com/, twitter.com/ and instagram.com/. For the Facebook and Twitter of meemoo, this is respectively meemoo.be and meemoo_be.

• Open a terminal window.
• Type in the terminal window the command to save the data into a text file. In the examples below we call this file output.jsonl, but you can give it a name of your choice.
• For a Facebook page: snscrape --jsonl --progress facebook-user name > output.jsonl (replace name with the name of the account).

• For a Twitter account: snscrape --jsonl --progress twitter-user name > output.jsonl (replace name with the name of the account).

• For an Instagram account: snscrape --jsonl --progress instagram-user name > output.jsonl (replace name with the name of the account).

• The progress will appear in the terminal.

Result

You have saved the complete textual content of the social media account into a JSON Lines file, including e.g. the URL, date and time of publication and content of the posts on the social media page.

The completeness of the text file can be checked by looking at the dates in the text file. The top line is the most recent, the bottom line the oldest message. Check whether the oldest message corresponds to the oldest message on the webpage.

Troubleshooting

snscrape gives an error when retrieving data from Facebook

In late March 2021, it was discovered that Facebook blocks snscrape ( issue #208). Until the developers behind snscrape fix this, this guide cannot be used for Facebook. You can try archiving Facebook using Browertrix instead. See the guide Archiving sociale media with Browsertrix(NL).

SyntaxError: invalid syntax or NameError: name `snscrape` is not defined on Windows

This error means that Windows cannot find the snscrape program on your computer. You can fix this by adjusting the command. Instead of snscrape [rest of command] use python -m snscrape [rest of command]

snscarpe: the term `snscrape` is not recognized as the name of a cmdlet, script file, or operable program in Windows

This error means that PowerShell cannot find the snscrape program on your computer. You can fix this by adjusting the command. Instead of snscrape [rest of command] use python -m snscrape [rest of command]