Saving Facebook accounts as structured text using Facebook-scraper

----

Facebook-scraper is opensourcesoftware to retrieve the content of public Facebook pages and groups into a structured text format. There’s an active community behind the software that regularly adds new features.

• Python 3.6 or higher: you can check if you have Python, and the correct version, via the command python -V;
• Chrome browser;
• a Facebook account.

• works on Windows, macOS and Linux;
• works for public and private Facebook pages, groups and individual accounts;
• doesn’t use APIs, meaning the software isn't dependent on their limitations;
• fast and automated;
• can also retrieve the content of comments, as well as the number of reactions (like a like) and the people who gave them;
• saves content in CSV format.

• only retrieves textual content, no media;
• difficult to capture an entire account;
• only works via the command line or via Python scripts;
• scraping a large amount of content can lead to a temporary IP ban.

Workflow

To make the commands understandable for people without experience with the command line, we work in this workflow always on the Desktop. If you have more experience, you can also use another location.

Step 1: install Facebook-scraper

• Open a terminal window, via for example Terminal, Command prompt or Windows Terminal
• Type in the window: pip3 install facebook-scraper (use pip install facebook-scraper if pip3 doesn't work)

Facebook-scraper is now installed.

Step 2: export cookies (optional)

Facebook-scraper can be used in combination with cookies. You can use them to download the data of private accounts, as well as the comments and reactions given on public posts. It’s also a good way to protect yourself from an IP ban.

There are several browser extensions that allow you to easily and without technical knowledge download cookies. In this guide we use Get cookies.txt, an extension for Google Chrome.

Install Get cookies.txt

• Open Google Chrome
• Go via this link to the extension page of EditThisCookie and click on Add to Chrome

• A window appears asking you to confirm that you want to add the extension. Click on Add Extension.
• Get cookies.txt is now installed. Then, fix the extension next to the address bar by clicking on the puzzle piece next to the address bar and clicking on the pin icon next to EditThisCookie.

Use Get cookies.txt

• Make sure you are logged out of Facebook and then log in with the Facebook account that you use to archive social media.

• Click on the cookie next to the address bar to view the cookies.

• Download the cookies by clicking on the Export button and save it as facebook_cookies.txt on your Desktop.

Step 3: download the content of a Facebook account into a text file

Facebook-scraper can now be used to store Facebook posts and associated metadata in a CSV file.

• Open a terminal window.
• Navigate to the Desktop:
• for Windows: cd c:\Users$username)\Desktop (replace (username) with your username)
• for macOS and Linux: cd ~/Desktop

Capturing Facebook accounts and pages

• Find the name of the account you want to archive. This is the text that appears behind facebook.com/ For the Facebook page of meemoo, this is meemoo.be.

• Type the command in the terminal window to save the data into a text file.
• facebook-scraper --encoding utf-8 naam_account (replace naam_account with the name of the account).
• if you downloaded cookies in Step 2, add the cookies to the command: facebook-scraper --encoding utf-8 --cookies facebook_cookies.txt naam_account (replace naam_account with the name of the account)
• to scrape the Facebook page of meemoo using cookies, this is: facebook-scraper --encoding utf-8 --cookies facebook_cookies.txt meemoo.be
• Once the command has been executed, a CSV will appear on your desktop with the filename: naam_account_posts.be

Capturing Facebook groups

With Facebook-scraper you can also capture posts in Facebook groups. For this purpose you use the --group option.

• Find the name of the group. This is often a number and can be found behind facebook.com/group/. For De geef van Gent this is 1029684767055401.

• Go back to the terminal window and adjust the command:
• facebook-scraper --cookies facebook_cookies.txt --encoding utf-8 --group naam_groep
• to archive the group De geef van Gent, use the following command: facebook-scraper --cookies facebook_cookies.txt --encoding utf-8 --group 1029684767055401

Capturing more posts

Facebook-scraper works with a number of pages from which the posts are captured. It is not clear what these pages stand for. By default, ten pages are requested and one page contains several posts.

The number of posts per page depends on the type of account:

• A Facebook page has four posts per page
• A Facebook group has 21 to 22 posts per page
• For a personal Facebook account, there are ten to twelve posts per page

If you want to capture more posts, you can adjust the number of pages you want to scrape in the command. Note! Specifying a large number of pages can lead to a temporary IP ban:.

• Go back to the terminal window and further adjust the command:
• facebook-scraper --pages aantal --cookies facebook_cookies.txt --encoding utf-8 naam_account
• to scrape 1000 pages (or 4000 posts) of meemoo, use the command as follows: facebook-scraper --pages 1000 --cookies facebook_cookies.txt --encoding utf-8 meemoo.be
• to scrape 1000 pages (or approximately 21000 posts) of De geef van Gent, the command becomes: facebook-scraper --pages 1000 --cookies facebook_cookies.txt --encoding utf-8 --group 1029684767055401

Capturing comments and reactions on each post

With Facebook-scraper you can also retrieve comments, reactions and the reactors (= the people who gave a reaction, incl. the type of reaction they gave) of each post. For this purpose you can extend the command with resp. the options --comments, --reactions and --reactors.

• Go back to the terminal window and further adjust the command:
• facebook-scraper --pages aantal --cookies facebook_cookies.txt --encoding utf-8 --comments --reactions --reactors naam_account
• to capture 4000 posts (= 1000 pages) with comments on the Facebook page of meemoo, use the command as follows: facebook-scraper --pages 1000 --cookies facebook_cookies.txt --encoding utf-8 --comments meemoo.be
• to capture approximately 21000 posts (= 1000 pages) with comments and reactors in the Facebook group De geef van Gent, the command becomes: facebook-scraper --pages 1000 --cookies facebook_cookies.txt --encoding utf-8 --comments --reactors --group 1029684767055401

Tests have shown that retrieving comments and reactions doesn't always work. The scripts under Advanced give better results so far.

Result

A CSV with e.g. the content of the posts, the publication date, the number of likes and comments.

Advanced

Facebook-scraper is not only a command line tool, but also a Python library that you can integrate into Python scripts. A library is a collection of code in a specific programming language that can be reused. This avoids the programmer having to reinvent the wheel. The Python library is more powerful than the command line interface and offers more possibilities for archiving social media accounts.

In the context of the project Best practices for the archiving of social media in Flanders and Brussels, some scripts were written to explore the possibilities of Facebook-scraper. These can be freely used. Note! These scripts are not further developed or maintained. Use at your own risk. You can adapt the scripts to your own needs.

Script 1: Archiving the info page of a Facebook page, group or account

The Python module offers the possibility to scrape the info- or about-page of different types of Facebook accounts ( example).

• Download the script facebook_info_scraping.py.
• Use the script as follows in a terminal window:
• python3 facebook_info_scraping.py --type account/group/page --cookies facebook_cookies.txt naam_account (choose the type of account you want to capture instead of account/group/page)
• to scrape the info page of meemoo, use the command python3 facebook_info_scraping.py --type page --cookies facebook_cookies.txt meemoo.be
• to scrape the info page of the group De geef van Gent, use the command python3 facebook_info_scraping.py --type group --cookies facebook_cookies.txt 1029684767055401

The result is a JSON file with the content of the info page.

Script 2: Archiving posts with comments and reactions from a Facebook page, group or account

It is also possible to write scripts that allow you to capture comments and reactions on posts. This script seems to give better results than the command line tool of Facebook-scraper itself.

• Download the script facebook_account_scraping.py
• Use the script as follows:
• python3 facebook_account_scraping.py --cookies facebook_cookies.txt --reactions --comments --pages aantal naam_account;
• to capture 4000 posts (= 1000 pages) with comments on the Facebook page of meemoo, use the command as follows: python3 facebook_account_scraping.py --cookies facebook_cookies.txt --reactions --comments --pages 1000 meemoo.be.

The result is a JSON Lines file with e.g. the text of a post, the comments and reactions.

Troubleshooting

Facebook-scraper gives an error: raise exceptions.InvalidCookies(f"Cookies are not valid")

This probably means that you logged out of Facebook after creating your cookies. As a result, the session has expired and the cookies are no longer valid. Re-export the cookies as in Step 2 and remain logged in as long as you want to scrape with these cookies.