DEV Community

Cover image for Python Facebook Posts Scraper with Requests and BeautifulSoup4
Hugo Sandoval
Hugo Sandoval

Posted on • Updated on

Python Facebook Posts Scraper with Requests and BeautifulSoup4

A few weeks ago I wrote about the Importance of using User Agents when we scrap data, and my examples shows the response from Twitter when we used the correct User Agent. This time I want to do the same but Facebook. We gonna scrape the posts in users profiles, Facebook pages and groups.

What we gonna get?

A list of items with the next values:

params description
published Formatted datetime of published
description Post text content
images List of images in posts
post_url The unique post url
external_links External links found in description
like_url The Like url

Now I have a developer blog, continue reading this post here: https://hhsm95.dev/blog/python-facebook-posts-scraper-with-requests-and-beautifulsoup4/

Thank you!

Top comments (17)

Collapse
 
jazzman16 profile image
jazzman16

Hugo, this is great. When I run this, as is, I'm only getting about 6 posts. Is this normal? Also, how would you recommend I edit to search for group posts with a specific tag_id? I tried to rewrite a few lines to specify my url with "groups/{group_id}/post_tags/?post_tag_id={tag_ID}" but I was only ever brought to the group's standard timeline.

Collapse
 
nassera2014 profile image
nassera2014

Hi, i had the same problem, when i run, i'm only getting 6 posts ! you resolved this ?
Thank you.

Collapse
 
sis4insa profile image
sis4insa • Edited

Hello Every one I need for research purpose 100 users already posted text.Per user i need from 120 to 200 posted text.But this python code download only 5 days post.Can I use this code for this purpose pls help me I kill 2 month for this purpose.I need to download a specific users profile post text all

Collapse
 
sis4insa profile image
sis4insa

please any one help me to this problem

Collapse
 
menbiamel profile image
menbiamel • Edited

hello Hugo thanks for sharing this .am working on scraping facebook data but the facebook blocked my account several times . please make a video to help as to see how it works.
how many posts does it extract ?

Collapse
 
hhsm95 profile image
Hugo Sandoval

The key to avoid being blocked is scrap slowly (about 3-5 between profiles) and use an User Agent like above (an old phone) that don't requires javascript so the detection is harder for Facebook.

Collapse
 
Sloan, the sloth mascot
Comment deleted
Collapse
 
hhsm95 profile image
Hugo Sandoval

Hi, I'm not sure if I can post an email here, but ask me any issues in the Github project.
github.com/adeoy/FacebookPostsScraper

Collapse
 
ds123 profile image
data science

getting error TypeError: 'module' object is not callable while running the code. What may be the problem ? Thanks in advance

Collapse
 
rakshit888 profile image
rakshit888

same error

Collapse
 
moukouba profile image
Moukouba

Hi!

Could you please advise on how to build a scraper that extracts comments on a specific subject on Facebook?
Thanks for assisting.

Collapse
 
hhsm95 profile image
Hugo Sandoval

Hi, the key to scraping on Facebook is the use of a User Agent from an old phone, as in this case use a Nokia C3. To extract the comments from the posts, it is necessary to enter the post and locate the CSS selectors that identify the comments, it is possible that I update the post adding this functionality.

Collapse
 
nassera2014 profile image
nassera2014

Hi, Thank you for sharing this, that's works well, but when i run the program, i get just 6 posts from a facebook page? how can i get more posts ?
Thank you.

Collapse
 
chalst1988 profile image
Surya Teja Challapalli • Edited

Hi @nassera2014

In order to get more number of posts I suggest to use Facebook_Scrapper Library where one can configure get_posts() with number of pages and posts per page to be scrapped. And by the way you could get all granular level info of reactions, like, comments, post_id, post date, etc.,

Please find the below link for your information,
pythonzero.com/2021/12/how-to-extr...

Collapse
 
omejiasq profile image
omejiasq

hello guys. Can i to use today this steps and code to download comments and post of facebook?

Collapse
 
mezaouryounes profile image
Younes Mezaour

Hello :) thank you for this share, can you please tell me the equivalent of "post_url_text" for the French version?

thanks :)

Collapse
 
benjaminbenteke profile image
Benjamin Benteke Longau

Hello Sir thank you for you tuto. How can we also collect for each post all comments, likens and number of share?