DEV Community

Jonas Johnson
Jonas Johnson

Posted on

web scraping with python

I’ve recently run into problems scraping the new google site. Anyone else?

Top comments (4)

Collapse
 
akashjeez profile image
Akash Jeez

Scrapping giant websites like Google, Amazon, AWS, Apple etc will work for short time but not for long run because if admin added more security into the site.
Better to look for API which can get the same data as you trying to scrape from the site.
for Google Search -> pypi.org/project/googlesearch-python/

Collapse
 
jonasjohnson profile image
Jonas Johnson

Without checking I can’t be completely certain but I believe I was using this and ran into some issues where I had to rewrite this to get the specifics that I needed. I will look into it again though, thanks!

Collapse
 
nfkt profile image
nfkt

Why scrape google

Collapse
 
jonasjohnson profile image
Jonas Johnson

I’m scraping the results to get more info about the websites based on a specific query.