Contents: intro, imports, what will be scraped, process, code, links, outro.
Intro
This blog post is a continuation of Google's web scraping series. Here you'll see how to scrape Google Top Stories from Organic Search Results using Python with beautifulsoup, requests libraries. An alternative API solution will be shown.
Imports
from bs4 import BeautifulSoup
import requests, lxml
from serpapi import GoogleSearch
What will be scraped
Process
Selecting container, published date, title CSS selectors.
Selecting link CSS selector.
Code
from bs4 import BeautifulSoup
import requests, lxml
headers = {
'User-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
params = {
'q': 'john cena',
'hl': 'en',
'gl': 'us',
}
html = requests.get('https://www.google.com/search?q=', headers=headers, params=params).text
soup = BeautifulSoup(html, 'lxml')
for result in soup.select('.yLWA8b'):
title = result.select_one('.nDgy9d').text.replace('\n', '')
link = result.select_one('.WlydOe')['href']
published_date = result.select_one('.ecEXdc span').text
print(f'{title}\n{link}\n{published_date}\n')
----------------
'''
Nikki Bella's Reported Response to John Cena's New Attitude on Having Kids Is All Class
https://www.yahoo.com/entertainment/nikki-bellas-reported-response-john-194941780.html
2 days ago
When will John Cena win his seventeenth World Championship?
https://www.sportskeeda.com/wwe/when-will-john-cena-win-seventeenth-world-championship
16 hours ago
Is Vin Diesel Really The Same Height As John Cena? | Screen Rant
https://screenrant.com/f9-john-cena-vin-diesel-height-comparison-same-different/
20 hours ago
'''
Using Google Top Stories API
SerpApi is a paid API with a free trial of 5,000 searches.
In this example, Google Top Stories API provides quite a few more results in relation to the code above. Also, if you do not interested in maintaining the parse, figuring out why things don't behave as they should, or bypassing blocks from Google, an API solution is a way to go.
from serpapi import GoogleSearch
import os, json
params = {
"api_key": os.environ["API_KEY"], # pycharm environment
"engine": "google",
"q": "john cena",
"gl": "us",
"hl": "en"
}
search = GoogleSearch(params)
results = search.get_dict()
for result in results['top_stories']:
print(json.dumps(result, indent=2, ensure_ascii=False))
-------------
'''
{
"title": "Nikki Bella's Reported Response to John Cena's New Attitude on Having Kids Is All Class",
"link": "https://www.yahoo.com/entertainment/nikki-bellas-reported-response-john-194941780.html",
"source": "Yahoo",
"date": "2 days ago",
"thumbnail": "https://serpapi.com/searches/60e30aaf1071f423feda11f5/images/b544986d11640bd39c43fea8c2bbc111dec815b3d8089f08.jpeg"
}
{
"title": "When will John Cena win his seventeenth World Championship?",
"link": "https://www.sportskeeda.com/wwe/when-will-john-cena-win-seventeenth-world-championship",
"source": "Sportskeeda",
"date": "17 hours ago",
"thumbnail": "https://serpapi.com/searches/60e30aaf1071f423feda11f5/images/b544986d11640bd38557cb4783461be1ca3dfe7ff6bcb2a0.jpeg"
}
...
'''
Links
Code in the online IDE • Google Top Stories API
Outro
If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.
Yours,
Dimitry, and the rest of SerpApi Team.
Top comments (0)