DEV Community

Cover image for Scrape Bing Related Searches with Python
Dmitriy Zub ☀️
Dmitriy Zub ☀️

Posted on • Updated on

Scrape Bing Related Searches with Python

Contents: intro, imports, what will be scraped, process, code, links, outro.

Intro

This blog post contains visual info about how to scrape Bing Related Searches using Python.

Imports

from bs4 import BeautifulSoup
import requests
import lxml
from serpapi import GoogleSearch
import os # used for creating an environment variable on replit.com
Enter fullscreen mode Exit fullscreen mode

What will be scraped

image

Process

Choosing the right CSS selectors. I'll show two approaches:

  1. Using Chrome Dev tools to see the HTML layout and test selectors with SelectorGadget extension
  2. Using solely SelectorGadget extension

From there either of selectors from two approaches will work:

Code

from bs4 import BeautifulSoup
import requests, lxml

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

html = requests.get('https://www.bing.com/search?q=lion king&hl=en', headers=headers)
soup = BeautifulSoup(html.content, 'lxml')

for related_search in soup.select('.b_rs ul li'):
    title = related_search.text
    link = f"https://www.bing.com{related_search.a['href']}"
    print(f'{title}\n{link}')

# part of the output:
'''
lion
https://www.bing.com/search?q=lion&FORM=QSRE1
jeremy irons
https://www.bing.com/search?q=jeremy+irons&FORM=QSRE2
'''
Enter fullscreen mode Exit fullscreen mode

Using Bing Related Searches API

SerpApi is a paid API with a free trial of 5,000 searches.
Here's how you can get the results:


from serpapi import GoogleSearch
import  os

params = {
  "api_key": os.environ["API_KEY"], # pycharm environment
  "engine": "bing",
  "q": "lion king"
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results['related_searches']:
    query = result['query']
    link = result['link']
    print(f'{query}\n{link}')

# part of the output:
'''
elton john circle of life
https://www.bing.com/search?q=elton+john+circle+of+life&FORM=QSRE1
lion king theatre
https://www.bing.com/search?q=lion+king+theatre&FORM=QSRE2
'''
Enter fullscreen mode Exit fullscreen mode

Links

Code in the online IDEBing Related Searches API

Outro

If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.

Yours,
Dimitry, and the rest of SerpApi Team.

Top comments (2)

Collapse
 
jamesbond126 profile image
james bond • Edited

I have been reading all you shared here. In this you explained everything very well. If i want any further guideline we will contact you here dev.to/dmitryzub/scrape-bing-related 49spredictions.com/

Collapse
 
gosloto90 profile image
Russia Gosloto Results

The Information you shared above is great. I have been reading all you shared here. In this you explained everything very well. If i want any further guideline we will contact you here dev.to/dmitryzub/scrape-bing-relat... floridaloteryresults.com/