In this post, we will learn to scrape Google Shopping Results using Node JS.
Requirements:
Before we begin, install these libraries so we can move forward and prepare our scraper.
To extract our HTML data we will use Unirest JS and for parsing the HTML data we will use Cheerio JS.
Target:
We will target to scrape the shopping results of Nike shoes.
Process:
We have installed all the things which we will need for our scraper. Now we will hit our target URL using Unirest JS to get our HTML data and then we will parse our extracted HTML data with the help of Cheerio JS.
We will target this URL:
https://www.google.com/search?q=nike shoes&tbm=shop&gl=us
Look at the tbm parameter and its value(shop, here). This value shop will tell Google that we are looking for shopping results.
Open this URL in your browser. Inspect the code. You will see that every organic shopping result is inside this tag .sh-dgr__gr-auto.
Now, we will search the tags for title, product link, price, rating, reviews, delivery, and source.
The above images are in the pattern of two at the top and one at the bottom.
We have completed our search for tags of organic shopping results. Now, we will search for the tags of ad results.
If you inspect the ad results you will see that all the ad results are inside the tag .sh-np__click-target. This tag contains all the information about the title, link, price, and source.
All the above things make our code look like this:
const unirest = require("unirest");
const cheerio = require("cheerio");
const getShoppingData = () => {
try
{
return unirest
.get("https://www.google.com/search?q=nike shoes&tbm=shop&gl=us")
.headers({
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
})
.then((response) => {
let $ = cheerio.load(response.body);
let ads = [];
$(".sh-np__click-target").each((i,el) => {
ads.push({
title: $(el).find(".sh-np__product-title").text(),
link: "https://google.com" + $(el).attr("href"),
source: $(el).find(".sh-np__seller-container").text(),
price: $(el).find(".hn9kf").text(),
delivery: $(el).find(".U6puSd").text(),
})
if($(el).find(".rz2LD").length)
{
let extensions = []
extensions = $(el).find(".rz2LD").text()
ads[i].extensions = extensions
}
})
for (let i = 0; i < ads.length; i++) {
Object.keys(ads[i]).forEach(key => ads[i][key] === "" ? delete ads[i][key] : {});
}
let shopping_results = [];
$(".sh-dgr__gr-auto").each((i,el) => {
shopping_results.push({
title: $(el).find(".Xjkr3b").text(),
link: $(el).find(".zLPF4b .eaGTj a.shntl").attr("href").substring($(el).find("a.shntl").attr("href").indexOf("=")+1),
source: $(el).find(".IuHnof").text(),
price: $(el).find(".XrAfOe .a8Pemb").text(),
rating: $(el).find(".Rsc7Yb").text(),
reviews: $(el).find(".NzUzee div").attr("aria-label") ? $(el).find(".NzUzee div").attr("aria-label").substring(0,$(el).find(".NzUzee div").attr("aria-label").indexOf(" ")) : "",
delivery: $(el).find(".vEjMR").text()
})
if($(el).find(".Ib8pOd").length)
{
let extensions = [];
extensions = $(el).find(".Ib8pOd").text();
shopping_results[i].extensions = extensions
}
})
for (let i = 0; i < shopping_results.length; i++) {
Object.keys(shopping_results[i]).forEach(key => shopping_results[i][key] === "" ? delete shopping_results[i][key] : {});
}
console.log(ads)
console.log(shopping_results)
})
}
catch(e)
{
console.log(e)
}
}
getShoppingData();
Result:
Our result should look like this 👆🏻.
With Google Shopping API
If you don't want to code and maintain the scraper in the long run then you can definitely try Serpdog | Google Search API.
Serpdog also offer 100 free requests on the first sign-up.
const axios = require('axios');
axios.get('https://api.serpdog.io/shopping?api_key=APIKEY&q=shoes&gl=us')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
Result:
Conclusion:
In this tutorial, we learned to scrape Google Shopping Results using Node JS. Feel free to comment anything you need clarification on. Follow me on Twitter Thanks for reading!
Top comments (3)
Hello, can you help me? I need to get the images displayed in the google shopping ads, I tried to use the other attributes as an example, but I still wasn't successful, can you tell me how it would be possible, I identified the class "SirUVb sh-img__image" but I couldn't access the![]()
tag
Here is the correct way of doing it: div.sh-img__image img
Hey,
Can you message me on this link: drift.me/darshankhandelwal12