How to Build Wayfair Price Tracker

This blog was originally posted to Crawlbase Blog

In this blog post, we'll explore creating a Wayfair price tracker, for Wayfair price tracking of the trends on this prominent online marketplace. Understanding the details of how Wayfair's prices work will help readers create a useful solution for making smart buying choices. This tutorial will demonstrate how to create a Wayfair price tracker that scrapes wayfair pricing data.

Let’s start!

Table of Contents

Prerequisites

Installing Dependencies

Setting Up Environment

Scrape data from Wayfair in HTML

• Step 1: Import the Crawlbase and fs Library
• Step 2: Specify Wayfair Page URL and API Token
• Step 3: Initializing the CrawlingAPI Object
• Step 4: Making a GET Request
• Step 5: Check the Response Status Code
• Step 6: Handle Errors

Scrape Wayfair Product’s Price Data in JSON

• Step 1: Import required libraries
• Step 2: Scrape Products price

Saving Wayfair Price Data in CSV File

Final Thoughts

Frequently Asked Questions

1. Prerequisites

Before you start tracking Wayfair prices using Crawlbase's Crawling API and JavaScript, make sure you have Node.js installed on your computer. Node.js is needed to run JavaScript code locally, which is important for scraping websites. You can get Node.js from its official website. It's also helpful to have a basic understanding of JavaScript, including things like variables, functions, loops, and changing webpage elements. Lastly, get a Crawlbase API Token for using their API efficiently. Sign up on Crawlbase's website and find your API tokens in your account settings. These tokens let you access and use the Crawling API's features.

2. Installing Dependencies

Let’s install the dependencies we’ll be using throughout the tutorial.

npm install crawlbase
npm install fs
npm install cheerio

To build the Wayfair price tracker, you'll need to install three npm libraries: crawlbase for efficient interaction with the Crawlbase API for seamless web scraping from Wayfair's site; fs for file system interaction, useful for reading from and writing to files for data processing from web scraping; and cheerio, a lightweight library implementing jQuery core features, ideal for parsing HTML/XML documents in Node.js to extract specific data during web scraping.

3. Setting Up Environment:

Open your terminal and type mkdir wayfair-price-tracker to make a new folder for your project.

mkdir wayfair-price-tracker

Now, Enter cd wayfair-price-tracker to go into the new folder. This helps you organize your project files more easily.

cd wayfair-price-tracker

Next, Type touch index.js to create a new file named index.js (you can pick a different name if you prefer).

touch index.js

4. Scrape data from Wayfair in HTML

Now that our coding environment is set up and we have our API credentials ready, let's start writing JavaScript code to extract HTML data from Wayfair's web page. Now, pick the Wayfair web page you want to scrape. In this example, we’ve chosen the Wayfair web page for the furniture category.

Step 1: Import the Crawlbase and fs Library:

const { CrawlingAPI } = require('crawlbase');
const fs = require('fs');

Step 2: Specify Wayfair Page URL and API Token:

const crawlbaseToken = 'YOUR_CRAWLBASE_TOKEN';
const wayfairPageURL = 'https://www.wayfair.com/furniture/sb0/sofas-c413892.html';

Step 3: Initializing the CrawlingAPI Object:

const api = new CrawlingAPI({ token: crawlbaseToken });

Step 4: Making a GET Request:

api.get(wayfairPageURL).then(handleCrawlResponse).catch(handleCrawlError);

Step 5: Check the Response Status Code:

function handleCrawlResponse(response) {
  if (response.statusCode === 200) {
    fs.writeFileSync('response.html', response.body);
    console.log('HTML saved to response.html');
  }
}

Step 6: Handle Errors:

function handleCrawlError(error) {
  console.error(error);
}

Complete Code:

const { CrawlingAPI } = require('crawlbase'),
  fs = require('fs'),
  crawlbaseToken = 'YOUR_CRAWLBASE_TOKEN',
  api = new CrawlingAPI({ token: crawlbaseToken }),
  wayfairPageURL = 'https://www.wayfair.com/furniture/sb0/sofas-c413892.html';

api.get(wayfairPageURL).then(handleCrawlResponse).catch(handleCrawlError);
function handleCrawlResponse(response) {
  if (response.statusCode === 200) {
    fs.writeFileSync('response.html', response.body);
    console.log('HTML saved to response.html');
  }
}

function handleCrawlError(error) {
  console.error(error);
}

Code Explanation:

• The script imports necessary modules: CrawlingAPI from 'crawlbase' and 'fs' for file system operations.
• It sets up a CrawlingAPI instance with a specified token and defines the URL to crawl, which is the Wayfair furniture category page.
• The script calls the get() method of the CrawlingAPI instance with the Wayfair page URL and specifies callback functions for handling success (handleCrawlResponse) and errors (handleCrawlError).
• In the handleCrawlResponse function, it checks if the response status code is 200 (indicating success), then writes the HTML content of the response to a file named 'response.html' using fs.writeFileSync() and logs a success message.
• In case of an error during crawling, the handleCrawlError function logs the error to the console.

HTML Output:

5. Scrape Wayfair Product’s Price Data in JSON

In this section, we'll learn scrape Wayfair Product’s price data in JSON for Wayfair price tracking. We'll do this by creating a custom JavaScript scraper using two libraries: cheerio, which is commonly used for web scraping, and fs, which helps with file operations. The script will parse the HTML code of the Wayfair webpage we obtained in previous example, extract the product prices from response.html, and organize them into a JSON array.

Step 1: Import required libraries:

const fs = require('fs'),
  cheerio = require('cheerio'),
  htmlContent = fs.readFileSync('response.html', 'utf-8'),
  $ = cheerio.load(htmlContent);

Step 2: Scrape Products price:

To get prices of products from a webpage, first, open the webpage in your browser. Then, right-click on a product and select "Inspect" to see its code. Look for the part of the code that shows the price of the product. This part is usually inside a box called a "card" that holds information about the product.

Once you find the card, identify the specific part of the code that represents the price. It might have a attribute name like “data-test-id”. Then, you can use a tool like Cheerio to find this part of the code using its attribute name.

After finding the price element with Cheerio, use the ".text()" method to grab the text inside it. This will give you the price value along with any extra characters or spaces. To clean it up and get only the price, use the ".trim()" method. This removes any unnecessary spaces around the price text, giving you a neat result.

const products = {};

// Select each product card and extract name and price
$('div[data-hb-id="Card"]').each((index, element) => {
  let productName = $(element).find('p[data-test-id="ListingCard-ListingCardName-Text"]').text().trim();
  let productPrice = $(element).find('span[data-test-id="PriceDisplay"]').text().trim();

  // Check if productName is empty, if so, assign default message
  if (productName === '') {
    productName = 'Name is not available';
  }

  // Add product details to the object
  products[productName] = { price: productPrice };
});

Complete Code:

const fs = require('fs');
const cheerio = require('cheerio');

// Read the HTML file
fs.readFile('response.html', 'utf8', (err, data) => {
  if (err) {
    console.error('Error reading file:', err);
    return;
  }

  // Load HTML content into cheerio
  const $ = cheerio.load(data);

  // Object to store product details
  const products = {};

  // Select each product card and extract name and price
  $('div[data-hb-id="Card"]').each((index, element) => {
    let productName = $(element).find('p[data-test-id="ListingCard-ListingCardName-Text"]').text().trim();
    let productPrice = $(element).find('span[data-test-id="PriceDisplay"]').text().trim();

    // Check if productName is empty, if so, assign default message
    if (productName === '') {
      productName = 'Name is not available';
    }

    // Add product details to the object
    products[productName] = { price: productPrice };
  });

  // Log product details as JSON
  console.log(JSON.stringify(products, null, 2));
});

JSON Output:

{
  "Name is not available": {
    "price": "$899.99"
  },
  "Mahwah 98'' Chenille Square Arm Sofa": {
    "price": "$689.99"
  },
  "Adelmina 88.6'' Upholstered Sofa": {
    "price": "$444.99"
  },
  "Azita 49'' Faux Leather Loveseat": {
    "price": "$217.99"
  },
  "Vroni 71.6\" Square Arm Loveseat": {
    "price": "$439.99"
  },
  "Arsha 84'' Upholstered Sofa": {
    "price": "$389.99"
  },
  "Sleeper Sofa": {
    "price": "$424.42"
  },
  "Bolware 79\" Handmade 3-Seater Sofa, Upholstered Tufted Coach, Solid Wood Sofa": {
    "price": "$389.99"
  },
  "Barrientez 85'' Upholstered Sleeper Sofa": {
    "price": "$419.99"
  },
  "Perdue 81.5\" Velvet Square Arm Convertible Sofa": {
    "price": "$399.99"
  },
  "Arvida 54'' Upholstered Loveseat": {
    "price": "$213.99"
  },
  "Benitez Twin 76'' Faux Leather Tufted Convertible Sofa": {
    "price": "$294.99"
  },
  "Liudmila 86'' Upholstered Sofa": {
    "price": "$405.99"
  }
}

6. Saving Wayfair Price Data in CSV File

In this section, we will implement a process to save the scraped product data into a CSV file. This involves utilizing the fs (File System) module to read the HTML response file, parsing it with Cheerio for data extraction, and then using the csv-writer library to write the extracted data into a CSV file.

The code reads an HTML file containing the response from a website, extracts product details such as name and price using Cheerio, and stores them in an array. Then, it uses csv-writer to create a CSV file named "wayfair-prices.csv" and writes the product details into it. Each product's name and price are stored in separate columns.

Additionally, they need to install the csv-writer library by running npm install csv-writer in their terminal/command prompt before executing the code. This library enables writing data to CSV files in a structured format, simplifying the process of storing scraped data for further analysis or usage.

npm install csv-writer

const fs = require('fs');
const cheerio = require('cheerio');
const csvWriter = require('csv-writer').createObjectCsvWriter;

// Read the HTML file
fs.readFile('response.html', 'utf8', (err, data) => {
  if (err) {
    console.error('Error reading file:', err);
    return;
  }

  // Load HTML content into cheerio
  const $ = cheerio.load(data);

  // Array to store product details
  const products = [];

  // Select each product card and extract name and price
  $('div[data-hb-id="Card"]').each((index, element) => {
    let productName = $(element).find('p[data-test-id="ListingCard-ListingCardName-Text"]').text().trim();
    let productPrice = $(element).find('span[data-test-id="PriceDisplay"]').text().trim();

    // Check if productName is empty, if so, assign default message
    if (productName === '') {
      productName = 'Name is not available';
    }

    // Push product details to the array
    products.push({ name: productName, price: productPrice });
  });

  // Write product details to CSV file
  const csvWriterInstance = csvWriter({
    path: 'wayfair-prices.csv',
    header: [
      { id: 'name', title: 'Product Name' },
      { id: 'price', title: 'Price' },
    ],
  });

  csvWriterInstance
    .writeRecords(products)
    .then(() => console.log('CSV file successfully written'))
    .catch((err) => console.error('Error writing CSV file:', err));
});

Final Thoughts

In this article, we learned how to make a Wayfair price tracker using the Crawlbase Crawling API and JavaScript. We started by getting the HTML data from the Wayfair webpage and saved it in a file called response.html. Then, we made a special scraper to get the names and prices of each product from the webpage. We hope this tutorial was helpful and easy to understand. If you have any questions or feedback about this article, please don't hesitate to reach out to Crawlbase support. Our team will get back to you within a day.

Related Guides:

How to Scrape Trulia

How to Scrape Wikipedia

How to Scrape TripAdvisor

How to Scrape Google News

Frequently Asked Questions

What is a Wayfair price tracker?

Wayfair price tracker allows you to monitor the prices of products listed on the Wayfair online marketplace. It automatically gathers pricing information for specific items and provides users with updates on any changes in price over time. By utilizing such a tool, shoppers can stay informed about fluctuations in prices, enabling them to make more informed purchasing decisions and potentially save money by capitalizing on discounts or price drops offered by Wayfair sellers.

What is Wayfair price tracking?

Wayfair price tracking involves monitoring the prices of products on the Wayfair platform to stay informed about fluctuations, discounts, and promotions. It helps consumers make informed purchase decisions and potentially save money by timing their purchases strategically.

How Does Wayfair Pricing Work?

Wayfair pricing operates on a dynamic model influenced by various factors including product demand, availability, and competition. Sellers on Wayfair set their own prices, which can fluctuate based on market conditions. Additionally, Wayfair may adjust prices periodically to remain competitive within the online marketplace. Discounts, promotions, and sales events also contribute to pricing variations. This dynamic pricing system allows Wayfair to adapt to market changes swiftly while offering customers a range of pricing options for products across different categories.

Are prices subject to change on Wayfair?

Wayfair is known for its fluctuating prices, often changing across different locations and even within the span of a single day. These price shifts are primarily driven by Wayfair's algorithmic pricing model, which constantly collects and analyzes data in real-time. Due to these dynamic fluctuations, it can be challenging for consumers to determine a fixed price for any given product. Tracking these price changes on Wayfair and its affiliated websites becomes even more challenging for users.

To navigate through these price variations and find the best deals on Wayfair products, a reliable price tracking and monitoring service is essential. By utilizing a solution like Crawlbase Crawling API, users can effectively monitor Wayfair's prices. This enables users to make informed purchasing decisions and secure the best possible deals on Wayfair products.

Is it legal to use Wayfair price tracker?

Using a Wayfair price tracker is generally legal, as it involves monitoring publicly available information on the Wayfair website. However, users should ensure that they comply with Wayfair's terms of service and any applicable laws or regulations regarding data scraping and online activity. It's essential to use the price tracker for personal use only and avoid any actions that could be considered abusive or violate the website's terms of use. Always consult legal advice if unsure about the legality of using such tools.

How to track price drops on Wayfair?

To track price drops on Wayfair, one effective method is to utilize a web scraping tool like Crawlbase Crawling API. This API can extract product prices from Wayfair's website in real-time, allowing users to monitor changes efficiently. By integrating the Crawlbase API into their system, users can automate the process of collecting price data from Wayfair, enabling them to stay informed about price drops and make timely purchasing decisions to secure the best deals.