Introduction to webscraping

What is web scraping?

Web scraping or screen scraping or web data extraction, etc is the act of extracting data from web pages in an automated way. The extracted data is usually in an unstructured format. After the extraction, the data usually needs to be cleaned up and be presented in a format that will be useful depending on the purpose why it was extracted.

There are different techniques and languages employed in web scraping and in my opinion, puppeteer has redefined how web scraping is done with its incredible simple API - no language war intended 😄. The goal is to extract the data and turn it into something useful, nobody cares how you get the data, well, probably your engineering manager does 😎.

Who needs web scraping?

Web scraping can be applied to countless niches ranging from e-commerce sites, real estate, finance, legal firms, entartainment, news, fashion, social media, etc. The e-commerce niche seems to be the primary target of web scrapers. Many e-commerce stores monitor prices of products on competitors' websites just to set optimum prices of products on their stores.

For example, if Amazon and eBay are selling a similar product, both stores can monitor each other's products prices to ensure they are not offering that product at a charity price and at the same time, they are not going way overboard with the price. In short, web scraping helps e-commerce stores maintain competitive edge in that niche.

Marketers can also generate quality leads through web scraping public databases, some people may consider this act as less than legal - and that leads us to the next point. Regardless of the case, extracting contact information of potential customers can be done in a split second with web scraping.

Is web scraping legal and free?

Whatever niche webscraping is used, we are talking about "free data" which translates to free value. Well, not so fast, "free" here can come with some legal implications. Knowing the restrictions attached to web scraping is a most-know for every aspiring data miner because it's going to save you alot of headaches in the future.

Most websites have a file called robots.txt placed at the root directory of their site. Example amazon.com/robots.txt. This file contains rules for scraping the site, it specifies which endpoints are allowed to be hit and which ones are not. It is very important you adhere to these rules to avoid being blocked from that site or even being sued. If you are new to robots.txt concepts, here's an article by Patrick Sexton, he did an excellent job breaking the nitty-gritty concepts The robots.txt file.

A rule of thumb applies - never overload any website that you are scraping. When you spam them with thousands of concurrent requests per second, you are making them burn more resources (bandwidth is expensive 💳). Performance of the service will be adversely affected which is something you don't want to do. oOher users may experience slow response or even server downtime. Always ensure the servers are blinking green when scraping. Let's be our brothers' keepers 👍

What and how can I monetize my web scraping skills?

It is only natural to think of the benefits of learning web scraping. After you have dedicated time and effort into learning web scraping, and then what next??? Do you just scrape for fun? Would it be cool if I tell you that you can learn this skill and generate some kind of revenue from it? Here are some few ways you can monetize your new found skills.

Getting hired - some companies hire developers solely for web scraping purposes. You don't have to be a genius in backend development before you get hired as a web scraper by a company. There are "web-scraping-specific" jobs that you can apply for. Example Ziprecruiter Webscraping Jobs
Freelancing - There are tonnes of gigs available on freelancing sites that only have to do with web scraping. You can set your price, do the task and watch the money flowing Truelancer Webscraping Jobs
Build a startup - If you like the taste of freedom from "bosses", you can decide to venture into the startup world by creating a web service that people can use and pay you for your services. There are companies that offer products using webscraping. Example Truelancer Webscraping Jobs

If you have come this far in this journey, thank you very much for your time and patience. Tab yourself three times on the back because you are a hero. 🍷

Shameless plug 😊

If you enjoyed this article and are feeling super pumped, I run 🔗 webscrapingzone.com where I teach advanced webscraping techniques by building real-world projects and how you can monetize your webscraping skills instantly without even being hired. It's still in beta stage but you can join the waiting list and get 💥 50% 💥 off when the course is released.

You can follow me on twitter - @microworlds

Thank you for your time 👍