In this post, I am going to discuss another cloud-based scraping tool that takes care of many of the issues you usually face while scraping websites. This platform has been introduced by ScrapingBee, a cloud-based Scraping tool.
What is ScrapingBee
If you visit their website, you will find something like below:
ScrapingBee API handles headless browsers and rotates proxies for you.
As it suggests, it is offering you all the things to deal with the issues you usually come across while writing your scrapers, especially the availability of proxies and headless scraping. No installation of web drivers for Selenium, yay!
Development
ScrapingBee is based on REST API hence it can be consumed in any programming language. Since this post is related to Python so I’d be mainly focusing on requests library to use this tool.
Once you get registered and confirmed with them they will provide you an API KEY which you can see on the dashboard. They provide 1000 free API calls which are enough to test their platform. Otherwise, they offer different plans from starter to the enterprise which you can view here.
Let’s try a simple example.
On executing this code, you will see output like below:
Did you notice, every time it returns a new IP address, cool, isn’t it?
You can also pass your own custom headers where needed. If I change URL_TO_SCRAPE to https://httpbin.org/headers, it will dump the default headers.
So I changed the URL of httpbin to display headers. First I set the parameter forward_headers as True and then I override the existing header, in this case, User-Agent parameter as Scn-User-Agent. As per documents, you have to set Scn- as a prefix to override the existing one or add your own one. In our case, I added a new header as Scn-X-MyOwn-Header. When I run this it displays the following:
Cool, No?
Creating OLX Scraper
Like previous scraping related posts, I am going to pick OLX again for this post. I will iterate the list first and then will scrape individual items. Below is the complete code.
I am using Beautifulsoup to parse HTML. I have only extracted Price here because the purpose is to tell about the API itself than Beautifulsoup. You should see my post here in case you are new to scraping and Python.
Here I introduced another parameter, country_code which I used in the USA. This parameter ensures to use a proxy from the US only instead of any random country. Anyways, when I scrape Tablets section of OLX it prints following:
Conclusion
In this post, you learned how to use ScrarpingBee API for scraping purposes. Whatever you can do with this API you can do it by other means as well; this API provides you everything under the umbrella, especially rendering of pages via Javascript for which you need headless browsers which, at times become cumbersome to set things up on remote machines for headless scraping. ScrarpingBee API is taking care of it and charging nominal charges for individuals and enterprises. The company I work with spend 100s of dollars on a monthly basis just for the proxy IPs.
Oh if you sign up here with my referral link or enter promo code ADNAN , you will get a 10% discount on it. In case you do not get the discount then just let me know via email on my site and I’d sure help you out.
In the coming days, I’d be writing more posts about ScrarpingBee API discussing further features.
Originally published at http://blog.adnansiddiqi.me on April 4, 2020.
Top comments (1)
Wow, embarking on your first web scraping journey with ScrapingBee API and Python sounds like a cool adventure! Personally, I've had some great experiences with ScrapingBee, BrightData, and Crawlbase. They're like the trusty sidekicks in my scraping endeavors, always reliable and efficient. Can't wait to hear about your experiences diving into the world of web scraping!