Hello world,
Today, drawing on our team's years of experience, we're launching Crawlee, the the web scraping and browser automation library for Node.js that's built for fastest development and maximum reliability in production.
Main features
🖼 Supports headless browsers with Playwright or Puppeteer
⚡️ Supports raw HTTP crawling with Cheerio or JSDOM
🎛 Automated parallelization and scaling of crawlers for top performance
🐾 Avoids blocking using smart sessions, proxies, and browser fingerprints
🚎 Simple management and persistence of queues of URLs to crawl
🗜 Written completely in TypeScript for type safety and code autocompletion
📚 Comprehensive documentation, code examples, and tutorials
💪🏼 Actively maintained and developed by Apify—we use it ourselves!
Getting started
Visit crawlee.dev or run the following command:
npx crawlee create my-crawler
Liked Crawlee?
💛 You can support the project on GitHub, Product Hunt, or Hacker News
Top comments (3)
Amazing! 😁
Do you plan on writing more posts using it to showcase?
I would love to see the typical stand-alone example of a google maps crawler using Playwright and Crawlee! 😍
If you don't mind an actual production grade example then this Google Maps scraper uses Apify SDK v2, from which we spun Crawlee off. Apify SDK and Crawlee interface is almost the same.
Crawlee is essentially Apify SDK v3, without the things that could only be used on the Apify platform. If you wish, you can read here, why we did the spin off.
We will migrate the Google Maps scraper to Crawlee soon, to leverage some of the new features.
Than you for all this information, much appreciated 😁