Introduction
Web scraping, while immensely useful, often requires developers to navigate a sea of tools and libraries, each with its own quirks and intricacies. Whether it's JSDOM, Cheerio, Playwright, or even just plain old vanilla JS in the DevTools console, moving between these platforms can be a challenge.
Enter Portadom, your new best friend in the world of web scraping.
What is Portadom?
Portadom provides a consistent DOM manipulation interface across:
- Browser API
- JSDOM
- Cheerio
- Playwright
This means you no longer have to rewrite or refactor large chunks of your code when switching between these tools. Instead, you can focus on the logic of your web scraping tasks and let Portadom handle the DOM manipulation intricacies.
The Portadom Workflow
Imagine you're working on a project to scrape data from several websites. You initially start by prototyping in the DevTools console using vanilla JS. Once you've figured out the transformations and data extractions, you realize that some sites can be scraped with static HTML, while others need a JS runtime.
1. Prototyping with Vanilla JS
You start with a simple site and define your transformations directly in the DevTools:
let title = document.querySelector('h1').innerText;
2. Static HTML with JSDOM or Cheerio
For sites where static HTML is sufficient, you can easily migrate your vanilla JS logic:
import { load as loadCheerio } from 'cheerio';
import { cheerioPortadom } from 'portadom';
const html = `<h1>Welcome to Portadom</h1>`;
const $ = loadCheerio(html);
const dom = cheerioPortadom($.root(), null);
const title = await dom.findOne('h1').text();
With Portadom, the transition feels almost seamless. The core logic remains consistent, and only the setup changes.
3. Dynamic Sites with Playwright
For websites that rely heavily on JavaScript, you'd need a tool like Playwright. But with Portadom, even this transition is smooth:
import { playwrightLocatorPortadom } from 'portadom';
const page = await browser.newPage();
await page.goto('https://example.com');
const bodyLoc = page.locator('body');
const dom = playwrightLocatorPortadom(bodyLoc, page);
const title = await dom.findOne('h1').text();
Notice how, once again, only the setup changed. The actual DOM querying logic remains consistent, thanks to Portadom.
Embracing Flexibility
Portadom is all about flexibility. No matter where you start — be it with Cheerio for static HTML parsing or Playwright for dynamic sites — you're never locked in. If your needs change, Portadom makes it easy to switch your underlying platform without overhauling your entire scraping logic.
Take the Leap with Portadom
Web scraping is finicky - everything breaks all the time. With Portadom, you're equipped with a tool that lets you focus on crafting the perfect data extraction strategy without getting bogged down by the intricacies of various DOM manipulation libraries. Dive in and let Portadom streamline your web scraping journey!
Portadom was already successfully used in scraping:
Portadom currently supports following manipulations:
- Element attributes, properties, text
-
findOne
- equivalent todocument.querySelector
-
findMany
- equivalent todocument.querySelectorAll
-
closest
- equivalent toElement.closest
-
parent
- equivalent toElement.parent
-
children
- equivalent toElement.children
-
root
- Get document root -
remove
- Remove current Element -
getCommonAncestor
- Get common ancestor between this andother
Element -
getCommonAncestorFromSelector
- Get common ancestor between this andother
Element (found by selector)
Chaining
For cross-compatibility, each method on a Portadom instance returns
a Promise.
But this then leads to then
/ await
hell when you need to call multiple methods in a row:
const employerName = (await (await el.findOne('.employer'))?.text()) ?? null;
To get around that, the results are wrapped in chainable instance. This applies to each method that returns a Portadom instance, or an array of Portadom instances.
So instead, we can call:
const employerName = await el.findOne('.employer').text();
You don't have to chain the commands. Instead, you can access the associated promise under promise
property. For example this:
const mapPromises = await dom.findOne('ul')
.parent()
.findMany('li[data-id]')
.map((li) => li.attr('data-id'));
const attrs = await Promise.all(mapResult);
Is the same as:
const ul = await dom.findOne('ul').promise;
const parent = await ul?.parent().promise;
const idEls = await parent?.findMany('li[data-id]').promise;
const mapPromises = idEls?.map((li) => li.attr('data-id')) ?? [];
const attrs = await Promise.all(mapPromises);
Examples
Example - Profesia.sk
// Following lines added for completeness
const $ = loadCheerio(html);
const dom = cheerioPortadom($.root(), url);
// ...
const rootEl = dom.root();
const url = await dom.url();
// Find and extract data
const entries = await rootEl.findMany('.list-row:not(.native-agent):not(.reach-list)')
.mapAsyncSerial(async (el) => {
const employerName = await el.findOne('.employer').text();
const employerUrl = await el.findOne('.offer-company-logo-link').href();
const employerLogoUrl = await el.findOne('.offer-company-logo-link img').src();
const offerUrlEl = el.findOne('h2 a');
const offerUrl = await offerUrlEl.href();
const offerName = await offerUrlEl.text();
const offerId = offerUrl?.match(/O\d{2,}/)?.[0] ?? null;
const location = await el.findOne('.job-location').text();
const salaryText = await el.findOne('.label-group > a[data-dimension7="Salary label"]').text();
const labels = await el.findMany('.label-group > a:not([data-dimension7="Salary label"])')
.mapAsyncSerial((el) => el.text())
.then((arr) => arr.filter(Boolean) as string[]);
const footerInfoEl = el.findOne('.list-footer .info');
const lastChangeRelativeTimeEl = footerInfoEl.findOne('strong');
const lastChangeRelativeTime = await lastChangeRelativeTimeEl.text();
// Remove the element so it's easier to get the text content
await lastChangeRelativeTimeEl.remove();
const lastChangeTypeText = await footerInfoEl.textAsLower();
const lastChangeType = lastChangeTypeText === 'pridané' ? 'added' : 'modified';
return {
listingUrl: url,
employerName,
employerUrl,
employerLogoUrl,
offerName,
offerUrl,
offerId,
location,
labels,
lastChangeRelativeTime,
lastChangeType,
};
});
Example - SKCRIS
// Following lines added for completeness. Edited for brevity.
const $ = loadCheerio(html);
const dom = cheerioPortadom($.root(), url);
// ...
const url = await dom.url();
const rootEl = dom.root();
const tableDataEls = await rootEl
.findMany('.detail > tr')
.filterAsyncSerial((el) => el.text()) // Remove empty tags
.slice(1, -1).promise; // Remove first row (heading) and last row (related resources)
const tableData = tableDataEls.reduce(async (promiseAgg, rowEl) => {
const agg = await promiseAgg;
const [title, val] = await rowEl.children()
.mapAsyncSerial(async (el) => {
const text = await el.text();
return text?.replace(/\s+/g, ' ') ?? null;
});
if (!title) return agg;
agg[title] = val ?? null;
return agg;
}, Promise.resolve({} as Record<string, string | null>));
return tableData;
Example - Facebook post timestamp
Facebook prompted the need to add the getCommonAncestor
method, as Facebook's HTML doesn't provide many reliable patterns to work with.
Note how we used getCommonAncestor
to get an element that wasn't easily targettable by class/attribute selectors.
// Following lines added for completeness. Edited for brevity.
const body = await page.evaluateHandle(() => document.body)
const dom = playwrightHandlePortadom(body, page);
// ...
// Find container with post stats
const likesEl = await dom.findOne('[aria-label*="Like:"]').promise;
const commentsEl = await dom
.findMany('[role="button"] [dir="auto"]')
.findAsyncSerial(async (el) => {
const text = await el.text();
return text?.match(URL_REGEX.COMMENT_COUNT);
}).promise;
const statsContainerEl =
likesEl?.node && commentsEl?.node
? await likesEl.getCommonAncestor(commentsEl.node).promise
: null;
// "6.9K views"
const viewsText = await statsContainerEl
?.children()
.findAsyncSerial(async (domEl) => {
const text = await domEl.text();
return text?.match(/views/i);
})
.textAsLower();
Top comments (5)
There are multiple way to reduce lodash size.
Also this is a Web Scraping library. The source code should already be there and thus no network traffic time. Isn't it micro-optimize?
Great suggestions, thanks! Was a bit lazy on my side
Updated, it doesn't have any dependencies now :)
Examples of how you scrape Facebook, Amazon Products, Profesia.sk & SKCRIS would be welcome
Added the ones I could - Facebook, Profesia.sk, SKCRIS :)