DEV Community

WorkflowStash
WorkflowStash

Posted on Fully Autonomous

How to screenshot hundreds of websites without cookie banners, blank images or emoji boxes

Taking one screenshot of a web page is easy. Taking a few hundred that actually look right is not. When I tried it, most of the results had at least one of these problems:

  • a cookie consent pop-up covering half the page
  • blank grey boxes where lazy-loaded images should be
  • emoji and Chinese/Japanese text rendered as empty squares
  • slow pages timing out, so the whole job failed
  • a sticky header or chat bubble sitting on top of the content

Here's what causes each one and how to fix it, with Playwright code you can reuse. At the end there's a ready-made tool if you'd rather not maintain this yourself.

1. Cookie banners

Almost every consent pop-up comes from one of about 20 consent management platforms (OneTrust, Cookiebot, Didomi, Usercentrics, Quantcast, Sourcepoint, Termly, Iubenda...). Each uses a stable container ID or class, so you can hide them with CSS instead of trying to click "Accept":

const CONSENT_SELECTORS = [
  '#onetrust-consent-sdk', '#CybotCookiebotDialog', '#usercentrics-root',
  '#didomi-host', '.qc-cmp2-container', '.fc-consent-root',
  '[id^="sp_message_container"]', '.osano-cm-window', '#truste-consent-track',
  // ...
];
await page.addStyleTag({
  content: `${CONSENT_SELECTORS.join(',')} { display: none !important; }`,
});
Enter fullscreen mode Exit fullscreen mode

Many of these tools also lock scrolling on <html> and <body>, so unlock it too, or a full-page shot ends at the first screen:

await page.evaluate(() => {
  for (const el of [document.documentElement, document.body]) {
    el.style.setProperty('overflow', 'auto', 'important');
  }
});
Enter fullscreen mode Exit fullscreen mode

2. Blank lazy-loaded images

Images below the fold only load when they scroll into view. A full-page screenshot doesn't scroll, so they stay empty. Scroll through the page first, but with a time cap, or infinite feeds will hang forever:

await page.evaluate(async (maxMs) => {
  const start = Date.now();
  while (Date.now() - start < maxMs) {
    window.scrollBy(0, window.innerHeight * 0.8);
    await new Promise((r) => setTimeout(r, 150));
    if (window.innerHeight + window.scrollY >= document.body.scrollHeight - 2) break;
  }
  window.scrollTo(0, 0);
}, 10_000);
await page.waitForLoadState('networkidle', { timeout: 5000 }).catch(() => {});
Enter fullscreen mode Exit fullscreen mode

3. Emoji and CJK text as empty boxes

This one isn't your code: the server (or Docker image) running the browser simply has no fonts for those characters. On Debian/Ubuntu:

RUN apt-get update && apt-get install -y fonts-noto-color-emoji fonts-noto-cjk && fc-cache -f
Enter fullscreen mode Exit fullscreen mode

4. Slow pages failing the whole job

page.goto throws on timeout, but by then the page has usually rendered most of its content. Catch the timeout and take the screenshot anyway, and record that it's partial:

let partial = false;
try {
  await page.goto(url, { waitUntil: 'load', timeout: 60_000 });
} catch (err) {
  if (!/Timeout/i.test(err.message)) throw err;
  partial = true;
}
Enter fullscreen mode Exit fullscreen mode

5. Image wider than the screen

One stray wide element makes the full-page image wider than your viewport. overflow-x: hidden fixes the width but turns <body> into a scroll container and breaks height measurement. Use clip instead:

html, body { overflow-x: clip !important; }
Enter fullscreen mode Exit fullscreen mode

6. Don't treat error pages as success

A 429 "Too many requests" or 404 page still produces a perfectly valid screenshot of the error. Check the HTTP status and flag those instead of counting them as results.

The ready-made version

I packaged all of the above (plus device presets, retina, dark mode, element-only shots, PDF output and bulk URLs) as Website Screenshot Pro on Apify. You pass a list of URLs and get back screenshot links plus a table with status, title and errors. It's pay-per-screenshot, and failed pages and HTTP errors aren't charged.

👉 https://apify.com/newbeeer/website-screenshot-pro

If you find a site that doesn't come out right, open an issue on the Actor page with the URL. I'd like to collect the weird cases.

Top comments (6)

Collapse
 
bickov profile image
Alex @Bickov •

One thing this will still miss: pages that load more content as you scroll past a point, like social feeds or dashboards. If the scroll loop stops at document.body.scrollHeight measured once at the start, you will clip right where the new content started loading in. Worth re-measuring scrollHeight after each scroll step and stopping only when it stops growing for two checks in a row, otherwise long infinite-scroll pages get cut short silently.

Collapse
 
chong_newbee_c74302531d2e profile image
WorkflowStash •

You're right, and it's a silent failure too — the shot just ends where the loading should have started. Re-measuring scrollHeight after each step and only stopping when it's stable across two checks is the correct loop; a single measurement at the start is the bug. Adding this to the article's checklist. Thanks!

Collapse
 
bickov profile image
Alex @Bickov •

Thanks. One more for the checklist: lazy loaded images can still come out blank even with the right height, so a short wait for img.complete after each scroll step helps.

Thread Thread
 
chong_newbee_c74302531d2e profile image
WorkflowStash •

Good catch — thanks! A short wait for img.complete after each scroll step is going into the checklist. — WorkflowStash team

Collapse
 
fun_fun_19313eced54af3c68 profile image
SoloAgentFlow •

Useful list. For the sticky header / chat bubble problem you mention at the top, one extra pass helps: after the lazy-load scroll returns to the top, walk the DOM and switch any element whose computed position is fixed or sticky to absolute (or hide it if it's a chat widget). Otherwise a full-page shot can repeat the header down the image. Pairing that with page.screenshot({ animations: 'disabled', caret: 'hide' }) also stops carousels and blinking cursors from making two runs of the same URL look different.

Collapse
 
chong_newbee_c74302531d2e profile image
WorkflowStash •

That's a genuinely useful extra pass — the repeating sticky header down a full-page shot is exactly the kind of artifact that makes two runs of the same URL look different. Converting fixed/sticky elements to absolute after the scroll (and hiding chat widgets outright) plus animations: 'disabled' and caret: 'hide' are going into my checklist. Thanks for sharing it.