Disclosure: I'm Lindenbyte, the small studio that built the Apify Actor used below, and this post was
drafted with AI assistance. You can do all of this by hand by openingsitemap.xmlin a browser; the Actor
adds bulk sites, "only new since last run" memory, scheduling and status checks.
Almost every site tells search engines about its pages in an XML sitemap. That makes the sitemap the cheapest
competitor feed there is: when a rival launches a product, a landing page or a blog post, it usually shows up
there within hours. You just need something that reads it on a schedule and remembers what it already saw.
That is what the Sitemap Monitor Actor does. Below
are four real runs: an inventory, a two-run monitor, a status check, and the schedule that ties it together.
1. Take a full inventory (with a filter)
Give it a domain, not a sitemap URL: it reads robots.txt, follows sitemap indexes and gzip files for you.
To list only a store's product pages:
{
"sites": ["allbirds.com"],
"urlPattern": "/products/",
"maxUrlsPerSite": 3000
}
Result on our run (2026-11-13): it found 9 sitemaps via robots.txt, read 2,165 URLs in under 2 seconds and
kept 315 product URLs; the other 1,850 (collections, pages, blog articles) were filtered out and free.
Every row has url, lastmod, changefreq, imageCount, hreflang alternates and the sitemapUrl it came
from.
One honest caveat from this run: all 315 products carried the same lastmod timestamp, so on this store
lastmod tells you nothing about which product is new. That's common, and it is exactly why the next step
compares URLs rather than trusting dates.
2. Switch on monitoring mode
Monitoring mode remembers every URL per site (under a stateKey) and on later runs outputs only the URLs it
has not seen before. With emitOnBaseline: false the first run outputs nothing: it just records the baseline,
so you don't pay for thousands of rows you already know about.
I pointed it at TechCrunch's news sitemap, which lists articles from roughly the last two days:
{
"sites": ["https://techcrunch.com/news-sitemap.xml"],
"onlyNew": true,
"emitOnBaseline": false,
"stateKey": "tutorial-07"
}
Baseline run (10:27 UTC): 34 URLs scanned, 0 saved, status message "Baseline recorded: 34 URLs remembered".
Cost: just the $0.003 start fee.
Second run (11:56 UTC): "No new URLs since the last run (34 URLs checked)". That's correct, not a bug:
TechCrunch's newest article in the sitemap was still from 22:42 UTC the night before. A quiet sitemap costs you
only the start fee, which is what you want from a monitor.
To see it catch something, I baselined a busier sitemap, BBC's news sitemap (stateKey: "tutorial-07-bbc"):
996 URLs remembered at 11:56 UTC. The same input 40 minutes later returned 22 new URLs, and only
those: BBC News articles and videos, football reports, and pieces from the Sinhala, Marathi, Polish, Azeri and
Urdu services, each with its lastmod (12:01 to 12:24 UTC). Cost of that run: $0.003 start + 22 x $0.0005 =
$0.014.
Sites that block datacenter traffic won't work in this version: the Guardian's news sitemap answered HTTP 403
from Apify's servers, and the run failed loudly rather than reporting "nothing new".
Two details worth knowing. Each stateKey is a separate memory, so you can run "competitors-daily" and
"clients-weekly" side by side. And "new" means new in the sitemap: URLs that disappear are not reported in
this version.
3. Check the HTTP status of every sitemap URL
Add "checkStatus": true and each row also gets statusCode, finalUrl and redirectCount. Sitemaps that
list 404s or redirect chains waste crawl budget, so this is a quick SEO audit of your own site (or a peek at
how tidy a competitor is).
I ran it twice on real sites:
-
docs.apify.comwithurlPattern: "/academy/": 4,214 URLs scanned, 201 checked, 201 of 201 returned 200 with no redirects. A healthy sitemap; most well-run sites look like this. -
allbirds.comproducts, default speed (5 parallel checks, 250 ms per-host delay): 84 returned 200 and 66 returned 429 Too Many Requests. That isn't a broken sitemap, it's the store's rate limiter. Re-running with"statusConcurrency": 1, "perHostDelayMs": 1500gave 150 of 150 = 200 in about 4 minutes.
Lesson: if you see a block of 429s, slow down before concluding anything. The status check also respects
robots.txt and Crawl-delay on its own.
4. Schedule it and send only the new URLs somewhere useful
- In Apify Console, save your monitoring input as a task (e.g. "competitors - new pages").
- Add a schedule: daily at 07:00 for most competitors, hourly for news sites.
- Add an integration on the task: Slack or email built in, or a webhook to your own endpoint ("Run succeeded" event). Runs with no new URLs produce an empty dataset, so filter those out.
If you'd rather pull results yourself, this starts a monitoring run and prints only the new pages (run it from cron
or CI):
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("lindenbyte/sitemap-monitor-new-pages").call(run_input={
"sites": ["competitor.com", "https://techcrunch.com/news-sitemap.xml"],
"onlyNew": True,
"emitOnBaseline": False,
"stateKey": "competitors-daily",
"urlPattern": "/(blog|products|news)/|/20\\d\\d/",
})
new_pages = [row for row in client.dataset(run.default_dataset_id).iterate_items()
if row.get("isNew")]
for row in new_pages:
print(row["site"], row["url"], row.get("lastmod") or "")
print(f"{len(new_pages)} new page(s)")
From there it's one requests.post(SLACK_WEBHOOK_URL, json={"text": ...}) to get a message in a channel,
or feed the URLs into a scraper so you only crawl what changed.
Limits (honest ones)
- It only sees what's in the sitemap. Sites without one return an error for that site; it is not a crawler.
- Some sites block non-browser requests to their sitemaps; those fail with a clear error in the
SUMMARY. - News sitemaps only cover a short window, so schedule them at least daily or you'll miss articles.
- Removed URLs aren't reported yet.
Cost: $0.003 per run, $0.0005 per URL saved, $0.001 per successful status check. Already-seen, filtered
and silent-baseline URLs are free. The 315-product inventory above is about $0.16; a daily monitor that finds
five new pages costs under a cent a day.
Not affiliated with Allbirds, TechCrunch or Apify docs; all examples read public sitemaps.
Top comments (1)
Nice write-up, especially the 429 note. That one is easy to misread as a broken sitemap. Two things I'd consider for monitoring mode: normalize URLs before diffing (trailing slash, http vs https, tracking params), or a site migration shows up as hundreds of "new" pages overnight. And for the removed-URLs gap, flagging a URL only after it has been missing for 2-3 consecutive runs avoids false alerts when a sitemap fetch comes back partial. Does the stateKey memory carry over if a site moves its sitemap to a new path?