DEV Community

Aurelix Solutions
Aurelix Solutions

Posted on Edited on

A broken-link and SEO audit of your whole site, with a ranked fix list for your developer

Most sites collect small SEO problems quietly: a page that now 404s, a link to a partner site that died, a title that grew past 60 characters, a noindex left on after a migration, a new section nobody added to the sitemap. None of them is urgent on its own, which is why they pile up. What a site owner or an agency needs is not a 40-page report but a short list, worst first, with the exact URLs to fix.

This post sets that up as a scheduled audit in Python.

Disclosure: I'm Aurelix, and I built the Broken Link Crawler & SEO Audit. It's a paid tool (prices below).

What one run returns

You give it one site. It reads the site's XML sitemaps (from robots.txt or /sitemap.xml), follows internal links on the same host up to maxPages, and returns:

  • One page row per URL: status_code, redirects, title and title_length, meta_description and its length, h1_count, canonical_url, noindex, word_count, link and image counts, broken_links (each with its URL and status code), images_missing_alt_count, in_sitemap, response_time_ms, and issues: the issue codes on that page, most severe first.
  • One summary row: pages_audited, health_score (the share of pages with no high or medium issue), status-code counts, and issues ranked by severity and pages affected, each with up to five example URLs.

Broken means a real 404/410, a 5xx or a dead host. Links that answer 401, 403 or 429 (common for sites that block automated checks) are counted as unverified, so they don't clutter the fix list.

A fix list and a broken-link sheet

pip install apify-client
Enter fullscreen mode Exit fullscreen mode
import csv
import os

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/seo-audit-crawler").call(
    run_input={
        "siteUrl": "https://books.toscrape.com",
        "maxPages": 200,
        "useSitemap": True,
        "checkExternalLinks": True,
        "maxLinkChecks": 1000,
    }
)
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
summary = next(r for r in rows if r["record_type"] == "summary")
pages = [r for r in rows if r["record_type"] == "page"]

print(f"Health score {summary['health_score']} over {summary['pages_audited']} pages")
for i in summary["issues"][:5]:
    print(f"  {i['severity']:<6} {i['issue']}: {i['pages']} pages, e.g. {i['examples'][0]}")

severity = {i["issue"]: i["severity"] for i in summary["issues"]}
with open("fix_list.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["severity", "issue", "url", "status_code", "title"])
    for p in pages:
        for code in p["issues"]:
            if severity.get(code) in ("high", "medium"):
                w.writerow([severity[code], code, p["url"], p["status_code"], p["title"]])

with open("broken_links.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["on_page", "broken_link", "status_code", "error"])
    for p in pages:
        for b in p["broken_links"]:
            w.writerow([p["url"], b["url"], b["status_code"], b["error"]])
Enter fullscreen mode Exit fullscreen mode

Replace siteUrl with your own site, or a client's with their permission. fix_list.csv is the developer's to-do list, sorted by nothing more clever than severity; broken_links.csv tells an editor which link to change on which page.

To track progress, save the input as a task in the Apify Console and schedule it weekly or monthly, then compare the summary rows: health_score and the pages count of each issue. There is no built-in diff between runs.

A few input choices that matter:

  • Quick pre-launch pass: "useSitemap": false, "checkExternalLinks": false, "maxLinkChecks": 0 checks titles, meta, H1, canonicals and noindex fast, without testing links.
  • Sitemap gaps: with useSitemap on, indexable pages missing from the sitemap get not_in_sitemap, and sitemap URLs that redirect, error or are noindex get sitemap_url_not_ok.
  • Link budget: maxLinkChecks (up to 5,000) caps the extra requests for external links and files. Links over the cap are counted in unchecked_links_count, never reported as fine.

Why this tool

  • $2.50 per 1,000 pages audited ($1.75 on paid plans), cheaper than the leading tools.
  • On-page checks (title, meta, H1, canonical, noindex) and every link tested, up to 5,000 extra link checks per run at no extra charge.
  • Indexable pages missing from the sitemap are listed.
  • 401, 403 and 429 answers are kept apart from broken links.
  • A summary that ranks what to fix first, with a health score.
  • What it doesn't cover: Open Graph, hreflang and schema checks, Core Web Vitals and accessibility.

If what you need from the site is its content rather than an audit, for search or an AI assistant, the Website to Markdown API crawls the same way and returns clean Markdown per page.

Limits

  • One host per run. www. and the bare domain count as one site; other subdomains are checked as external links, not crawled.
  • No JavaScript rendering. It audits the HTML the server sends; links injected only by JavaScript are not found, and there are no Lighthouse or Core Web Vitals scores. response_time_ms is the download time from Apify's servers.
  • Polite pace: at most two requests a second per site, slower with a robots.txt Crawl-delay, about 1,100 pages an hour. A full 2,000-page audit needs a run timeout of about 2 hours.
  • Up to 2,000 pages per run. HTML pages over 2 MB are skipped and listed in the summary.
  • robots.txt is always obeyed, for the site and for every external link checked.
  • Duplicate titles and descriptions are compared only across the pages audited in that run.

Cost

Each audited page with an HTTP status is charged once, including redirects and 404s, because those are findings too. The summary row, every link check, pages blocked by robots.txt, pages that could not be loaded, throttled pages and non-HTML files are free, so a run against a site that is down costs only the start fee. The current price per page is on the Store page; Apify's free plan includes $5 of credit a month, enough for a monthly audit of a site with a few hundred pages.

Links

Top comments (2)

Collapse
 
steven_mock_afbb801174fc3 profile image
Steven Mock •

The ranked fix list is the part that makes this actually usable for a small business. A 40-page audit usually gets filed away, but a short worst-first list with exact URLs is something a developer — or an owner — can work through in an afternoon. Separating 401/403/429 from genuinely broken links is a smart call too, and the noindex-left-on-after-a-migration example is painfully familiar. Do you find most small sites' biggest losses are broken links on pages that still get traffic, or pages that quietly fell out of the sitemap?

Collapse
 
aurelix_solutions profile image
Aurelix Solutions •

Thanks! It really depends on the site, and the audit doesn't guess traffic. It ranks by severity, then by how many pages are affected: server errors and broken links are high, while noindex pages and sitemap URLs that redirect or are noindex are medium. So a noindex left on after a migration still makes the list. One caveat: a stray noindex on a key page can cost more than any broken link, because the whole page drops out of search. If one shows up on a page that matters, fix it first.