Splitting the pipeline into pool, fetcher, parser and storage makes it easy to swap pieces later. Two parts stood out to me:
Soft-block detection in the parser. Many tutorials skip this, and a 200 response with a CAPTCHA page quietly polluting your database is a very common failure.
Cooldown instead of deleting proxies. IPs often recover after a rest, so this saves you from burning through a pool unnecessarily.
A few things I’d add when scaling this up:
Honor Retry-After on 429s. When a site sends it, it’s a better signal than a fixed exponential backoff.
Thread safety. With ThreadPoolExecutor, the failure counters in ProxyPool and the shared SQLite connection can race. A threading.Lock around report_failure and the save call avoids odd edge cases.
Per-domain rate limiting. Once you scrape multiple sites, a per-domain delay or token bucket works better than a single global sleep.
Curious whether you’ve compared block rates between datacenter and residential proxies on the same target. A follow-up with real numbers would be great.
Thanks, glad the separation of pool, fetcher, parser and storage came across. Keeping the pieces swappable was the main goal. The soft-block check and the cooldown logic both came from getting burned in practice. A 200 with a CAPTCHA page in the database is a nasty one because nothing errors out, and I was throwing away proxies that worked fine after a short rest.
Your three additions are all good ones:
Retry-After on 429s: Agreed. When a site tells you how long to wait, that beats a guessed backoff. Reading the header and falling back to exponential backoff when it’s missing is the right approach.
Thread safety: Fair catch. With ThreadPoolExecutor the failure counters and the shared SQLite connection can race, and a threading.Lock around report_failure and the save call is the simplest fix. A per-thread connection or a single writer thread would also work.
Per-domain rate limiting: Also agreed. A single global sleep breaks down once you hit several sites. A per-domain delay or token bucket is the better design at that point.
I haven’t run a clean datacenter vs residential comparison on the same target yet. A follow-up with real block rates, cost per successful request and where each type falls over would be useful, so I’ll consider it. Thanks for the detailed feedback.
Thanks for the detailed reply, and for confirming the Retry-After and per-domain points. The CAPTCHA-behind-a-200 case is the one that bites most people, since a bad row looks exactly like a good one.
I’d definitely read the datacenter vs residential follow-up. Metrics I’d find most useful:
Success rate on the same target, split by easy sites and ones behind Cloudflare or similar
Cost per 1,000 successful requests, not per request, since that’s where residential often wins despite the higher price
Latency and retries per success
How quickly each type gets soft-blocked, with and without rotation or sticky sessions
If you run it, test one site with light protection and one with aggressive bot detection, because the gap tends to show up there. Looking forward to it!
The soft-block check in Step 4 is the part most tutorials skip, and parsing a CAPTCHA page as real data is exactly how bad rows end up in a database without anyone noticing.
Two small things I’d add for anyone taking this to production:
Honor Retry-After on 429s. If the server tells you how long to wait, using that value instead of a fixed backoff saves a lot of burned retries.
Track proxy health per domain, not globally. A proxy that’s blocked on one site is often fine on another, so a per-domain cooldown keeps good IPs in rotation longer.
Also, a 407 usually means a proxy auth/config problem rather than a blocked IP, so it might be worth logging separately from 403/429.
Curious how you’d handle sticky sessions in this setup. Pin a proxy per session ID in the pool?
Totally agree on the soft-block check. A CAPTCHA page that returns a 200 is the sneakiest failure, because everything looks healthy until someone audits the data.
Your three additions are all solid:
Retry-After: Good call. Respecting the server’s own wait time beats a fixed backoff, and the pipeline’s retry logic should read that header first and only fall back to exponential backoff when it’s missing.
Per-domain proxy health: Agreed, this is the better design. A global blocklist throws away proxies that are perfectly fine on other targets, so a per-domain cooldown is the right production upgrade.
Logging 407 separately: Yes, a 407 points to a proxy auth or config issue, not a block. Mixing it in with 403/429 would send people debugging the wrong thing.
On sticky sessions: yes, pinning a proxy per session ID is how I’d do it. Keep a session_id → proxy map and reuse that proxy for the whole session (login, cart, pagination). Only reassign it if the proxy gets blocked or hits a failure threshold, and in that case start a fresh session rather than switching IPs mid-session, since the site will often invalidate the cookies anyway. Add a TTL so idle sessions don’t hold proxies forever.
Thanks, the session_id → proxy map with a TTL is what I’d go with too. One addition: pin the whole identity to the session, not just the IP. Keep the same User-Agent, headers, and cookie jar for the session’s lifetime, because a stable IP with a changing fingerprint is just as easy to flag.
When a session dies (block, failure threshold, or TTL), retire the entire session object and start a clean one, so no half-valid cookies carry over. It’s also worth logging which session a block happened on and how many requests it had made. That shows you the real per-domain threshold, so you can rotate proactively just before it instead of reacting after the ban.
Good addition, and it fits the TLS fingerprint point from earlier in this thread. A stable IP with a changing fingerprint is just as easy to flag as the reverse, so the session should own the whole identity, not only the proxy.
Pin the full identity to the session: Same proxy, User-Agent, headers and cookie jar for the session’s lifetime. With curl_cffi, the impersonation profile should be pinned too, so the User-Agent never disagrees with the TLS handshake.
Retire the whole session on death: Whether it ends by block, failure threshold or TTL, discard the entire object and start clean. Reusing the old cookie jar with a new IP carries half-valid state, which is often worse than starting from nothing.
Log the block context: Recording which session got blocked, on which domain, after how many requests turns the per-domain threshold into a number you can measure. I’d keep that data per domain, since limits vary a lot between targets.
For proactive rotation, I’d rotate at a margin below the observed threshold (say, around 70-80% of the typical block point) rather than right at it. Thresholds drift with concurrency, time of day and site changes, so a hard cutoff will eventually be wrong. I haven’t benchmarked this myself, so treat those numbers as a starting point to tune against your own logs.
I’ll fold this into the session-handling section of the hardening follow-up. Thanks for the detailed addition.
Keeping the pool, fetcher, parser, and storage separate makes this easy to extend. Putting proxies in a cooldown instead of deleting them is a good touch, since IPs do recover.
A few things I’d tweak if you take this to production:
404s count against the proxy. resp.raise_for_status() raises httpx.HTTPStatusError, which the except httpx.HTTPError branch catches. A plain 404 or 410 therefore penalizes a healthy proxy and triggers pointless retries. I’d return early on those, or retry only on the block codes and 5xx.
Respect Retry-After on 429s. If the server tells you how long to wait, using that value beats a generic exponential backoff.
Soft-block detection can false-positive. Plenty of normal pages mention “captcha” in a script tag or footer (reCAPTCHA on a login form, for example). Checking that the expected selector exists, plus a minimum content length, is more reliable than keyword matching alone.
Thread safety. With ThreadPoolExecutor, several workers update failures and cooldown_until at once. A threading.Lock around report_failure or get avoids odd race conditions.
TLS fingerprinting. Some sites still return 403 even with clean proxies and good headers, because httpx and requests have a recognizable TLS/JA3 fingerprint. curl_cffi with browser impersonation often fixes this without a headless browser.
On your question about which targets cause trouble: Cloudflare-protected sites are the ones I hit most, and the TLS fingerprint was usually the culprit rather than the IP.
Would you consider a follow-up on async (httpx.AsyncClient plus semaphores) for the same architecture?
You’re right about the 404s. As written, raise_for_status() feeds every HTTP error into the proxy penalty path, so a dead URL gets blamed on a healthy proxy and retried for nothing. The fix is to return early on 404/410 and only retry on the block codes (403, 429) and 5xx.
Respecting Retry-After on 429s is also better than blind exponential backoff. I’d parse it, fall back to backoff when the header is missing, and cap the wait so one bad response can’t stall a worker.
On soft-block detection, keyword matching is the weakest part of the example. A reCAPTCHA script on a normal login page would trip it. Checking for an expected selector plus a minimum content length is much more reliable, and I’ll add that as the recommended approach.
The thread-safety point is fair too. With ThreadPoolExecutor, failures and cooldown_until are shared state, so a threading.Lock around get and report_failure is the right call for production.
The TLS fingerprinting note is the one I most want to work into the post. It explains why some sites keep returning 403 even with clean proxies and good headers. curl_cffiwith browser impersonation is a lighter fix than jumping straight to a headless browser. And since your experience with Cloudflare-protected sites points to the fingerprint rather than the IP, that’s a useful pointer for anyone debugging the same thing.
An async follow-up with httpx.AsyncClient and semaphores is a good idea, and the pool/fetcher/parser/storage split should carry over cleanly. I’ll plan on it, and I’ll fold your fixes into the production section.
Full-stack developer and tech writer who enjoys making complex ideas simple. I write about web development, system design, and practical coding techniques to help developers improve more quickly.
Nice walkthrough. Splitting it into pool, fetcher, parser and storage makes it easy to swap parts without rewriting everything. Putting proxies in a cooldown instead of deleting them is a good detail, since IPs often recover.
One thing I’d like to see in a follow-up: right now looks_blocked() runs in the parser, but the proxy pool never hears about it. A CAPTCHA page that comes back as a 200 still counts as report_success, so a burned proxy stays in rotation. Feeding soft blocks back into the pool’s failure count would make the health tracking more accurate.
Two small things I noticed:
Creating a new httpx.Client on every attempt skips connection reuse, which adds up at scale.
A plain 404 also goes through report_failure, so dead URLs can push healthy proxies into cooldown.
For me the hard part is sites that fingerprint TLS, where good proxies still get blocked with a clean header set. Are you planning to cover that, or the Playwright fallback, next?
The soft-block gap is the real flaw. looks_blocked() lives in the parser, so a CAPTCHA page that returns a 200 still calls report_success and the burned proxy stays in rotation. The fix is to run that check inside the fetcher (or return a signal the fetcher can act on) and call report_failure on a soft block, so the pool’s health tracking matches what the target is really doing.
The 404 point is also fair. raise_for_status() throws an HTTPError, which lands in the same except branch, so dead URLs count against healthy proxies. 404s and 410s should skip report_failure and return early. And a single httpx.Client per proxy, reused across attempts, would give you connection reuse instead of a new handshake every time.
On TLS fingerprinting, you’ve hit the limit of this setup. Proxies only change the IP, and plain httpx still presents a very recognizable TLS signature, so good IPs plus clean headers can still get blocked. Two usual ways out are curl_cffi with browser impersonation, or a real browser via Playwright. I’ll cover both in a follow-up, along with the soft-block feedback loop and the fixes above.
Separating the pool, fetcher, parser and storage means you can swap a proxy provider or move from SQLite to Postgres without touching the rest. I especially liked the cooldown approach in ProxyPool: putting a proxy in timeout after max_failures instead of deleting it fits how rotating IPs behave, since many recover after a rest. The soft-block check in the parser is also the right call. A 200 response with a CAPTCHA page is the failure that quietly corrupts a dataset, and returning None before storage keeps that out.
Two things I’d consider adding. First, honor the Retry-After header on 429 responses instead of a fixed exponential backoff, and weight proxy selection by recent success rate rather than random.choice, so a degrading IP gets picked less before it hits the cooldown threshold. Second, with check_same_thread=False and five workers sharing one SQLite connection, you may hit “database is locked” errors or interleaved commits. A threading.Lock around save(), or WAL mode (PRAGMA journal_mode=WAL), would make it safer under concurrency. Great guide overall!
Thanks for the detailed read, really glad the separation of pool, fetcher, parser and storage came across as intended. The cooldown in ProxyPool was a deliberate choice for exactly the reason you gave: rotating IPs often recover after a rest, so permanent removal wastes good capacity.
Both suggestions are fair. Honoring Retry-After on 429s is more respectful to the target and more efficient than blind exponential backoff, and weighting proxy selection by recent success rate would let a degrading IP fade out before it hits the cooldown threshold instead of after.
You’re also right about the SQLite setup. With check_same_thread=False and five workers on one connection, “database is locked” errors or interleaved commits are a real risk. A threading.Lock around save() is the simplest fix, and WAL mode (PRAGMA journal_mode=WAL) is the better option once reads and writes overlap. For heavier concurrency, moving to Postgres is where the modular storage layer pays off.
Treating soft blocks (200 responses with a CAPTCHA page) as failures is the part most tutorials skip, and it’s what keeps bad data out of storage.
A few things I’d tweak when taking this to production:
Lock the SQLite writes: With check_same_thread=False and 5 worker threads sharing one connection, concurrent save() calls can collide. A threading.Lock around the write, or a single writer thread fed by a queue, avoids “database is locked” errors.
Handle 407 differently: It usually means a proxy auth problem, not a block. Putting that proxy straight into cooldown is better than burning retries on it.
Reuse clients: Creating a new httpx.Client per attempt skips connection pooling. One client per proxy would speed things up.
Tighten the soft-block check: A plain substring match on “captcha” can false-positive on pages that merely mention it in a script or footer. Combining it with a missing-selector check helps.
Good call on the cooldown instead of deleting proxies too, since IPs often recover. Have you tried a weighted pick based on per-proxy success rate instead of random.choice?
Thanks for the detailed feedback, this is exactly the kind of hardening the tutorial version skips.
You’re right on all four:
SQLite locking: sharing one connection across 5 threads is fine for a demo but risky at scale. A single writer thread fed by a queue is the cleaner fix, and a threading.Lock around save() is the quick one.
407 handling: it’s an auth problem, not a block, so retrying it just wastes attempts. It should go straight to cooldown, or be flagged as misconfigured.
Client reuse: one httpx.Client per proxy keeps connection pooling and cuts per-request overhead.
Soft-block check: a substring match on “captcha” will false-positive on pages that only mention it in a script or footer. Pairing it with a missing-selector check is much more reliable.
On weighted picking, I haven’t benchmarked it against random.choice yet, but it’s the natural next step. I’d track success and failure per proxy over a rolling window, weight by success rate, and keep a small exploration share so recovering proxies still get tested. Have you tried this in your own setup? I’d be curious how much it improved your success rate.
Mayur Upadhyay delivers tech-driven solutions in Web Development, e-commerce development, and mobile app development, helping businesses to grow worldwide.
Quick note for anyone trying this: the soft-block check in Step 4 saved me the most headaches. A 200 response with a CAPTCHA page quietly poisons your data if you don’t catch it.
Curious what everyone else is running into: are you getting hard blocks (403/429), soft blocks (CAPTCHA pages), or JavaScript-rendered pages where plain HTTP just doesn’t work? Drop your target type below and I’ll cover it in a follow-up.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
Soft blocks are the nastiest kind, because a 200 hides the failure and the bad row looks like real data. Your fix is a good one. A row count plus one required-field check per run catches most of these without much code.
Failing the run instead of writing is the key part. A bad batch that never lands is easier to deal with than one you have to clean out of the dataset later. Do you also keep the raw HTML of the failed pages? It makes it quick to tell a captcha from a login wall, and that tells you whether to rotate the proxy or slow down.
Totally agree on the soft-block check. A 200 with a CAPTCHA page is the sneakiest failure because nothing errors out, your pipeline just keeps happily saving garbage. Checking for known CAPTCHA markers and suspiciously small or oddly structured responses before parsing is cheap insurance.
For the follow-up, I’d love to know more about what people are hitting:
Hard blocks (403/429): rate limits, IP bans, or blocks that kick in after a certain request volume?
JS-rendered pages: sites where the data only shows up after scripts run?
Drop your target type (e-commerce, social, travel, listings, etc.) and roughly what scale you’re working at, and I’ll cover the most common one in the next post.
Sidra Jefferi, Marketing Director, uses her knowledge of corporate actions and global markets to help improve reliable internet access in rural and underserved areas.
Separating pool, fetcher, parser and storage makes this easy to extend. A few things I’d harden before running it at scale:
Soft blocks never reach the pool. looks_blocked() returns None in the parser, but by then fetch() has already called report_success(proxy). A proxy that only serves CAPTCHA pages keeps a clean failure count forever. Running the soft-block check inside the fetcher fixes this.
Thread safety. ProxyPool mutates failures and cooldown_until from 5 worker threads with no lock. The same goes for the shared SQLite connection with check_same_thread=False. A threading.Lock around both, or a single writer thread fed by a queue, avoids “database is locked” errors.
407 isn’t a block. It’s a proxy auth failure, so it should pull that proxy out immediately rather than count toward 3 strikes.
Honor Retry-After on 429s instead of a fixed exponential delay.
TLS fingerprint. A Chrome User-Agent over httpx’s TLS handshake is a mismatch some WAFs flag. curl_cffi with browser impersonation closes that gap.
Also, creating a new httpx.Client per attempt drops connection reuse, so you pay a TLS handshake every time. Is a per-proxy client cache worth it in your testing?
Soft blocks: You’re right. report_success() fires before the parser ever sees the CAPTCHA page, so a proxy that only serves challenges never accrues failures. Moving the block check into the fetcher and calling report_failure() from there is the right fix.
Thread safety: Agreed. The pool state and the shared SQLite connection are only safe in the tutorial because the load is light. A lock around both works, but a single writer thread fed by a queue is cleaner at scale.
407: Good catch. It’s an auth failure, not a block, so that proxy should be pulled immediately instead of burning through 3 strikes.
Retry-After: Fair point. Respecting the header on 429s beats a fixed exponential backoff, with the backoff as a fallback when the header is missing.
TLS fingerprint: This is a real gap. A Chrome User-Agent on top of httpx’s handshake is an easy mismatch to flag. curl_cffi with impersonation is the better option for protected targets.
Client per attempt: You’re right that it throws away connection reuse. A per-proxy client cache is the obvious improvement, and I’d expect it to cut latency noticeably. I didn’t benchmark it for the post, so I can’t give you numbers, but it’s worth testing.
I’ll probably cover these in a follow-up on hardening the pipeline. Thanks for taking the time to write it up.
The 3-5 workers point is the one I’d stress most. Cranking up threads usually just burns through the proxy pool faster and spikes the block rate.
Per-proxy success logging is worth doing early too. A single bad proxy can quietly drag down the whole run, and you only spot it once you track failures per IP. Sticky sessions for login and multi-step flows are the right call as well, since rotating mid-session often triggers extra checks.
On your question, I mostly see 429s on high-volume listing pages, and 403s once a site’s fingerprinting kicks in. CAPTCHAs tend to show up when both happen together. What are you running into on your side?
That 429 vs 403 split matches what I see too. 429s on listing pages are usually a rate problem, which fewer workers, jittered delays and a wider pool fix. 403s after fingerprinting are a different issue. Rotating IPs won’t help if the TLS fingerprint and headers still look like a script, so that’s where I’d look at a real browser profile or a client that mimics one.
For my own runs, the blocks mostly show up on paginated search results and product listings, with a few CAPTCHAs once the request volume ramps up. I’m now watching the block rate per proxy rather than overall, which makes it much easier to see whether the problem is the IP or the request pattern.
Since you’re seeing 429s on listing pages, do they start at a consistent request count per IP, or only when concurrency goes up? That would tell us whether to fix pacing or the pool size first, and I’ll cover it in the follow-up.
Agreed on the split. 429s are a pacing problem and 403s are an identity problem, and mixing them up is how people end up burning through proxy pools for nothing. Tracking block rate per proxy is a great call too. It’s the quickest way to tell a bad IP from a bad request pattern.
On your question, I haven’t seen a fixed request count per IP that triggers 429s. It tends to depend on concurrency and how bursty the traffic is, more than on a set number of requests. If the same IP is fine at low concurrency and starts getting 429s when workers go up, that points to pacing. If a fresh IP gets blocked within a few requests, that points to the fingerprint or headers. So I’d fix pacing first (fewer workers, jitter, backoff on 429), then size the pool.
I’ll cover the pacing-vs-pool-size test in the follow-up. If you’ve seen a pattern on your side, I’d like to hear it.
Solid tutorial! The cooldown approach for proxy health management is such an underrated pattern. So many beginner scraping guides just pop() a failing proxy out of the list permanently, which wastes good residential/datacenter IPs that were just temporarily rate-limited.
A few thoughts and additions from building production scraping pipelines:
Watch Out for httpx / requests TLS Fingerprinting
Even with clean residential proxies and updated User-Agent headers, modern Cloudflare / Akamai WAFs will inspect your JA3 / JA4 TLS fingerprint. Standard Python HTTP client libraries (like httpx or requests) use Python's built-in ssl module, which exposes distinct cipher suites and SSL handshakes that instantly flag the request as a script.
Pro-tip: If you find yourself getting instant 403s despite using clean proxies, swap httpx for libraries like curl_cffi or tls-client, which impersonate actual browser TLS signatures.
ThreadPoolExecutor vs. Asyncio (httpx.AsyncClient)
For scaling this past ~5–10 concurrent requests, swapping ThreadPoolExecutor for asyncio with httpx.AsyncClient cuts down on thread overhead and allows much cleaner rate-limiting (e.g., using asyncio.Semaphore per proxy).
Python
Quick snippet for async pool concurrency control
import asyncio
semaphore = asyncio.Semaphore(10) # limit global active connections
async def safe_fetch(url):
async with semaphore:
return await async_fetch(url, pool)
Circuit Breaker Pattern
If your target site goes down completely (500/502/504 errors), your ProxyPool might burn through all available proxy cooldowns trying to retry an endpoint that is fundamentally dead. Adding a global circuit breaker (e.g., if 80% of proxies fail within 60 seconds, pause the entire job for 5 minutes) saves a ton of unnecessary proxy bandwidth/billing.
Thanks for laying out the architecture so cleanly—separating the Fetcher, Pool, and Sanity Parser is definitely the right way to write maintainable scrapers!
permanent pop() is one of those habits that looks fine in a tutorial and then quietly burns through your good IPs in production.
Your three additions are all good ones:
TLS fingerprinting: Agreed, and it’s an easy one to miss. People rotate proxies and User-Agents for hours before realizing the block happens at the handshake. curl_cffi is the cleanest swap since its API is close to requests, so it drops into the Fetcher without much change. I kept the tutorial on httpx to keep the focus on the architecture, but this belongs in a “when you still get 403s” follow-up.
ThreadPoolExecutor vs. asyncio: You’re right that threads work for learning and small jobs but stop scaling well past a handful of concurrent requests. A semaphore per proxy, on top of your global one, would also stop a single IP from getting hammered, which is a nice way to keep the cooldowns from triggering in the first place.
Circuit breaker: This is the missing piece in the pool as written. Without it, a dead target just drains every proxy’s cooldown and your bandwidth for nothing. A failure-rate threshold with a long pause is a much better failure mode than retrying a dead endpoint.
Thanks for the production notes. I may fold the async and circuit breaker ideas into a follow-up. And yes, keeping the Fetcher, Pool, and Parser separate is what makes swaps like curl_cffi painless.
Splitting the pipeline into pool, fetcher, parser and storage makes it easy to swap pieces later. Two parts stood out to me:
A few things I’d add when scaling this up:
Retry-Afteron 429s. When a site sends it, it’s a better signal than a fixed exponential backoff.ThreadPoolExecutor, the failure counters inProxyPooland the shared SQLite connection can race. Athreading.Lockaroundreport_failureand thesavecall avoids odd edge cases.Curious whether you’ve compared block rates between datacenter and residential proxies on the same target. A follow-up with real numbers would be great.
Thanks, glad the separation of pool, fetcher, parser and storage came across. Keeping the pieces swappable was the main goal. The soft-block check and the cooldown logic both came from getting burned in practice. A 200 with a CAPTCHA page in the database is a nasty one because nothing errors out, and I was throwing away proxies that worked fine after a short rest.
Your three additions are all good ones:
threading.Lockaroundreport_failureand the save call is the simplest fix. A per-thread connection or a single writer thread would also work.I haven’t run a clean datacenter vs residential comparison on the same target yet. A follow-up with real block rates, cost per successful request and where each type falls over would be useful, so I’ll consider it. Thanks for the detailed feedback.
Thanks for the detailed reply, and for confirming the Retry-After and per-domain points. The CAPTCHA-behind-a-200 case is the one that bites most people, since a bad row looks exactly like a good one.
I’d definitely read the datacenter vs residential follow-up. Metrics I’d find most useful:
If you run it, test one site with light protection and one with aggressive bot detection, because the gap tends to show up there. Looking forward to it!
The soft-block check in Step 4 is the part most tutorials skip, and parsing a CAPTCHA page as real data is exactly how bad rows end up in a database without anyone noticing.
Two small things I’d add for anyone taking this to production:
Retry-Afteron 429s. If the server tells you how long to wait, using that value instead of a fixed backoff saves a lot of burned retries.Also, a 407 usually means a proxy auth/config problem rather than a blocked IP, so it might be worth logging separately from 403/429.
Curious how you’d handle sticky sessions in this setup. Pin a proxy per session ID in the pool?
Totally agree on the soft-block check. A CAPTCHA page that returns a 200 is the sneakiest failure, because everything looks healthy until someone audits the data.
Your three additions are all solid:
On sticky sessions: yes, pinning a proxy per session ID is how I’d do it. Keep a
session_id → proxymap and reuse that proxy for the whole session (login, cart, pagination). Only reassign it if the proxy gets blocked or hits a failure threshold, and in that case start a fresh session rather than switching IPs mid-session, since the site will often invalidate the cookies anyway. Add a TTL so idle sessions don’t hold proxies forever.Thanks, the session_id → proxy map with a TTL is what I’d go with too. One addition: pin the whole identity to the session, not just the IP. Keep the same User-Agent, headers, and cookie jar for the session’s lifetime, because a stable IP with a changing fingerprint is just as easy to flag.
When a session dies (block, failure threshold, or TTL), retire the entire session object and start a clean one, so no half-valid cookies carry over. It’s also worth logging which session a block happened on and how many requests it had made. That shows you the real per-domain threshold, so you can rotate proactively just before it instead of reacting after the ban.
Good addition, and it fits the TLS fingerprint point from earlier in this thread. A stable IP with a changing fingerprint is just as easy to flag as the reverse, so the session should own the whole identity, not only the proxy.
curl_cffi, the impersonation profile should be pinned too, so the User-Agent never disagrees with the TLS handshake.For proactive rotation, I’d rotate at a margin below the observed threshold (say, around 70-80% of the typical block point) rather than right at it. Thresholds drift with concurrency, time of day and site changes, so a hard cutoff will eventually be wrong. I haven’t benchmarked this myself, so treat those numbers as a starting point to tune against your own logs.
I’ll fold this into the session-handling section of the hardening follow-up. Thanks for the detailed addition.
Keeping the pool, fetcher, parser, and storage separate makes this easy to extend. Putting proxies in a cooldown instead of deleting them is a good touch, since IPs do recover.
A few things I’d tweak if you take this to production:
proxy. resp.raise_for_status()raises httpx.HTTPStatusError, which theexcept httpx.HTTPErrorbranch catches. A plain 404 or 410 therefore penalizes a healthy proxy and triggers pointless retries. I’d return early on those, or retry only on the block codes and 5xx.Retry-Afteron 429s. If the server tells you how long to wait, using that value beats a generic exponential backoff.ThreadPoolExecutor, several workers updatefailuresandcooldown_untilat once. Athreading.Lockaroundreport_failureorgetavoids odd race conditions.httpxandrequestshave a recognizable TLS/JA3 fingerprint.curl_cffiwith browser impersonation often fixes this without a headless browser.On your question about which targets cause trouble: Cloudflare-protected sites are the ones I hit most, and the TLS fingerprint was usually the culprit rather than the IP.
Would you consider a follow-up on async (
httpx.AsyncClientplus semaphores) for the same architecture?You’re right about the 404s. As written,
raise_for_status()feeds every HTTP error into the proxy penalty path, so a dead URL gets blamed on a healthy proxy and retried for nothing. The fix is to return early on 404/410 and only retry on the block codes (403, 429) and 5xx.Respecting
Retry-Afteron 429s is also better than blind exponential backoff. I’d parse it, fall back to backoff when the header is missing, and cap the wait so one bad response can’t stall a worker.On soft-block detection, keyword matching is the weakest part of the example. A reCAPTCHA script on a normal login page would trip it. Checking for an expected selector plus a minimum content length is much more reliable, and I’ll add that as the recommended approach.
The thread-safety point is fair too. With
ThreadPoolExecutor,failuresandcooldown_untilare shared state, so athreading.Lockaroundgetandreport_failureis the right call for production.The TLS fingerprinting note is the one I most want to work into the post. It explains why some sites keep returning 403 even with clean proxies and good headers.
curl_cffiwith browser impersonation is a lighter fix than jumping straight to a headless browser. And since your experience with Cloudflare-protected sites points to the fingerprint rather than the IP, that’s a useful pointer for anyone debugging the same thing.An async follow-up with
httpx.AsyncClientand semaphores is a good idea, and the pool/fetcher/parser/storage split should carry over cleanly. I’ll plan on it, and I’ll fold your fixes into the production section.Nice walkthrough. Splitting it into pool, fetcher, parser and storage makes it easy to swap parts without rewriting everything. Putting proxies in a cooldown instead of deleting them is a good detail, since IPs often recover.
One thing I’d like to see in a follow-up: right now
looks_blocked()runs in the parser, but the proxy pool never hears about it. A CAPTCHA page that comes back as a 200 still counts asreport_success, so a burned proxy stays in rotation. Feeding soft blocks back into the pool’s failure count would make the health tracking more accurate.Two small things I noticed:
httpx.Clienton every attempt skips connection reuse, which adds up at scale.report_failure, so dead URLs can push healthy proxies into cooldown.For me the hard part is sites that fingerprint TLS, where good proxies still get blocked with a clean header set. Are you planning to cover that, or the Playwright fallback, next?
The soft-block gap is the real flaw.
looks_blocked()lives in the parser, so a CAPTCHA page that returns a 200 still callsreport_successand the burned proxy stays in rotation. The fix is to run that check inside the fetcher (or return a signal the fetcher can act on) and callreport_failureon a soft block, so the pool’s health tracking matches what the target is really doing.The 404 point is also fair.
raise_for_status()throws anHTTPError, which lands in the sameexceptbranch, so dead URLs count against healthy proxies. 404s and 410s should skipreport_failureand return early. And a singlehttpx.Clientper proxy, reused across attempts, would give you connection reuse instead of a new handshake every time.On TLS fingerprinting, you’ve hit the limit of this setup. Proxies only change the IP, and plain
httpxstill presents a very recognizable TLS signature, so good IPs plus clean headers can still get blocked. Two usual ways out arecurl_cffiwith browser impersonation, or a real browser via Playwright. I’ll cover both in a follow-up, along with the soft-block feedback loop and the fixes above.Separating the pool, fetcher, parser and storage means you can swap a proxy provider or move from SQLite to Postgres without touching the rest. I especially liked the cooldown approach in
ProxyPool: putting a proxy in timeout aftermax_failuresinstead of deleting it fits how rotating IPs behave, since many recover after a rest. The soft-block check in the parser is also the right call. A 200 response with a CAPTCHA page is the failure that quietly corrupts a dataset, and returningNonebefore storage keeps that out.Two things I’d consider adding. First, honor the
Retry-Afterheader on 429 responses instead of a fixed exponential backoff, and weight proxy selection by recent success rate rather thanrandom.choice, so a degrading IP gets picked less before it hits the cooldown threshold. Second, withcheck_same_thread=Falseand five workers sharing one SQLite connection, you may hit “database is locked” errors or interleaved commits. Athreading.Lockaroundsave(), or WAL mode(PRAGMA journal_mode=WAL), would make it safer under concurrency. Great guide overall!Thanks for the detailed read, really glad the separation of pool, fetcher, parser and storage came across as intended. The cooldown in ProxyPool was a deliberate choice for exactly the reason you gave: rotating IPs often recover after a rest, so permanent removal wastes good capacity.
Both suggestions are fair. Honoring Retry-After on 429s is more respectful to the target and more efficient than blind exponential backoff, and weighting proxy selection by recent success rate would let a degrading IP fade out before it hits the cooldown threshold instead of after.
You’re also right about the SQLite setup. With check_same_thread=False and five workers on one connection, “database is locked” errors or interleaved commits are a real risk. A threading.Lock around save() is the simplest fix, and WAL mode (PRAGMA journal_mode=WAL) is the better option once reads and writes overlap. For heavier concurrency, moving to Postgres is where the modular storage layer pays off.
Treating soft blocks (200 responses with a CAPTCHA page) as failures is the part most tutorials skip, and it’s what keeps bad data out of storage.
A few things I’d tweak when taking this to production:
check_same_thread=Falseand 5 worker threads sharing one connection, concurrentsave()calls can collide. Athreading.Lockaround the write, or a single writer thread fed by a queue, avoids “database is locked” errors.httpx.Clientper attempt skips connection pooling. One client per proxy would speed things up.Good call on the cooldown instead of deleting proxies too, since IPs often recover. Have you tried a weighted pick based on per-proxy success rate instead of
random.choice?Thanks for the detailed feedback, this is exactly the kind of hardening the tutorial version skips.
You’re right on all four:
threading.Lockaroundsave()is the quick one.httpx.Clientper proxy keeps connection pooling and cuts per-request overhead.On weighted picking, I haven’t benchmarked it against
random.choiceyet, but it’s the natural next step. I’d track success and failure per proxy over a rolling window, weight by success rate, and keep a small exploration share so recovering proxies still get tested. Have you tried this in your own setup? I’d be curious how much it improved your success rate.Quick note for anyone trying this: the soft-block check in Step 4 saved me the most headaches. A 200 response with a CAPTCHA page quietly poisons your data if you don’t catch it.
Curious what everyone else is running into: are you getting hard blocks (403/429), soft blocks (CAPTCHA pages), or JavaScript-rendered pages where plain HTTP just doesn’t work? Drop your target type below and I’ll cover it in a follow-up.
mostly soft blocks on our side, a 200 that is really a login or captcha page and the parser stores it as a row
what helped was counting rows and checking one required field per run, and failing the run when either looks off instead of writing it
Soft blocks are the nastiest kind, because a 200 hides the failure and the bad row looks like real data. Your fix is a good one. A row count plus one required-field check per run catches most of these without much code.
Failing the run instead of writing is the key part. A bad batch that never lands is easier to deal with than one you have to clean out of the dataset later. Do you also keep the raw HTML of the failed pages? It makes it quick to tell a captcha from a login wall, and that tells you whether to rotate the proxy or slow down.
Totally agree on the soft-block check. A 200 with a CAPTCHA page is the sneakiest failure because nothing errors out, your pipeline just keeps happily saving garbage. Checking for known CAPTCHA markers and suspiciously small or oddly structured responses before parsing is cheap insurance.
For the follow-up, I’d love to know more about what people are hitting:
Drop your target type (e-commerce, social, travel, listings, etc.) and roughly what scale you’re working at, and I’ll cover the most common one in the next post.
Separating pool, fetcher, parser and storage makes this easy to extend. A few things I’d harden before running it at scale:
looks_blocked()returnsNonein the parser, but by thenfetch()has already calledreport_success(proxy). A proxy that only serves CAPTCHA pages keeps a clean failure count forever. Running the soft-block check inside the fetcher fixes this.ProxyPoolmutatesfailuresandcooldown_untilfrom 5 worker threads with no lock. The same goes for the shared SQLite connection withcheck_same_thread=False. Athreading.Lockaround both, or a single writer thread fed by a queue, avoids “database is locked” errors.Retry-Afteron 429s instead of a fixed exponential delay.Also, creating a new
httpx.Clientper attempt drops connection reuse, so you pay a TLS handshake every time. Is a per-proxy client cache worth it in your testing?Thanks, this is a really useful list.
report_success()fires before the parser ever sees the CAPTCHA page, so a proxy that only serves challenges never accrues failures. Moving the block check into the fetcher and callingreport_failure()from there is the right fix.curl_cffiwith impersonation is the better option for protected targets.I’ll probably cover these in a follow-up on hardening the pipeline. Thanks for taking the time to write it up.
A few quick tips if you’re testing this on a real target:
What are you scraping at the moment, and are you seeing 403s, 429s, or CAPTCHA pages? Share it here, and I’ll try to cover it in a follow-up.
The 3-5 workers point is the one I’d stress most. Cranking up threads usually just burns through the proxy pool faster and spikes the block rate.
Per-proxy success logging is worth doing early too. A single bad proxy can quietly drag down the whole run, and you only spot it once you track failures per IP. Sticky sessions for login and multi-step flows are the right call as well, since rotating mid-session often triggers extra checks.
On your question, I mostly see 429s on high-volume listing pages, and 403s once a site’s fingerprinting kicks in. CAPTCHAs tend to show up when both happen together. What are you running into on your side?
That 429 vs 403 split matches what I see too. 429s on listing pages are usually a rate problem, which fewer workers, jittered delays and a wider pool fix. 403s after fingerprinting are a different issue. Rotating IPs won’t help if the TLS fingerprint and headers still look like a script, so that’s where I’d look at a real browser profile or a client that mimics one.
For my own runs, the blocks mostly show up on paginated search results and product listings, with a few CAPTCHAs once the request volume ramps up. I’m now watching the block rate per proxy rather than overall, which makes it much easier to see whether the problem is the IP or the request pattern.
Since you’re seeing 429s on listing pages, do they start at a consistent request count per IP, or only when concurrency goes up? That would tell us whether to fix pacing or the pool size first, and I’ll cover it in the follow-up.
Agreed on the split. 429s are a pacing problem and 403s are an identity problem, and mixing them up is how people end up burning through proxy pools for nothing. Tracking block rate per proxy is a great call too. It’s the quickest way to tell a bad IP from a bad request pattern.
On your question, I haven’t seen a fixed request count per IP that triggers 429s. It tends to depend on concurrency and how bursty the traffic is, more than on a set number of requests. If the same IP is fine at low concurrency and starts getting 429s when workers go up, that points to pacing. If a fresh IP gets blocked within a few requests, that points to the fingerprint or headers. So I’d fix pacing first (fewer workers, jitter, backoff on 429), then size the pool.
I’ll cover the pacing-vs-pool-size test in the follow-up. If you’ve seen a pattern on your side, I’d like to hear it.
Solid tutorial! The cooldown approach for proxy health management is such an underrated pattern. So many beginner scraping guides just pop() a failing proxy out of the list permanently, which wastes good residential/datacenter IPs that were just temporarily rate-limited.
A few thoughts and additions from building production scraping pipelines:
Pro-tip: If you find yourself getting instant 403s despite using clean proxies, swap httpx for libraries like curl_cffi or tls-client, which impersonate actual browser TLS signatures.
Python
Quick snippet for async pool concurrency control
import asyncio
semaphore = asyncio.Semaphore(10) # limit global active connections
async def safe_fetch(url):
async with semaphore:
return await async_fetch(url, pool)
Thanks for laying out the architecture so cleanly—separating the Fetcher, Pool, and Sanity Parser is definitely the right way to write maintainable scrapers!
permanent
pop()is one of those habits that looks fine in a tutorial and then quietly burns through your good IPs in production.Your three additions are all good ones:
TLS fingerprinting: Agreed, and it’s an easy one to miss. People rotate proxies and User-Agents for hours before realizing the block happens at the handshake.
curl_cffiis the cleanest swap since its API is close torequests, so it drops into the Fetcher without much change. I kept the tutorial onhttpxto keep the focus on the architecture, but this belongs in a “when you still get 403s” follow-up.ThreadPoolExecutor vs. asyncio: You’re right that threads work for learning and small jobs but stop scaling well past a handful of concurrent requests. A semaphore per proxy, on top of your global one, would also stop a single IP from getting hammered, which is a nice way to keep the cooldowns from triggering in the first place.
Circuit breaker: This is the missing piece in the pool as written. Without it, a dead target just drains every proxy’s cooldown and your bandwidth for nothing. A failure-rate threshold with a long pause is a much better failure mode than retrying a dead endpoint.
Thanks for the production notes. I may fold the async and circuit breaker ideas into a follow-up. And yes, keeping the Fetcher, Pool, and Parser separate is what makes swaps like
curl_cffipainless.this is great awesome, have you tried proxychain?