There’s one edge case I’d pay extra attention to in the GitHub fallback: concurrency.
If fetchRepo() eventually runs those four metadata requests through Promise.all, mutating a shared headers object after the first 401 creates a race. Some requests may already have left with the expired token. I’d keep request headers immutable and make the retry decision at the individual request level: retry that public GET once without authorization, rather than changing shared state while other requests are in flight.
There’s also a scaling detail worth considering. GitHub’s unauthenticated REST API limit is 60 requests per hour per originating IP. On a serverless deployment, multiple users can end up sharing outbound IP capacity, so a fallback that works perfectly during development might become a rate-limit bottleneck under real traffic.
I’d also make the Haiku fail-open path observable. Returning a valid report is the right availability decision, but a fallback counter or structured warning would make sure repeated AI failures don't quietly become the new normal.
The interesting part of this post-mortem is that none of these issues requires a major architectural failure. Small assumptions around shared state, credentials, and observability are enough to change production behavior.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Thank you for laying this out so clearly. Deciding the retry per request is the cleaner design, and Mikhail has a PR open that does it: each request remembers whether it carried the token and retries its own 401. It doesn’t make the headers immutable, which is the fuller version of your point.
The shared IP detail is the one I hadn’t sat with. Scans end up unauthenticated now, so 60 calls an hour is the ceiling, and on Vercel that address isn’t only mine. Fine for a hackathon tool with a handful of visitors. Not fine if it ever gets busy.
And yes to making the fallback countable. Each Haiku failure writes a reason code to the receipt, but nothing adds them up. Small assumptions was the whole post, and you named it better than I did.
That makes sense. The part I'd look at next is how these limits interact under concurrent scans. Even if each request handles its own 401 correctly, multiple scans can still consume the same unauthenticated rate-limit budget independently.
For example, imagine 8 users triggering scans at roughly the same time, with each scan making 4 GitHub API calls. That's 32 requests competing for the same outbound IP quota, and if several Vercel functions run independently, they have no shared view of how much quota is left.
A shared rate-limit coordinator could help here. Redis could track the remaining quota and reset timestamp across instances, with a distributed lock or atomic counter to prevent multiple workers from making decisions based on stale quota information. Requests could then be queued or delayed before GitHub starts returning 403 or 429 responses.
Same idea for the Haiku fallback: having reason codes in receipts is a good start, but tracking their frequency over time would help distinguish occasional provider instability from a recurring failure pattern. I'd also separate failures by provider, model and error type to make the metrics actionable.
I think that's where the interesting engineering challenge starts: not just making each request recover, but making the whole system behave predictably under load.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
This is the right next problem, and it’s bigger than the tool is today. It’s a hackathon project, so I’d hit GitHub’s limit long before I needed a coordinator. If it ever gets traffic, a shared counter in Redis is where I’d start. Counting fallback reasons by provider and error type is the part I can use now. “Behave predictably under load” is the line I’m keeping.
One detail I'd be careful with when you eventually add Redis: make the counter an atomic reservation mechanism, not just a shared value that each Vercel function reads before starting its requests.
For example, if a scan needs four GitHub metadata calls, a Lua script could reserve all four slots in one operation. If the available budget is below four, the scan could wait or use a fallback rather than launching a batch that's likely to hit the limit halfway through. I'd key that budget by the actual egress identity and GitHub rate-limit resource, with a safety margin for requests outside the app.
Then reconcile the estimate using X-RateLimit-Remaining and X-RateLimit-Reset from responses. One subtle point: GitHub processes requests across multiple regions, so rate-limit headers can vary between responses. I'd use them for pacing and backoff, not as a perfectly synchronized global counter. Secondary limits also need separate cooldown handling through Retry-After when present, rather than assuming every 403 means the hourly quota is exhausted.
I wouldn't build all of this for a hackathon project today either. But when traffic justifies it, atomic reservation plus response-based reconciliation is a much safer starting point than a plain Redis increment.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Thank you, this is really helpful, and the “reserve all four at once” idea makes a lot of sense to me. Starting a scan you can’t finish is the same kind of half-done answer I’ve been trying to avoid. I’m not building it yet, but I’ve saved this for the day traffic calls for it. I appreciate you sticking with this thread!
This is a fantastic breakdown of the "hidden gaps" problem. When you have multiple services communicating, the real bugs rarely live in the code itself—they live in the assumptions of how the state flows between them.
As someone who builds complex systems with AI-assisted workflows, I've learned that normal unit tests aren't enough. You need strict invariants (rules that must always be true, like "a payment must never process without a ledger entry") and a dedicated verifier process to check them.
I'm curious: did the verifiers catch a state sync issue between the services, or was it a silent failure inside one of the external APIs? Thanks for sharing this—it's a great reality check for anyone building microservices.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Neither, actually. Nothing went wrong between the services. Two keys aged out around the code: the 30-day GitHub token expired, and my code treated that as fatal even for public repos that never needed it. The Anthropic key had to be rotated too. The one real bug inside a verifier was a substring match (“ava” hiding inside “javascript”), and ManSio caught that one, not me. Your invariants point lands, though. “A scan never fails just because an optional token is bad” is a rule I wish I’d written down sooner.
This is the ultimate "production reality check." The "ava" hiding inside "javascript" is a legendary regex trap—it perfectly highlights why simple substring matching is so dangerous. Word boundaries (\b) are your best friend in those verifiers, otherwise "scunthorpe" problems will haunt you forever.
And the expired GitHub token crashing the entire scan for public repos is the classic "fail-hard" anti-pattern. You just articulated exactly why I love your invariants approach: "Systems must degrade gracefully." If an optional authentication fails, the core function should still run.
You hit the nail on the head with that rule. "A scan never fails just because an optional token is bad" is exactly the kind of thing that needs to be written down in a README before a hackathon, not learned a month later. Thanks for the brutal honesty—this is a masterclass in why strict checks and graceful failures are the difference between a demo and a product.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Thank you, Bogdan. One small twist on word boundaries: plain \b treats a hyphen as a boundary, so “lodash” would still match inside “lodash-es”. ManSio’s fix uses custom lookarounds so hyphens and slashes count as part of the name. Scunthorpe problems really are everywhere. And agreed, that rule belongs in the README before day one, not a month later.
Spot on, Earl. You just exposed the classic trap of \b—it completely ignores hyphens because they are non-word characters. So lodash will happily match inside lodash-es or lodash/fp.
Using custom lookarounds like (?<![\w/-])lodash(?![\w/-]) is definitely the bulletproof way to handle package names and URLs. It forces the regex to treat the entire slug as a single token.
And you're 100% right about the Scunthorpe problem. It's the ultimate proof that naive regex will always bite you in production. This is exactly the kind of edge case that deserves a place in the README (or better yet, in a unit test) before you write a single line of validation logic. Great insight!
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The expired-token bit genuinely surprised me — that GitHub happily serves public repos with no token at all, but a dead one gets you a 401 and kills the whole scan before Jev or Haiku even run. I'd have assumed an expired token just degrades to anonymous, not that it actively blocks you. Your retry-without-the-header fix makes sense, though Reid's point about Promise.all has me wondering how long that shared-headers trick holds up once those four calls go parallel.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Same! I assumed a dead token would get treated like no token. Nope. And Reid was right about the parallel calls. Mikhail already has a PR up for it, with a test for that exact pair. This comment section is doing better code review than I planned for lol.
That’s a really important distinction, especially the point about the first request running independently before the parallel requests begin. It shows why understanding the actual execution flow matters just as much as understanding the fallback logic itself.
I also think this discussion highlights something broader about AI-assisted development: getting the code to work is only one part of building a reliable application. The real challenge is making sure the assumptions behind that code remain valid when credentials expire, dependencies become unavailable, requests overlap, or traffic increases unexpectedly.
What I particularly like about Mikhail’s per-request retry approach is that it addresses the concurrency problem without introducing unnecessary complexity into the overall architecture. Each request takes responsibility for its own recovery, which makes the behavior more predictable and easier to test.
One additional improvement I'd consider is turning these lessons into explicit regression tests and operational alerts. The tests protect against the same edge cases returning during future changes, while alerts help identify when a supposedly occasional fallback starts happening consistently in production.
I also agree that recording a Haiku failure in the scan receipt is useful, but there's a difference between storing diagnostic information and actively monitoring it. A failure reason that nobody reviews can remain hidden just as easily as an expired API key.
For me, the biggest takeaway from this entire discussion is that resilience should be designed into the normal execution path, not added only after something breaks.
And honestly, this is what makes post-hackathon debugging so valuable. The original build proves that the idea works; real-world failures reveal whether the engineering decisions can support it over time. Great discussion, and credit to everyone who contributed practical fixes rather than just pointing out problems.
A technologist working in automation. Here I write up what I actually learn building tools, engineering AI agents, and keeping systems running, along with the workflows and habits that stick.
Dropping headers.authorization on a 401 works cleanly when fetchRepo() runs calls sequentially, though if those four metadata requests ever run under Promise.all, all four already go out with the expired header before the first 401 returns.
The Vercel scope trap pairs awkwardly with fail-open Haiku summaries too. Because the UI still renders a fallback line when the key returns 401 in Production, an expired or mis-scoped key stays invisible unless the catch block emits a warning log or counter.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Good catch, and you were right. The first call runs alone, so an expired token gets caught before the parallel pair goes out. A token that dies mid-scan is a different story: both parallel calls come back 401 and only one gets its retry. Mikhail opened a PR for exactly that about an hour after your comment. Each request now remembers whether it carried the token and retries its own 401.
On fail-open: each Haiku failure does write a reason code to the scan receipt. Nothing taps me on the shoulder, though, which is how a dead key sat there for a month.
Civil engineer turned self-taught builder. I direct AI agents to write code while I do architecture, QA, and decisions. Currently building MSCodeBase Intelligence, an MCP server for codebase-aware AI
Thanks for the shoutout, Earl — really appreciate the credit!
Loved reading the honest post-mortem. Seeing an expired token, a dev-scoped key, and a hidden substring match all line up after a month is such a classic post-hackathon moment. Glad the mentionsWord fix helped, and it's awesome to see the scanner back to green!
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Thank you for the fix, and for three more PRs today! The per-request retry is the exact thing Reid and Sina were circling. Reading them now. Your name’s in the README where it belongs.
Relatable postmortem — nothing actually regressed in the verifiers; the failure was external credential and environment drift: an expired 30-day GitHub token and a Haiku key scoped only to Development in Vercel, both masked locally during the demo and surfaced only in deployment. That argues for putting token expiry and key scope into startup health checks and logs, rather than assuming a read-only public-repo tool stays authenticated. Your split of responsibilities is the right boundary too: deterministic rules settle what's true, models only propose buckets, so a flaky model can't corrupt the verdict.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Startup checks for token expiry and key scope would have caught both of these on day one. Cheaper than my method, which was finding out a month later lol. And thank you on the split. A flaky model not being able to touch the verdict is why Haiku failing didn’t break the report.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
That’s a different kind of build than mine. Models only label and summarize here, and plain code makes the calls. What broke wasn’t latency. It was an expired token and a key in the wrong environment.
The ava hiding inside javascript detail is the one that stuck with me. It is such a quiet failure mode, a check that looks correct until a short name shows up inside a longer word. The expired token story landed the same way for me. I had always assumed a dead token would behave like no token, and learning that GitHub treats an expired header as a hard rejection explains a lot of confusing 401s. Thanks for writing this up as a postmortem instead of just fixing it quietly.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
The ava inside javascript one still makes me laugh and wince. Credit to Mikhail for catching it. I had the same assumption about dead tokens. Writing it up beat fixing it quietly, because the comments here ended up improving the fix.
What I like about this post-mortem is that nothing in the verifiers was wrong. All three failures lived around the code: a token with a 30-day life, a key scoped to the wrong environment, and a substring check that only misbehaved on a real repo. The suggestion from mp3sss to put token expiry and key scope into a startup check is probably the cheapest fix on the list.
Environment drift is something we see a lot when rebuilding deploy pipelines at Ascendra Ventures. On one WordPress migration, the old setup was a single EC2 box, deploys were done by copying theme files over SSH, and editors entered every content change twice. The rebuild baked everything into an immutable image on Cloud Run, pulled license keys from Secret Manager instead of the machine, redeployed staging on every push and kept production behind an approval. Staging and production stopped being two hand-kept places that could quietly drift apart.
Would a scheduled weekly scan against one known public repo have caught the expired token before a user did?
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Probably yes. The token expired on a 30-day clock and only failed on a real call, so a weekly scan against one known public repo would have hit it before a user did. It’s the cheapest kind of canary. Your point about staging and production quietly drifting is the bigger lesson for me. Nothing in my code broke. Everything around it aged like milk. That's the thing with these hackathon builds, they die silently. So now, the next question, do I break them down after judging or let them live on?
A README checker tripping over a substring is a little poetic, since READMEs are full of long URLs and a URL is basically lowercase words glued together. Would stripping link targets before the coverage check catch more of these, or do you want URLs to count as mentions on purpose?
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
Turns out ManSio’s whole-token fix already covers it: a name right after a / never counts, so URL paths are out. Good nudge to add tests that keep it that way.
The ava-inside-javascript example makes the verifier bug easy to understand. I’d keep the token-expiry failure separate from the matching fixture so a future failing scan points to the right layer. For interview practice, this failing-test exercise gives a related challenge: isolate one failing behavior and explain why the smallest fix belongs there.
This is a brilliant breakdown of the 'hidden gaps' problem. As someone who just built a system entirely from scratch without any frameworks (pure vanilla PHP), I felt this deeply.
Without a framework, you literally have to build your own invariants. When I launched my free webmaster tool directory, I learned the hard way that normal validation isn't enough—bots bypassed my basic checks and flooded the system. I had to rewrite the state flow and manually implement OAuth just to ensure a 'submission' always came from a real human.
It's painful, but it forces you to truly understand every single assumption in your architecture. Great write-up!
Developer focused on building scalable web applications, AI-powered solutions, and modern cloud technologies.
Passionate about clean code, problem-solving, automation, and creating practical solutions
Great write-up! It's amazing how small things like expired 30-day tokens, environment variable scoping in Vercel, and simple substring checks (javascript hiding ava) can break an entire app after a hackathon. Also, regarding the deployment side, I checked the status using. you can check if Vercel is down for me there), and it looks like broader infrastructure and service issues are sometimes adding to these deployment and API verification hiccups right now. Thanks for sharing these valuable debugging field notes!
One area I would pay particular attention to in this implementation is how the GitHub fallback behaves under concurrent requests, because this is exactly the kind of detail that can look harmless during development but create unpredictable behavior in production.
Consider what happens when "fetchRepo()" executes four metadata requests concurrently using "Promise.all". If those requests share a mutable headers object and one of them receives a "401 Unauthorized" response, modifying that shared object to remove the expired authorization token can introduce a race condition.
At that point, some requests may already have been dispatched with the expired token, while others may observe the modified headers. This makes the fallback behavior dependent on timing rather than an explicit and predictable request policy.
I would avoid mutating shared request configuration altogether. Instead, each request should receive its own immutable headers object, and the authentication fallback should be handled independently at the individual request level.
For example, when a public GitHub GET request receives a "401", retry that specific request once without the authorization header. This keeps the retry logic isolated, prevents one request from influencing another, and makes the overall behavior much easier to reason about, test, and maintain.
There is another important consideration here: GitHub's unauthenticated REST API rate limit.
Unauthenticated requests are generally limited to 60 requests per hour per originating IP address. That might seem sufficient when testing locally, especially when traffic is low and requests are infrequent. However, the situation changes significantly in a serverless environment.
Multiple users and separate application instances can share the same outbound IP capacity. If the authenticated path starts failing and the application consistently falls back to unauthenticated requests, the fallback mechanism itself could become a source of rate-limit failures.
In other words, a fallback designed to improve availability could unintentionally introduce another reliability problem when traffic increases.
I would therefore treat the fallback as a controlled recovery mechanism rather than an unlimited alternative. It is worth considering how frequently it can be triggered, how rate-limit responses are handled, whether retries are bounded, and whether the application can expose enough information to distinguish authentication failures from rate-limit exhaustion.
The Haiku fail-open behavior deserves similar attention.
Returning a valid report when the AI service is unavailable is absolutely the right availability decision. A temporary dependency failure should not necessarily prevent users from receiving useful application output.
However, fail-open behavior should never mean fail-silently behavior.
I would introduce structured logging and a few lightweight metrics around this path. For example:
Count how often the Haiku fallback is activated.
Record AI request failures and relevant response categories.
Track how frequently GitHub authentication fallback occurs.
Distinguish successful fallback responses from complete request failures.
These signals would help identify whether an occasional dependency failure is being handled correctly or whether a recurring upstream problem is gradually becoming the application's normal operating mode.
I would also make sure that retries are bounded and that fallback behavior is covered by tests for concurrent requests, expired credentials, rate limiting, and AI service unavailability.
What makes this particularly interesting is that none of these concerns requires a major architectural redesign. The underlying approach can be perfectly reasonable, yet a few small implementation assumptions can have a disproportionate impact on production reliability.
Shared mutable state, credential handling, external API quotas, and insufficient observability are all relatively small details in isolation. But when they interact under real traffic, they can produce failures that are difficult to reproduce and even harder to diagnose.
My broader takeaway is that reliability is not just about having a fallback for when a dependency fails. It is about making sure the fallback itself is isolated, bounded, observable, and safe under concurrency.
A system is not truly resilient simply because it can recover from one failure. It is resilient when that recovery mechanism does not quietly create the next failure.
These are exactly the kinds of edge cases worth addressing before an implementation moves from working in development to handling unpredictable production traffic.
We're a place where coders share, stay up-to-date and grow their careers.
There’s one edge case I’d pay extra attention to in the GitHub fallback: concurrency.
If
fetchRepo()eventually runs those four metadata requests throughPromise.all, mutating a shared headers object after the first401creates a race. Some requests may already have left with the expired token. I’d keep request headers immutable and make the retry decision at the individual request level: retry that publicGETonce without authorization, rather than changing shared state while other requests are in flight.There’s also a scaling detail worth considering. GitHub’s unauthenticated REST API limit is 60 requests per hour per originating IP. On a serverless deployment, multiple users can end up sharing outbound IP capacity, so a fallback that works perfectly during development might become a rate-limit bottleneck under real traffic.
I’d also make the Haiku fail-open path observable. Returning a valid report is the right availability decision, but a fallback counter or structured warning would make sure repeated AI failures don't quietly become the new normal.
The interesting part of this post-mortem is that none of these issues requires a major architectural failure. Small assumptions around shared state, credentials, and observability are enough to change production behavior.
Thank you for laying this out so clearly. Deciding the retry per request is the cleaner design, and Mikhail has a PR open that does it: each request remembers whether it carried the token and retries its own 401. It doesn’t make the headers immutable, which is the fuller version of your point.
The shared IP detail is the one I hadn’t sat with. Scans end up unauthenticated now, so 60 calls an hour is the ceiling, and on Vercel that address isn’t only mine. Fine for a hackathon tool with a handful of visitors. Not fine if it ever gets busy.
And yes to making the fallback countable. Each Haiku failure writes a reason code to the receipt, but nothing adds them up. Small assumptions was the whole post, and you named it better than I did.
That makes sense. The part I'd look at next is how these limits interact under concurrent scans. Even if each request handles its own 401 correctly, multiple scans can still consume the same unauthenticated rate-limit budget independently.
For example, imagine 8 users triggering scans at roughly the same time, with each scan making 4 GitHub API calls. That's 32 requests competing for the same outbound IP quota, and if several Vercel functions run independently, they have no shared view of how much quota is left.
A shared rate-limit coordinator could help here. Redis could track the remaining quota and reset timestamp across instances, with a distributed lock or atomic counter to prevent multiple workers from making decisions based on stale quota information. Requests could then be queued or delayed before GitHub starts returning 403 or 429 responses.
Same idea for the Haiku fallback: having reason codes in receipts is a good start, but tracking their frequency over time would help distinguish occasional provider instability from a recurring failure pattern. I'd also separate failures by provider, model and error type to make the metrics actionable.
I think that's where the interesting engineering challenge starts: not just making each request recover, but making the whole system behave predictably under load.
This is the right next problem, and it’s bigger than the tool is today. It’s a hackathon project, so I’d hit GitHub’s limit long before I needed a coordinator. If it ever gets traffic, a shared counter in Redis is where I’d start. Counting fallback reasons by provider and error type is the part I can use now. “Behave predictably under load” is the line I’m keeping.
One detail I'd be careful with when you eventually add Redis: make the counter an atomic reservation mechanism, not just a shared value that each Vercel function reads before starting its requests.
For example, if a scan needs four GitHub metadata calls, a Lua script could reserve all four slots in one operation. If the available budget is below four, the scan could wait or use a fallback rather than launching a batch that's likely to hit the limit halfway through. I'd key that budget by the actual egress identity and GitHub rate-limit resource, with a safety margin for requests outside the app.
Then reconcile the estimate using
X-RateLimit-RemainingandX-RateLimit-Resetfrom responses. One subtle point: GitHub processes requests across multiple regions, so rate-limit headers can vary between responses. I'd use them for pacing and backoff, not as a perfectly synchronized global counter. Secondary limits also need separate cooldown handling throughRetry-Afterwhen present, rather than assuming every403means the hourly quota is exhausted.I wouldn't build all of this for a hackathon project today either. But when traffic justifies it, atomic reservation plus response-based reconciliation is a much safer starting point than a plain Redis increment.
Thank you, this is really helpful, and the “reserve all four at once” idea makes a lot of sense to me. Starting a scan you can’t finish is the same kind of half-done answer I’ve been trying to avoid. I’m not building it yet, but I’ve saved this for the day traffic calls for it. I appreciate you sticking with this thread!
This is a fantastic breakdown of the "hidden gaps" problem. When you have multiple services communicating, the real bugs rarely live in the code itself—they live in the assumptions of how the state flows between them.
As someone who builds complex systems with AI-assisted workflows, I've learned that normal unit tests aren't enough. You need strict invariants (rules that must always be true, like "a payment must never process without a ledger entry") and a dedicated verifier process to check them.
I'm curious: did the verifiers catch a state sync issue between the services, or was it a silent failure inside one of the external APIs? Thanks for sharing this—it's a great reality check for anyone building microservices.
Neither, actually. Nothing went wrong between the services. Two keys aged out around the code: the 30-day GitHub token expired, and my code treated that as fatal even for public repos that never needed it. The Anthropic key had to be rotated too. The one real bug inside a verifier was a substring match (“ava” hiding inside “javascript”), and ManSio caught that one, not me. Your invariants point lands, though. “A scan never fails just because an optional token is bad” is a rule I wish I’d written down sooner.
This is the ultimate "production reality check." The "ava" hiding inside "javascript" is a legendary regex trap—it perfectly highlights why simple substring matching is so dangerous. Word boundaries (\b) are your best friend in those verifiers, otherwise "scunthorpe" problems will haunt you forever.
And the expired GitHub token crashing the entire scan for public repos is the classic "fail-hard" anti-pattern. You just articulated exactly why I love your invariants approach: "Systems must degrade gracefully." If an optional authentication fails, the core function should still run.
You hit the nail on the head with that rule. "A scan never fails just because an optional token is bad" is exactly the kind of thing that needs to be written down in a README before a hackathon, not learned a month later. Thanks for the brutal honesty—this is a masterclass in why strict checks and graceful failures are the difference between a demo and a product.
Thank you, Bogdan. One small twist on word boundaries: plain \b treats a hyphen as a boundary, so “lodash” would still match inside “lodash-es”. ManSio’s fix uses custom lookarounds so hyphens and slashes count as part of the name. Scunthorpe problems really are everywhere. And agreed, that rule belongs in the README before day one, not a month later.
Spot on, Earl. You just exposed the classic trap of \b—it completely ignores hyphens because they are non-word characters. So lodash will happily match inside lodash-es or lodash/fp.
Using custom lookarounds like (?<![\w/-])lodash(?![\w/-]) is definitely the bulletproof way to handle package names and URLs. It forces the regex to treat the entire slug as a single token.
And you're 100% right about the Scunthorpe problem. It's the ultimate proof that naive regex will always bite you in production. This is exactly the kind of edge case that deserves a place in the README (or better yet, in a unit test) before you write a single line of validation logic. Great insight!
Agreed, a unit test is a better home for it than the README. Thanks for sticking with this thread!
The expired-token bit genuinely surprised me — that GitHub happily serves public repos with no token at all, but a dead one gets you a 401 and kills the whole scan before Jev or Haiku even run. I'd have assumed an expired token just degrades to anonymous, not that it actively blocks you. Your retry-without-the-header fix makes sense, though Reid's point about Promise.all has me wondering how long that shared-headers trick holds up once those four calls go parallel.
Same! I assumed a dead token would get treated like no token. Nope. And Reid was right about the parallel calls. Mikhail already has a PR up for it, with a test for that exact pair. This comment section is doing better code review than I planned for lol.
That’s a really important distinction, especially the point about the first request running independently before the parallel requests begin. It shows why understanding the actual execution flow matters just as much as understanding the fallback logic itself.
I also think this discussion highlights something broader about AI-assisted development: getting the code to work is only one part of building a reliable application. The real challenge is making sure the assumptions behind that code remain valid when credentials expire, dependencies become unavailable, requests overlap, or traffic increases unexpectedly.
What I particularly like about Mikhail’s per-request retry approach is that it addresses the concurrency problem without introducing unnecessary complexity into the overall architecture. Each request takes responsibility for its own recovery, which makes the behavior more predictable and easier to test.
One additional improvement I'd consider is turning these lessons into explicit regression tests and operational alerts. The tests protect against the same edge cases returning during future changes, while alerts help identify when a supposedly occasional fallback starts happening consistently in production.
I also agree that recording a Haiku failure in the scan receipt is useful, but there's a difference between storing diagnostic information and actively monitoring it. A failure reason that nobody reviews can remain hidden just as easily as an expired API key.
For me, the biggest takeaway from this entire discussion is that resilience should be designed into the normal execution path, not added only after something breaks.
And honestly, this is what makes post-hackathon debugging so valuable. The original build proves that the idea works; real-world failures reveal whether the engineering decisions can support it over time. Great discussion, and credit to everyone who contributed practical fixes rather than just pointing out problems.
Dropping
headers.authorizationon a 401 works cleanly whenfetchRepo()runs calls sequentially, though if those four metadata requests ever run underPromise.all, all four already go out with the expired header before the first 401 returns.The Vercel scope trap pairs awkwardly with fail-open Haiku summaries too. Because the UI still renders a fallback line when the key returns 401 in Production, an expired or mis-scoped key stays invisible unless the catch block emits a warning log or counter.
Good catch, and you were right. The first call runs alone, so an expired token gets caught before the parallel pair goes out. A token that dies mid-scan is a different story: both parallel calls come back 401 and only one gets its retry. Mikhail opened a PR for exactly that about an hour after your comment. Each request now remembers whether it carried the token and retries its own 401.
On fail-open: each Haiku failure does write a reason code to the scan receipt. Nothing taps me on the shoulder, though, which is how a dead key sat there for a month.
Thanks for the shoutout, Earl — really appreciate the credit!
Loved reading the honest post-mortem. Seeing an expired token, a dev-scoped key, and a hidden substring match all line up after a month is such a classic post-hackathon moment. Glad the
mentionsWordfix helped, and it's awesome to see the scanner back to green!Thank you for the fix, and for three more PRs today! The per-request retry is the exact thing Reid and Sina were circling. Reading them now. Your name’s in the README where it belongs.
Relatable postmortem — nothing actually regressed in the verifiers; the failure was external credential and environment drift: an expired 30-day GitHub token and a Haiku key scoped only to Development in Vercel, both masked locally during the demo and surfaced only in deployment. That argues for putting token expiry and key scope into startup health checks and logs, rather than assuming a read-only public-repo tool stays authenticated. Your split of responsibilities is the right boundary too: deterministic rules settle what's true, models only propose buckets, so a flaky model can't corrupt the verdict.
Startup checks for token expiry and key scope would have caught both of these on day one. Cheaper than my method, which was finding out a month later lol. And thank you on the split. A flaky model not being able to touch the verdict is why Haiku failing didn’t break the report.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
That’s a different kind of build than mine. Models only label and summarize here, and plain code makes the calls. What broke wasn’t latency. It was an expired token and a key in the wrong environment.
The ava hiding inside javascript detail is the one that stuck with me. It is such a quiet failure mode, a check that looks correct until a short name shows up inside a longer word. The expired token story landed the same way for me. I had always assumed a dead token would behave like no token, and learning that GitHub treats an expired header as a hard rejection explains a lot of confusing 401s. Thanks for writing this up as a postmortem instead of just fixing it quietly.
The ava inside javascript one still makes me laugh and wince. Credit to Mikhail for catching it. I had the same assumption about dead tokens. Writing it up beat fixing it quietly, because the comments here ended up improving the fix.
What I like about this post-mortem is that nothing in the verifiers was wrong. All three failures lived around the code: a token with a 30-day life, a key scoped to the wrong environment, and a substring check that only misbehaved on a real repo. The suggestion from mp3sss to put token expiry and key scope into a startup check is probably the cheapest fix on the list.
Environment drift is something we see a lot when rebuilding deploy pipelines at Ascendra Ventures. On one WordPress migration, the old setup was a single EC2 box, deploys were done by copying theme files over SSH, and editors entered every content change twice. The rebuild baked everything into an immutable image on Cloud Run, pulled license keys from Secret Manager instead of the machine, redeployed staging on every push and kept production behind an approval. Staging and production stopped being two hand-kept places that could quietly drift apart.
Would a scheduled weekly scan against one known public repo have caught the expired token before a user did?
Probably yes. The token expired on a 30-day clock and only failed on a real call, so a weekly scan against one known public repo would have hit it before a user did. It’s the cheapest kind of canary. Your point about staging and production quietly drifting is the bigger lesson for me. Nothing in my code broke. Everything around it aged like milk. That's the thing with these hackathon builds, they die silently. So now, the next question, do I break them down after judging or let them live on?
A README checker tripping over a substring is a little poetic, since READMEs are full of long URLs and a URL is basically lowercase words glued together. Would stripping link targets before the coverage check catch more of these, or do you want URLs to count as mentions on purpose?
Turns out ManSio’s whole-token fix already covers it: a name right after a / never counts, so URL paths are out. Good nudge to add tests that keep it that way.
The ava-inside-javascript example makes the verifier bug easy to understand. I’d keep the token-expiry failure separate from the matching fixture so a future failing scan points to the right layer. For interview practice, this failing-test exercise gives a related challenge: isolate one failing behavior and explain why the smallest fix belongs there.
This is a brilliant breakdown of the 'hidden gaps' problem. As someone who just built a system entirely from scratch without any frameworks (pure vanilla PHP), I felt this deeply.
Without a framework, you literally have to build your own invariants. When I launched my free webmaster tool directory, I learned the hard way that normal validation isn't enough—bots bypassed my basic checks and flooded the system. I had to rewrite the state flow and manually implement OAuth just to ensure a 'submission' always came from a real human.
It's painful, but it forces you to truly understand every single assumption in your architecture. Great write-up!
Great write-up! It's amazing how small things like expired 30-day tokens, environment variable scoping in Vercel, and simple substring checks (javascript hiding ava) can break an entire app after a hackathon. Also, regarding the deployment side, I checked the status using. you can check if Vercel is down for me there), and it looks like broader infrastructure and service issues are sometimes adding to these deployment and API verification hiccups right now. Thanks for sharing these valuable debugging field notes!
One area I would pay particular attention to in this implementation is how the GitHub fallback behaves under concurrent requests, because this is exactly the kind of detail that can look harmless during development but create unpredictable behavior in production.
Consider what happens when "fetchRepo()" executes four metadata requests concurrently using "Promise.all". If those requests share a mutable headers object and one of them receives a "401 Unauthorized" response, modifying that shared object to remove the expired authorization token can introduce a race condition.
At that point, some requests may already have been dispatched with the expired token, while others may observe the modified headers. This makes the fallback behavior dependent on timing rather than an explicit and predictable request policy.
I would avoid mutating shared request configuration altogether. Instead, each request should receive its own immutable headers object, and the authentication fallback should be handled independently at the individual request level.
For example, when a public GitHub GET request receives a "401", retry that specific request once without the authorization header. This keeps the retry logic isolated, prevents one request from influencing another, and makes the overall behavior much easier to reason about, test, and maintain.
There is another important consideration here: GitHub's unauthenticated REST API rate limit.
Unauthenticated requests are generally limited to 60 requests per hour per originating IP address. That might seem sufficient when testing locally, especially when traffic is low and requests are infrequent. However, the situation changes significantly in a serverless environment.
Multiple users and separate application instances can share the same outbound IP capacity. If the authenticated path starts failing and the application consistently falls back to unauthenticated requests, the fallback mechanism itself could become a source of rate-limit failures.
In other words, a fallback designed to improve availability could unintentionally introduce another reliability problem when traffic increases.
I would therefore treat the fallback as a controlled recovery mechanism rather than an unlimited alternative. It is worth considering how frequently it can be triggered, how rate-limit responses are handled, whether retries are bounded, and whether the application can expose enough information to distinguish authentication failures from rate-limit exhaustion.
The Haiku fail-open behavior deserves similar attention.
Returning a valid report when the AI service is unavailable is absolutely the right availability decision. A temporary dependency failure should not necessarily prevent users from receiving useful application output.
However, fail-open behavior should never mean fail-silently behavior.
I would introduce structured logging and a few lightweight metrics around this path. For example:
These signals would help identify whether an occasional dependency failure is being handled correctly or whether a recurring upstream problem is gradually becoming the application's normal operating mode.
I would also make sure that retries are bounded and that fallback behavior is covered by tests for concurrent requests, expired credentials, rate limiting, and AI service unavailability.
What makes this particularly interesting is that none of these concerns requires a major architectural redesign. The underlying approach can be perfectly reasonable, yet a few small implementation assumptions can have a disproportionate impact on production reliability.
Shared mutable state, credential handling, external API quotas, and insufficient observability are all relatively small details in isolation. But when they interact under real traffic, they can produce failures that are difficult to reproduce and even harder to diagnose.
My broader takeaway is that reliability is not just about having a fallback for when a dependency fails. It is about making sure the fallback itself is isolated, bounded, observable, and safe under concurrency.
A system is not truly resilient simply because it can recover from one failure. It is resilient when that recovery mechanism does not quietly create the next failure.
These are exactly the kinds of edge cases worth addressing before an implementation moves from working in development to handling unpredictable production traffic.