One thing worth adding to the checklist is pinning the prompt to the model and eval run it last passed with, because a provider model update can shift answers while the prompt text looks untouched. With that note in place, a drop in scores points straight to the model change instead of sending everyone back to rewrite a prompt that barely changed.
Great point, and it’s a gap in the checklist. The post logs model versions for observability and runs evals on every change, but it never says to tie the two together. I’d add it to item 3: pin each prompt to the model version and eval run it last passed with.
You’re right that a provider-side model update can shift outputs while the prompt text stays identical. If you’ve recorded that pairing, a score drop points straight at the model change. Without it, the team ends up rewriting a prompt that barely changed, which is the exact time sink the post is about.
Thanks for adding this. Have you hit a silent model update in production, and how did you catch it?
I don't have a production story worth telling there, so I won't make one up. The cheapest catch I know of is rerunning a small fixed eval set on a schedule even when nothing in the repo changed, so a score drop with no commit behind it stands out straight away.
Appreciate the honesty, and I’d rather see that than an invented war story. The scheduled eval rerun is a solid catch. A score drop with no commit behind it points straight at drift outside your code: a silent model update, a provider-side change, or shifting retrieval data.
One thing that makes it work even better is pinning the model version where you can, and logging the model ID and a hash of the prompt template with each run. When the score moves, you can tell in seconds whether it was your change or theirs.
It also fits the “80% everything else” point in the post: the unglamorous monitoring around the model is what keeps a prompt that worked in demo from quietly degrading in production.
Pinning the version and hashing the template is a neat pair, because it turns 'did something change?' into a lookup instead of a guess. It's probably the cheapest monitoring you can add, and it's the bit people only wish they'd had after the first silent drop.
We log the model version with every request, and tagging a prompt hash next to it turns “did something change?” into a lookup, as you say. Once both are on every log line, a quality dip lines up with a deploy, a template edit, or a provider-side change, and you know which to investigate first.
Two caveats from our experience:
Pinning only helps if you pin a dated snapshot. Floating aliases like “latest” can change under you, and even pinned versions eventually get deprecated, so the eval set has to run before every forced migration.
A template hash doesn’t cover what’s inside the template. Retrieved context can drift while the hash stays identical, so we also log the retrieval settings and index version.
Are you hashing the raw template or the fully rendered prompt? We went with the template plus a separate retrieval version, but I’m curious whether the rendered prompt has worked better for you.
Writing the prompt is a weekend hack; the real engineering is building the deterministic plumbing, tail-latency defenses, and state machines required to babysit an untrusted, non-deterministic syscall in production.
That’s a fair way to put it, and I agree with the framing. The prompt is the part you can write in a weekend. The part that takes months is making a probabilistic component behave inside a system that expects deterministic behavior.
You also named two things I only touched on. I covered latency mostly as streaming and model routing, but tail latency deserves more attention: p99 timeouts, retries that multiply cost, and hedged requests. In my case the retries after timeouts were a big reason the first month’s bill came in at roughly triple the estimate. The state machine point is just as valid. Once the model can take actions rather than only write text, you need explicit states, idempotent steps, and narrow permissions so a bad output can’t push the workflow somewhere unrecoverable. Output validation, fallbacks to a human, and scope limits are my versions of that, but “babysitting an untrusted syscall” is a cleaner mental model than what I wrote.
Curious what’s worked for you on the state machine side. Do you gate every model-initiated action behind a validator, or only the irreversible ones?
The three versions of the refund policy is the detail that will be familiar to anyone who has shipped one of these. The model did what it was asked and summarised contradictory sources faithfully, so no prompt change could have fixed it.
Your point that retrieval quality sets the ceiling on answer quality matches what we see on client builds. The work that decides the outcome is choosing which document is authoritative, adding metadata so retrieval can filter, and keeping the index fresh after launch.
Treating the eval set as a deliverable is the change I would push hardest. A hundred or so real inputs with facts that must appear and things that must never be said turns "it feels better" into a result you can compare between versions.
The reply about content ownership may point at the biggest gap of all. Who owns the source documents six months after launch, the support team or whoever built the assistant?
Thanks for this. Once the model summarises three conflicting sources faithfully, there’s nothing left to fix in the prompt. The fix is upstream.
On your ownership question, in my experience it’s usually the gap that decides whether a build lasts. The split that has worked best for us is that the support team owns the content and the build team owns the pipeline. The support team decides what is true and the build team makes sure the index reflects it. Without that split, the assistant slowly drifts away from the real policies and nobody notices until a customer quotes a wrong answer back.
A few things make that split work in practice:
A named owner for each document set, not just “the support team”. A person gets pinged, a team doesn’t.
Review dates in the metadata, so retrieval can deprioritise or flag anything past its date.
A route from bad answers back to the owner. Low-rated or escalated conversations should land with whoever owns the source, not only with the engineers.
A freshness check in the refresh pipeline that alerts when a doc hasn’t been touched in a set window.
Agreed on evals too. Once there’s a set you can run, “it feels better” stops being an argument. Do you already have an owner model like this on your builds, or does it tend to get settled after the first stale answer?
The line about retrieval quality setting the ceiling on answer quality is the one I’d put on a poster. When an answer is bad, most people go straight to tweaking the prompt, when the real problem is what went into the context window.
To answer your question, the part that surprises people most is probably evals. A prompt change that “feels better” on five examples is easy to ship, and it’s hard to notice that it quietly broke a whole category of questions. Treating a regression like a failing test changes how a team works.
I also liked the point about the system saying “I’m not sure” instead of delivering a polished guess. That builds more trust than a fluent wrong answer.
One thing I’d add to the checklist is ownership of the source content. The refund policy example shows it: if nobody owns the docs, the knowledge base drifts and the model just reproduces the mess confidently.
Thanks, I’m glad that line about retrieval landed. It’s the one I wish I’d understood before the three months of debugging prompts that were never the problem.
Your point about content ownership is a good one, and I should have made it a bigger part of the checklist. The refund policy mess came from exactly that. Nobody owned the docs, so every team added its own version and the model blended them. Now I’d put an owner and a review date on every source the model can retrieve, so stale content gets flagged before it turns into a confident wrong answer.
And yes, evals are the shift that changed how our team works. Once a regression counts as a failing test, “it feels better” stops being a good enough reason to ship.
Mayur Upadhyay delivers tech-driven solutions in Web Development, e-commerce development, and mobile app development, helping businesses to grow worldwide.
The point about retrieval quality setting the ceiling on answer quality is the one I’d put on a poster. When output looks bad, the instinct is to rewrite the prompt, but the real problem is usually what’s sitting in the context window. Three conflicting versions of a refund policy will get blended into something that matches none of them, no matter how good the prompt is.
The eval section also resonated. Without an eval set, every change is just a vibe check. Treating a regression like a failing test is a simple mindset shift that changes how a team ships.
To answer your question, the part that surprises people most seems to be data, because nobody budgets time for cleaning and versioning source content until retrieval starts returning contradictions.
One thing I’d add to the checklist is a plan for who owns the content after launch. A refresh pipeline only works if someone is accountable for keeping the source docs accurate.
Great write-up. The “demo lies to you” framing is spot on.
Glad the retrieval point landed. It’s the lesson I learned the hard way, since I spent days tweaking prompts before realizing the model was just faithfully summarizing contradictory docs.
You’re right about content ownership too. It’s the unglamorous gap that quietly kills these projects: the index is fresh at launch, and six months later nobody owns the source docs and answers start drifting. I’d treat it like any other production dependency, with a named owner and a review cadence.
Appreciate you reading and sharing your take!
We're a place where coders share, stay up-to-date and grow their careers.
One thing worth adding to the checklist is pinning the prompt to the model and eval run it last passed with, because a provider model update can shift answers while the prompt text looks untouched. With that note in place, a drop in scores points straight to the model change instead of sending everyone back to rewrite a prompt that barely changed.
Great point, and it’s a gap in the checklist. The post logs model versions for observability and runs evals on every change, but it never says to tie the two together. I’d add it to item 3: pin each prompt to the model version and eval run it last passed with.
You’re right that a provider-side model update can shift outputs while the prompt text stays identical. If you’ve recorded that pairing, a score drop points straight at the model change. Without it, the team ends up rewriting a prompt that barely changed, which is the exact time sink the post is about.
Thanks for adding this. Have you hit a silent model update in production, and how did you catch it?
I don't have a production story worth telling there, so I won't make one up. The cheapest catch I know of is rerunning a small fixed eval set on a schedule even when nothing in the repo changed, so a score drop with no commit behind it stands out straight away.
Appreciate the honesty, and I’d rather see that than an invented war story. The scheduled eval rerun is a solid catch. A score drop with no commit behind it points straight at drift outside your code: a silent model update, a provider-side change, or shifting retrieval data.
One thing that makes it work even better is pinning the model version where you can, and logging the model ID and a hash of the prompt template with each run. When the score moves, you can tell in seconds whether it was your change or theirs.
It also fits the “80% everything else” point in the post: the unglamorous monitoring around the model is what keeps a prompt that worked in demo from quietly degrading in production.
Pinning the version and hashing the template is a neat pair, because it turns 'did something change?' into a lookup instead of a guess. It's probably the cheapest monitoring you can add, and it's the bit people only wish they'd had after the first silent drop.
We log the model version with every request, and tagging a prompt hash next to it turns “did something change?” into a lookup, as you say. Once both are on every log line, a quality dip lines up with a deploy, a template edit, or a provider-side change, and you know which to investigate first.
Two caveats from our experience:
Are you hashing the raw template or the fully rendered prompt? We went with the template plus a separate retrieval version, but I’m curious whether the rendered prompt has worked better for you.
Writing the prompt is a weekend hack; the real engineering is building the deterministic plumbing, tail-latency defenses, and state machines required to babysit an untrusted, non-deterministic syscall in production.
That’s a fair way to put it, and I agree with the framing. The prompt is the part you can write in a weekend. The part that takes months is making a probabilistic component behave inside a system that expects deterministic behavior.
You also named two things I only touched on. I covered latency mostly as streaming and model routing, but tail latency deserves more attention: p99 timeouts, retries that multiply cost, and hedged requests. In my case the retries after timeouts were a big reason the first month’s bill came in at roughly triple the estimate. The state machine point is just as valid. Once the model can take actions rather than only write text, you need explicit states, idempotent steps, and narrow permissions so a bad output can’t push the workflow somewhere unrecoverable. Output validation, fallbacks to a human, and scope limits are my versions of that, but “babysitting an untrusted syscall” is a cleaner mental model than what I wrote.
Curious what’s worked for you on the state machine side. Do you gate every model-initiated action behind a validator, or only the irreversible ones?
The three versions of the refund policy is the detail that will be familiar to anyone who has shipped one of these. The model did what it was asked and summarised contradictory sources faithfully, so no prompt change could have fixed it.
Your point that retrieval quality sets the ceiling on answer quality matches what we see on client builds. The work that decides the outcome is choosing which document is authoritative, adding metadata so retrieval can filter, and keeping the index fresh after launch.
Treating the eval set as a deliverable is the change I would push hardest. A hundred or so real inputs with facts that must appear and things that must never be said turns "it feels better" into a result you can compare between versions.
The reply about content ownership may point at the biggest gap of all. Who owns the source documents six months after launch, the support team or whoever built the assistant?
Thanks for this. Once the model summarises three conflicting sources faithfully, there’s nothing left to fix in the prompt. The fix is upstream.
On your ownership question, in my experience it’s usually the gap that decides whether a build lasts. The split that has worked best for us is that the support team owns the content and the build team owns the pipeline. The support team decides what is true and the build team makes sure the index reflects it. Without that split, the assistant slowly drifts away from the real policies and nobody notices until a customer quotes a wrong answer back.
A few things make that split work in practice:
Agreed on evals too. Once there’s a set you can run, “it feels better” stops being an argument. Do you already have an owner model like this on your builds, or does it tend to get settled after the first stale answer?
The line about retrieval quality setting the ceiling on answer quality is the one I’d put on a poster. When an answer is bad, most people go straight to tweaking the prompt, when the real problem is what went into the context window.
To answer your question, the part that surprises people most is probably evals. A prompt change that “feels better” on five examples is easy to ship, and it’s hard to notice that it quietly broke a whole category of questions. Treating a regression like a failing test changes how a team works.
I also liked the point about the system saying “I’m not sure” instead of delivering a polished guess. That builds more trust than a fluent wrong answer.
One thing I’d add to the checklist is ownership of the source content. The refund policy example shows it: if nobody owns the docs, the knowledge base drifts and the model just reproduces the mess confidently.
Thanks, I’m glad that line about retrieval landed. It’s the one I wish I’d understood before the three months of debugging prompts that were never the problem.
Your point about content ownership is a good one, and I should have made it a bigger part of the checklist. The refund policy mess came from exactly that. Nobody owned the docs, so every team added its own version and the model blended them. Now I’d put an owner and a review date on every source the model can retrieve, so stale content gets flagged before it turns into a confident wrong answer.
And yes, evals are the shift that changed how our team works. Once a regression counts as a failing test, “it feels better” stops being a good enough reason to ship.
The point about retrieval quality setting the ceiling on answer quality is the one I’d put on a poster. When output looks bad, the instinct is to rewrite the prompt, but the real problem is usually what’s sitting in the context window. Three conflicting versions of a refund policy will get blended into something that matches none of them, no matter how good the prompt is.
The eval section also resonated. Without an eval set, every change is just a vibe check. Treating a regression like a failing test is a simple mindset shift that changes how a team ships.
To answer your question, the part that surprises people most seems to be data, because nobody budgets time for cleaning and versioning source content until retrieval starts returning contradictions.
One thing I’d add to the checklist is a plan for who owns the content after launch. A refresh pipeline only works if someone is accountable for keeping the source docs accurate.
Great write-up. The “demo lies to you” framing is spot on.
Glad the retrieval point landed. It’s the lesson I learned the hard way, since I spent days tweaking prompts before realizing the model was just faithfully summarizing contradictory docs.
You’re right about content ownership too. It’s the unglamorous gap that quietly kills these projects: the index is fresh at launch, and six months later nobody owns the source docs and answers start drifting. I’d treat it like any other production dependency, with a named owner and a review cadence.
Appreciate you reading and sharing your take!