DEV Community

Cover image for My LLM Pipeline Never Crashed. It Just Quietly Got Worse.
Roy Levy
Roy Levy

Posted on Originally published at roylevy.dev AI-assisted

My LLM Pipeline Never Crashed. It Just Quietly Got Worse.

Normal software fails loudly. My article pipeline failed with HTTP 200 and valid JSON.

When a normal app breaks, it tells you.

A webhook is missing a field. A database constraint trips. An exception bubbles up, and the stack trace points at the exact line.

Annoying? Sure. But at least you know.

I've been building a pipeline that writes blog articles for real sites. A planner writes a brief, a writer drafts the article, an editor revises it. Each stage hands the next one its instructions.

This week I found this inside one of those briefs, in a section plan for an article about one of my own case studies:

Use note b51d42cf-cf42-4c28-ae

That's it. That's where the instruction ends.

Half a UUID. No error. No alert. Every check passed.

So how do you debug a bug that looks exactly like success?

Half a UUID, zero errors

Here's what happened.

The planner returns its brief as structured JSON. To keep briefs tight, the schema had length limits: 280 characters per outline point, 240 per note for the writer, 155 for the meta description, 1,600 for the brief itself.

Reasonable, right? Here's roughly what we sent the model (simplified):

outline: {
  type: "array",
  items: { type: "string", maxLength: 280 },
}
Enter fullscreen mode Exit fullscreen mode

With OpenAI's strict structured outputs, the model doesn't fail when it hits a maxLength. In our runs, it just stopped the string. Mid-sentence. Sometimes mid-word.

Here's the outline point from the hook, exactly as the planner returned it (wrapped here so you can read it). It's 280 characters:

## Sniffo.ai separates collected inputs from
scoring decisions (220 words): Use note
4e7d7a7c-1c6e-422b-b865-4953727d9931 to
explain SourceItem rows, the separate
Opportunity mapper, business-profile
application, and re-scoring without network
work. Use note b51d42cf-cf42-4c28-ae
Enter fullscreen mode Exit fullscreen mode

A few more from the same two weeks:

"…a product can assess separatel"

"…or validate the planned duration through a complete"

The response was valid JSON. Our Zod schema checked the length, and the length was fine (it was exactly the limit). So the repair step we had for bad briefs never ran.

Then the writer got its instructions with the ending missing. And the ending is usually the important part: which note to use, what the section should prove, how it should land.

The writer doesn't complain either. It works with whatever it got.

So I counted. Two weeks of briefs, 14 of them, across 6 sites:

  • 23 of 120 writer notes stopped at exactly 240 characters.
  • 7 of 96 outline points stopped at exactly 280.
  • 11 of the 14 briefs had at least one field cut off.

Eleven out of fourteen.

Nothing crashed. Not once.

The guards were the bugs

Once I started looking for this shape, I kept finding it. Same week, same pipeline:

The link check removed every internal link. After a prompt update, the planner started writing links as bare slugs (/some-post) instead of real paths (/blog/some-post). The link check couldn't match them, so it did its job and unlinked them. All of them. Every target page actually existed. The drafts came out with orphan sentences like "read Anno 1800 vs Farthest Frontier." pointing at nothing.

A capital letter threw away a whole edit. The editor capitalized a link's anchor text. The check that makes sure the editor didn't break links compared links by their text, saw a "changed" link, and discarded the entire editorial pass. The article kept its unedited draft.

Thai text blew the token budget. The budget assumed English-like characters per token. A Thai draft of 12.4k characters is about 11k tokens, against a 9k budget. The editor's reply came back as invalid JSON, so the edit was skipped. Silently.

A word count halved the articles. The new prompts asked for a target word count in each brief. New drafts came in at 722–1,133 words. The old ones ran 1,128–2,193. The worked examples, the most useful part, were what got cut. No error. Just thinner articles. They lost a side-by-side comparison against the old prompts, 4–1.

Four guards, four green checks, four things quietly deleted.

Look at those again.

Every one of them is a safety check doing exactly what it was built to do. Too long? Cut it. Link doesn't match? Remove it. Edit looks wrong? Throw it away.

Each guard was built to keep the pipeline running.

So when something went wrong, it didn't stop. It deleted something and kept going.

That's not graceful degradation.

That's silent data loss with good manners.

More rules made it worse

The obvious move when an AI gets something wrong is to add a rule.

I tried that. It backfired twice.

First try: the platform prompts were full of "never claim X" and "don't add Y" lines. Those lists planted the exact words they were warning against. My own site's first generated article put a disclaimer on my own 30% result, because the prompts were full of warnings about numbers. Rewriting every rule as "do this" instead of "never do that" took the avoid-lines from 46–86 per site down to 0–3.

Second try: some articles had worked examples with numbers that went past real limits. So I added a "real limits" rule to the writer, the planner and the prompt generators. The planner turned it into "numerical examples must be supported by item data." One article dropped its example numbers completely. Others started hedging ("explicitly illustrative"). It lost the blind test to the version without the rule, 3–2.

Same lesson both times.

Tell the writer to be careful, and it gets vague.

What worked was moving the check out of the writer. The editor now walks through each worked example, checks it against the article's own rules, and fixes the result from the example's own inputs. Same drafts, blind comparison: 4 wins, 1 tie, 0 losses.

Let the generators write. Make a separate stage check.

What replaces the stack trace

If errors won't tell you, you need other signals. This is what I use now.

Look for outputs sitting exactly on a limit. One output at exactly 280 characters is a coincidence. A pile of them is truncation. It's one short query, and it's what showed me how big this was:

select
  count(*) filter (where length(note) = 240) as at_limit,
  count(*) as total
from content_briefs b,
  jsonb_array_elements_text(b.brief_json -> 'notesForGenerator') as note
where b.created_at > now() - interval '14 days';
Enter fullscreen mode Exit fullscreen mode

If at_limit isn't close to zero, the model isn't finishing its sentences.

Don't put limits where the model can cut. The fix removes maxLength from the schema we send. Each field's description says "At most N characters." Our own code checks the length after the call and runs one repair pass if it's over:

function textSchema(limit: BriefTextLimit): JsonSchema {
  // No maxLength: strict outputs would stop mid-word to obey it.
  return { type: "string", description: `At most ${limit.max} characters.` };
}

// After the call: our own Zod check. Over the limit? One repair pass
// asks the model to shorten that field.
Enter fullscreen mode Exit fullscreen mode

The limits also went up to what the writer actually needs (600 for an outline point, 800 for a note). Replayed on six sites first: nothing landed on a limit, and the one meta description that ran long (160) was repaired to 144. It shipped the same day. The first two real briefs after it had nothing on a limit, and notes ran up to 593 characters. That's how much the writer was missing.

Make links impossible to get wrong instead of checking them after. The planner now picks links by ID (L1, L2…) from the same list the link check uses, and code turns the IDs into paths. On replay, 14 of 14 links reached the page they were meant for.

Replay before you ship. Every prompt or schema change gets replayed offline against stored briefs and drafts first. It's cheaper than finding out on a customer's site, and it's the only way to see every case at once instead of one symptom per bug report.

Compare blind. Old output vs new, labels hidden. If you know which one is "the fix", you'll find a reason to like it.

Blind isn't enough on its own, either. In one model bake-off, I had three judges rank the same articles, labels hidden and order shuffled. Claude's judge put the two Claude-written articles first. GPT's judge put the GPT article first. Gemini, the only one with no horse in the race, sided with Claude.

Nobody told them who wrote what. They just liked the sound of their own voice. (Relatable, honestly.)

That's one topic, so it proves nothing. But if you use AI judges, use more than one company's.

Where this approach falls short

I don't want to oversell this.

Five-article blind tests are a reading, not a measurement. They caught real regressions. But 4–1 on five topics wouldn't survive a statistics class. They tell you which way to lean, not how big the effect is.

I still can't see how an article actually performs. Right now, a prompt change gets judged on how the drafts read, not on how they do in search. Keeping every version of every article (the first step toward that) is still being built.

Strict schemas are still worth it. Structured outputs mean I never parse broken JSON, and that's a big deal. The limit was the problem, not the schema.

Some failures are loud, and that's fine. Auth, billing and database errors still throw, still hit Sentry, still alert me. Normal engineering still applies. It just doesn't cover the part that matters most here: whether the article got worse.

What I'd do in your pipeline

If you're chaining LLM calls, check these this week:

  • Remove maxLength from any schema a model fills in. State the limit in the description and check it in your own code.
  • Count outputs that land exactly on a limit. That number should be close to zero.
  • Find every guard that deletes or skips on failure. Make each one log what it dropped, with a counter you actually look at.
  • Count token budgets in tokens, not characters, and test them with a non-Latin language.
  • When the model gets something wrong, fix it in a checking stage, not with another rule for the generator.
  • Replay stored inputs and compare blind before every prompt change.

So why is this harder?

Here's my honest answer.

It's not that LLMs are random. The truncation was perfectly deterministic. Same limit, same cut, every time.

It's that "correct" stopped being something code can check. A normal system is either right or it throws. A pipeline like this has a third state: valid, passing, and worse.

Loud failures vs quiet ones.

Every guard I added made the pipeline safer from crashing, and better at hiding its mistakes.

In an LLM pipeline, HTTP 200 doesn't mean it worked.

This pipeline is NichePressa, the content engine I built and run on my own sites.


Running LLM calls in a chain? I'd love to hear what's quietly breaking in yours, and how you caught it. Send me a note.

Originally published at roylevy.dev.

Top comments (8)

Collapse
 
kolmopdf profile image
KolmoPDF •

The maxLength truncation behavior under OpenAI structured outputs is brutal because constrained decoding forces token generation to halt cleanly at the exact limit, leaving a syntactically valid JSON string that passes every downstream schema check.

We ran into the exact same shape of failure in document parsing pipelines with table and list cell budgets: setting a strict schema limit didn't raise a validation exception, it just silently truncated trailing columns or footnotes mid-token. Moving the constraint out of JSON schema validation and into an explicit description with a post-generation repair pass is much cleaner. A secondary audit query checking for outputs whose string length equals limit is also a great operational smoke test: it turns what looks like quiet success into a measurable regression alert.

Collapse
 
roylevy profile image
Roy Levy •

Tables and footnotes are a nasty version of this, because a table missing its last column still looks like a complete table.

Did you end up with a repair pass too, or did you handle the cell budget some other way? Curious whether the “equals the limit” count caught it for you, or something else showed up first.

Collapse
 
gautamtalksdev profile image
Gautam K •

The capital letter case is the one I'd generalise. A guard that compares by a field the other stage is allowed to change will fire on every legitimate edit. Comparing links by normalised href, ignoring anchor text and case, would have kept that editorial pass.

The rule I'd add to all four guards: a guard may block or flag, but never delete on its own, and it counts how often it fired per run, so going from zero unlinked links to all of them shows up as a number.

Did any of the four have a counter like that, or did you find each one by reading the output?

Collapse
 
roylevy profile image
Roy Levy •

Honest answer: none of them had a counter. I found every one by reading drafts and replays, not from a number. Each started as “this looks off” and then digging.

Your normalised href is exactly where that fix landed. The check now compares links by destination, so the editor can change anchor text all it wants. Bare slugs got the same treatment: the planner now picks links by ID, so there’s nothing to mismatch.

And “a guard may block or flag, but never delete on its own, and it counts every time it fires” is a better rule than anything I wrote in the post. A counter that jumps from zero to every link in the draft is the stack trace I said was missing. That’s going on my list.

Collapse
 
prpatel05 profile image
Pratik Patel •

The half UUID is the detail I'd build a check around, because it was the one cut that was machine-checkable. Every note ID in a brief should resolve to a real note before the writer sees it. b51d42cf-cf42-4c28-ae doesn't resolve, so that brief fails loud even if nobody ever thinks to count outputs sitting on a limit.

I'd point the same check at the new repair pass. "Shorten this field" is another stage that's allowed to delete, and the ending is still the part most likely to go. If the original field referenced a note ID or link ID and the repaired one doesn't, that's a dropped instruction, not a successful repair. Your 160 to 144 meta description is harmless. A 900-character note repaired down to 800 that loses its ID is the same bug again, just with no limit left to sit on.

Collapse
 
roylevy profile image
Roy Levy •

You’re right about the half UUID, and it’s the cleanest kind of check because there’s no judgment involved. An ID either resolves or it doesn’t.

That one is in now. The planner only keeps note IDs it was actually shown, so b51d42cf-cf42-4c28-ae gets dropped before the writer ever sees it. Since this week it’s counted too.

But “dropped and counted” is still quiet. Nobody gets told. So I’m making it an alert: if a brief loses a note it picked, that hour pages me, same as when it loses most of its links.

On the repair pass, I got a bit lucky with the structure. Note IDs and links are their own fields in the brief, not text inside a paragraph. The repair pass only rewrites text fields, so it can shorten a hook but it can’t reach an ID. If IDs ever move into free text, your rule is exactly the check I’d want: whatever IDs the original had, the repaired version must still have.

“A stage that’s allowed to delete” is a good way to put it. I’m going to start listing those stages on purpose.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

The detail that a truncated string was exactly the limit, and therefore passed the length check, is a great catch. The repair step was correct for the failure you expected and invisible to the one you got. Checking for values that sit exactly at the limit seems like a cheap rule worth adding to every schema with a maxLength.

What I take from the post is that the pipeline never had a state for "finished, but suspicious". Everything was either an error or a success, and a half UUID counted as success.

We had the same shape of problem in a candidate enrichment pipeline at Ascendra Ventures. Nothing crashed. Candidates just stopped somewhere in the middle and stayed there. The fix was to rebuild it as a durable workflow where every person has to end in a final status, either finished or visibly failed, with fallback polling when callbacks went quiet. Taking stuck away as a possible state is the same move as your equals-the-limit check: it gives the quiet failure somewhere visible to land.

Gautam's idea of a counter on each guard fits that well. Which of the four guards would have fired most often if it had been counting from the start?

Collapse
 
roylevy profile image
Roy Levy •

“Finished, but suspicious” is a better name for it than anything in my post. That’s exactly the state that was missing.

Honest answer: the length cut, by a mile. I know that because it’s the only one I could count after the fact. 11 of 14 briefs had at least one field cut off. That’s not an edge case, it’s the normal path.

The link check comes second, but I can’t give you a number. It only went wild after a prompt update, and nothing was counting back then.

That’s the part that stings. The guard that fired the most was also the one I trusted the most.

The counters are live now, and one of them fires on any dropped source note. So the next half UUID ends up somewhere I can see it.