Codex Never Crashed. It Just Quietly Did Nothing.
From the agentpolicy v0.1.0 release: a local-first, Cedar-based policy decision point for AI agents (decide allow, deny, ask, rate-limit, or redact for every agent action, with shipped rule packs). Now on PyPI. This story is journal Group 21 of the v0.1.0 field test report.
AI disclosure: this article was drafted and edited with AI assistance. The field test, the defects, the fixes, and every number are real and linked below.
Three bugs stood between me and a live Codex CLI integration. Not one of them printed an error. Every one of them let my harness sit there looking finished while producing nothing. This is the kind of debugging session that makes you stop trusting a quiet process.
Trap one: the API you assumed still exists
I wired Codex CLI against my local model server the way I had wired everything else: chat completions, the endpoint every OpenAI-compatible server speaks. Codex refused at startup: wire_api = "chat" is no longer supported. The CLI dropped chat completions entirely. The only transport it accepts now is the OpenAI Responses API.
The lesson is not "read the changelog," although yes. The lesson is that my local server, oMLX, happened to serve /v1/responses natively, and I only found that out with a curl probe instead of an assumption. One probe, a real resp_* body, HTTP 200. The entire integration unblocked with zero translation layers.
Vendor APIs drift. Probe the transport fresh for every CLI-to-endpoint pairing, every CLI version. Yesterday's working config is today's silent startup rejection.
Trap two: the stdin hang
codex exec, run headless from a test script, printed "Reading additional input from stdin..." and then sat there. No error. No timeout. No exit. My harness, which was watching the process and its output, looked "busy." The run produced zero evidence and appeared, from a process list, to be working.
The cause: codex exec reads stdin, and in a harness stdin is whatever the parent left connected, which is never a closing TTY. The fix is two characters of shell: </dev/null. Close stdin by default in every headless invocation.
This one deserves a rule of its own: any headless CLI that consumes stdin will hang your harness quietly. An unattended process with a blocked read is the most patient bug in the world. It will wait forever, and your CI will show a spinner.
Trap three: the 404s that blamed the server
With transport sorted and stdin closed, every model call returned 404. Uniform, instant 404s. I spent a while on the server side, checking routes, because 404 says "server problem," right?
Codex appends /responses to the provider base_url I configured. My proxy URL was http://127.0.0.1:PORT, so Codex called /responses. The server serves /v1/responses. The fix: base_url = "http://127.0.0.1:PORT/v1". Off by one suffix, every call dead, server completely healthy the whole time.
Paths are config. Config is a test surface. A uniform 404 across every request is almost never "the server is broken." It is "the client is asking somewhere else."
The honest scoping problem
There was a fourth decision, not a bug, a design call. Claude Code has a pausable per-call approval hook. I could wire the policy engine directly into the ask-a-human loop and test it live. Codex, headless, does not. Its governance surface is the sandbox and the config file: --sandbox read-only, provider config, executed commands.
I could have forced parity and written a fake approval flow. Instead the case tests what the surface actually guarantees. Allow means the command executed, a real cat that printed real output. Deny means the destructive command was blocked, the target directory stayed intact with its keep.txt, and the agent replanned. Plus a parity check: the same three commands decided by the engine before execution, with an audit file showing allow, escalate, deny matching the in-process engine exactly.
A harness test should assert what the surface guarantees and say so out loud. Stretching a different harness's shape over a vendor surface produces tests that pass and prove nothing.
The final wiring, for the record
Codex CLI 0.135.0 against the box-local model, no OpenAI key. A stdlib logging proxy (t1_proxy.py) between the CLI and the model server captured 4 request and 4 response pairs, sanitized, all 2xx. The parity audit trail has 3 decisions. The allow executed, the deny held and replanned, and the whole thing reproduces from the committed config (t1_codex.sh, codex-home/config.toml in the case's evidence directory).
What worked
- Probing before wiring. The Responses API question was settled with one curl before any config existed.
- Capturing LLM traffic as a hard requirement. Every pair on disk or the case fails. That rule is the only reason "looked busy, did nothing" runs got caught at all.
- Testing the pieces standalone before composing them: policy through
evaluatefirst, then the proxy with curl, then the CLI through the proxy. Each failure lived in exactly one layer.
What did not work
- Trusting that chat completions would still be the transport. It was not, and finding out cost me a full debug cycle.
- Leaving stdin connected. Obvious in hindsight, invisible in the process list.
- Reading 404 as "server broken" before reading it as "client asking in the wrong place."
More detail and evidence
- The full Codex story: field test report, journal Group 21 (FT-T1-CODEX), plus Field Notes 12 for the write-up version of the same traps.
-
The harness, in the repo:
t1_codex.sh(config, parity triple, sandbox triple) ·t1_proxy.py(the/responses-aware capture proxy) ·t1_policy.yaml(the shared Tier-1 policy). -
The captured evidence:
field-test/v0.1.0/results/s18-tier1/cases/FT-T1-CODEX/in the repo (live transcripts, audit trail, 4 LLM pairs, provider config). - The plan row that scoped the case honestly: FIELD_TEST_PLAN.md, the FT-T1-CODEX row and the Tier-1 R-map.
-
The product: agentpolicy,
pip install agentsec-agentpolicy, PyPI 0.1.0, docs reference (adapters, authzen, surfaces), part of the agentic-suite agent-security stack.
What is the most patient bug you have shipped in a harness? The one that produced no output, no error, and no exit, just quiet nothing. I want to know I am not alone.
Top comments (11)
Mine was a loop over a host list that ran the first host and silently skipped the rest. ssh was eating the remaining lines off stdin, the loop hit EOF looking done, no error, no nonzero exit. ssh -n fixed it, same shape as your stdin trap, fd 0 was connected and something consumed it.
Two habits catch this class for me now. Every headless call gets a hard timeout with a kill signal, a blocked read never heals itself. And a step writing zero bytes for a few minutes fails the run, quiet nothing is a failure state, not busy.
Your base_url trap is the one I keep relearning, print the exact assembled URL before touching the server.
ssh quietly eating the loop's stdin is the ur-version of this trap, and fd 0 really is where the most patient bugs live. Your two habits map exactly onto the two fixes I landed after this run: every headless call gets a hard timeout with the kill aimed at the process group, because a blocked read never heals itself, and zero useful output over a stretch of time is treated as a failure state rather than business. In my harness the zero-evidence check is now a hard rule, the case does not pass unless the captured model traffic is on disk, which is what turned "looked busy, did nothing" from a vibe into a caught failure. The base_url one I keep relearning too, apparently.
Good point on aiming the kill at the process group. A plain kill only reaches the one pid, and anything that process spawned keeps running, still holding the port or the file handle, until you signal the whole group. I run the risky steps under setsid now and kill the negative pgid, which caught a couple of stragglers the plain kill never reached.
The captured traffic on disk is a stronger check than the one I described. Log growth only proves motion, and a loop that retries forever grows the log while doing nothing. Bytes of real traffic per case prove the work. I am adopting that distinction.
A timeout catches the process that never returns; I’d also make a successful exit prove that the expected work actually happened. For example, the harness could compare the number of planned cases with distinct case IDs in the receipt and mark missing or duplicate IDs as incomplete, even if the CLI exits zero. That catches the quieter case where a run stops consuming input but still terminates normally.
Agreed, and the planned-versus-distinct-IDs check is a nice concrete form of it. Exit zero is a statement about the process, never about the work, so the receipt has to carry the work. In the harness behind that story, a verdict only counts if its assertions are embedded in it and re-derived from disk, and a pass with no grounded assertions behind it is flagged as over-claimed instead of accepted as green. Same shape as your missing-or-duplicate-ID check: the run has to prove the cases exist, individually, or the exit code does not get to speak for them. Your framing also matches the kill-time rule I run elsewhere: cessation is probed, not inferred, and so is completion.
Trap three is the one that got me — I spent an afternoon blaming the server for uniform 404s before printing the assembled URL and seeing the doubled path. Your line is the whole lesson, really: a uniform 404 is almost never "the server is broken", it's the client asking in the wrong place. Stealing the "close stdin by default" rule too, that's going straight into my harness.
An afternoon lost to uniform 404s is the tax everyone pays once, and I like that your takeaway is the general one rather than the specific URL. A uniform failure across every request is almost never a broken server; it is a consistent client asking consistently in the wrong place. It feels like a server problem because nothing varies, so nothing points anywhere. Enjoy the close-stdin rule. Two characters that buy back entire afternoons.
Thanks for the thoughtful reply, Debashish! The architectural separation makes complete sense—keeping prompt bodies and model internals out of the policy engine while letting rules condition on normalized call attributes is definitely the cleanest path forward.
Per your suggestion, I just opened Issue #1 on limecloud/agentpolicy outlining a draft context attribute payload (tokens_remaining, utilization_pct) and matching defer/escalate and obligation: compact_context decisions: github.com/limecloud/agentpolicy/i...
Looking forward to collaborating on the schema and running some harness field tests against it!
The common thread here is that all three failures were silent, and I think that’s the real lesson. A crash is cheap to debug because it announces itself. A process that blocks on a read, or a client that politely gets 404s, produces nothing to react to, so the only defense is a harness that treats “no evidence” as a failure.
Your rule that every LLM request/response pair must be on disk or the case fails is the strongest part of this. It converts “looked busy” into a binary, checkable outcome. I’d take that one step further and generalize it: every test case should declare the artifact it must produce, and absence of that artifact is a failure, regardless of exit status. That catches the stdin hang, the 404 loop, and a whole class of bugs you haven’t met yet.
A few additions on each trap:
Trap 1 (API drift). Your curl probe is a good habit worth automating. A tiny preflight that hits the endpoint the CLI will actually use, and asserts on the response shape (a resp_* ID, a 200), takes seconds and can run before every suite. Pinning the CLI version in the harness also helps, so a drift shows up as a deliberate upgrade rather than a surprise.
Trap 2 (stdin). </dev/null is the right fix, but I’d pair it with a hard wall-clock timeout on every headless invocation (timeout 120 codex exec ... </dev/null). Closing stdin prevents this particular hang, while the timeout covers the next unknown one, like a blocked network read or an interactive prompt nobody anticipated. A hung process should become a loud, attributable failure instead of a spinner.
Trap 3 (the 404s). Your point that a uniform 404 usually means the client is asking in the wrong place is a great heuristic. Your logging proxy is what makes it provable, because you can see the exact path being requested instead of inferring it. I’d also add a startup check that logs the fully resolved URL the client will call, so a missing /v1 is visible in the first line of output.
I also appreciated the scoping decision. Testing what the surface actually guarantees (sandbox, config, executed commands) instead of faking an approval loop to match Claude Code is the honest call. A test that passes only because you bent a different harness’s shape over it proves very little. The parity check, where the same three commands get the same allow/escalate/deny decisions as the in-process engine, is a smart way to keep the two surfaces comparable without pretending they’re identical.
One question: for the “allow” case, you verified a real cat executed with real output. Did you also add a negative control that proves the harness can detect a no-op? For example, deliberately running a case that should do nothing and confirming it gets flagged as a failure. Given the theme of the article, I’d be curious whether the harness is tested against its own silent-failure mode.
To answer your closing question: the most patient bugs I know of are the ones where a retry loop swallows errors and reports success after the final attempt, so everything looks green while doing nothing. Same family as yours. Thanks for writing up the failures in this much detail.
Silent failure modes are by far the most insidious part of building agent harnesses.
We hit this exact phenomenon when profiling local model servers under agent loops:
Building deterministic policy decision points (like Cedar) before the model dispatches tool calls is a massive step forward. Have you considered adding a pre-flight token budget / context threshold check to the policy rules?
That truncation-to-null-action failure is a nasty addition to the trap list, because the agent does not even get the chance to be wrong: the envelope runs out mid-thought and something downstream invents the next move. On your question: not today, and the honest reason is architectural. The policy engine decides per tool call across risk dimensions like action class, environment, and data sensitivity, and it never sees the prompt or the context size. The closest shipped primitives are the session budgets, cumulative ceilings like 10 files, 20 requests, $5, and crossing one returns a rate-limit decision rather than a silent slowdown. A pre-flight context check would need the harness to pass token counts in as attributes on the call, and since the rules can condition on any attribute, that is a rule pack away rather than a redesign. If you want it, file it on the agentpolicy issues and I will happily point a field test at it.