DEV Community

Cover image for Stop Paying LLM Prices for an if Statement: A Field Guide to Decision Models in Late 2026
amananandrai
amananandrai

Posted on

Stop Paying LLM Prices for an if Statement: A Field Guide to Decision Models in Late 2026

Speed claims face skepticism

I keep coming back to one small question: why do we ask a model to write a paragraph when all our software wants is a yes, a no, or a number? This is my notebook on how Jev answered it, who answered back, and what I would try first. Written 19 November 2026.


The short version

  • On 15 September 2026, TypeSafe AI released Jev, a model that never writes text. You hand it some state (text or JSON) plus typed questions, and it hands back a choice, a score or a probability.
  • What drew me in is how ordinary the problem is. Agent loops pay frontier-LLM prices, and wait seconds, for tiny calls like routing, gating and relevance checks that software then turns back into an if statement.
  • In about eight weeks, Cloudflare, OpenAI, Microsoft, AWS, Google and PostHog shipped their own decision models or APIs, and at least seven open-source projects showed up.
  • I also want to be fair about the numbers. The headline figures (up to 193.6x faster, 444.6x cheaper) come from TypeSafe's own four workflows. Outside checks are more modest, and one viral demo did not survive being measured. So I read them as a hypothesis to test, not a result.

Why this needed to exist

Picture an agent harness as a loop: the model proposes, the harness checks, the model carries on. When I look at what the harness actually checks, it is all small stuff. Which skill should handle this? Is this bash command safe? Is that tool result relevant? Should a human see this? Is the agent really done?

Each of those is a bounded judgment, and the usual answer was to ask a general LLM and parse whatever came back. Every decision then carries a model request, output parsing, recovery code for malformed answers, and waiting in line behind the previous call. LangChain's post on building a harness with Jev makes the same argument I find convincing: let the generative model plan and write, and move the small decisions somewhere cheaper.

Three stacked layers: a reasoning LLM on top, a decision model in the middle returning a typed answer and probability, and plain code at the bottom that acts

TypeSafe's founder, Diogo Almeida, is a former OpenAI researcher and co-author of the InstructGPT paper. His bet is that autoregressive text generation is an awkward fit for software that only needs an answer from a fixed menu. The company calls this class System One models and describes the product as a function call: unstructured state in, typed probabilistic decisions out. The name nods to the Jevons paradox, which I find charming: make a judgment cheap enough and people will start making it constantly.

Most of what an agent harness asks a model is not "write me something". It is "which one?"

This is the launch thread most explainers link to, with the 20-200x faster and 40-400x cheaper framing:

How Jev works, in plain words

  • Input: a state (any text or JSON) and a map of typed questions you define in code. Version 1.14 added image input.
  • Output: only values from the options you declared, so the answer can never come back malformed. It can still be wrong, and I will keep repeating that.
  • Primitives: choice (pick one of N), score (rate against an ordered rubric) and noul (the probability that a yes/no statement is true).
  • Parallel, not sequential: every candidate and every question in a request is evaluated together instead of token by token.
  • Calibration: TypeSafe says it trains with Reinforcement Learning for Calibrated Decisions (RLCD), so a stated 0.8 should be right about 80% of the time for that kind of question.
  • Access: a hosted API at POST /v1/systemone, $0.042 per million input tokens, free output. The weights are not public.

The eight weeks that followed

Eight weeks, six vendors and at least seven open-source projects. If you blinked in October, you missed three launches.

Timeline of decision model launches from Jev on 15 September to Google Gemini Decision 1 on 11 November 2026, colored by open weights or hosted API

Every launch, date by date
Date (2026) Event
15 Sep TypeSafe AI emerges from stealth with a reported $40M seed and releases Jev (v1.13) in early access.
17-18 Sep LangChain publishes Building a Harness with Jev and ships langchain-typesafe with a guardrail middleware.
23 Sep DataCamp roundup of seven open-source Jev alternatives: Laya, Nimble, Kev, SemIf, Rizzo Flow, Von, NanoJev.
29 Sep OpenAI announces the Decisions API at DevDay. PostHog releases Jeeves, a 9B open model that reasons before it decides.
1 Oct Cloudflare releases Clef (27B) and Clef-flash (9B) with Apache 2.0 weights and an RL fine-tuning service. AWS releases the open Strands Decider 2B.
6 Oct OpenAI opens the Decisions API in public beta.
9 Oct Microsoft-Decision-1 becomes available in Microsoft Foundry. Cloudflare adds Clef-omni.
22 Oct OpenAI Decisions API generally available. TechCrunch reports TypeSafe is valued at $7.6B.
11 Nov Google launches the Gemini Decisions API with Gemini Decision 1.


How the field answered

1. The contract became the standard. State plus typed questions, returning probabilities. Cloudflare's Clef is described as API-compatible with Jev. Google's codelab shows djev-run, a server that reimplements Jev's /v1/systemone endpoint on top of the open DiffusionGemma model. Kev can run behind a local Jev-compatible server, and Pydantic AI added a TypeSafeModel so a decision agent can be swapped between Jev and an LLM.

LangChain's day-zero support for Google's API shows how far the contract travelled: it shares a base class with its TypeSafe classifiers.

2. Several recipes competed for the same dish.

Six cards showing six ways to build a decision model: purpose-built, tiny encoders, LLM plus adapter, pointer head, logit readers and a repurposed giant model

  • Purpose-built: Jev.
  • Small bidirectional encoders with a decision head: Laya and Von, a few hundred million parameters, runnable locally.
  • LLM fine-tunes with decision heads or LoRA: Nimble, Kev, NanoJev, Microsoft-Decision-1.
  • Pointer heads on a small LLM torso: AWS Strands Decider 2B swaps the text head for a tiny scoring head. PostHog Jeeves does the same on a 9B model, with a reasoning chain first.
  • Training-free wrappers reading option logits from an existing LLM: SemIf and Rizzo Flow.
  • A repurposed general model: OpenAI's first Decisions API version constrained GPT-6 Luna to structured answers and tuned it for fast first output. Its team has credited Jev as the inspiration, which I think is a lovely thing to admit.

3. The fight moved to benchmarks and fine-tuning. Cloudflare reports Clef scoring highest on 7 of 10 decision benchmarks. Google reports Decision 1 beating Jev 1.14 on 5 of 7 shared benchmarks. Independent tracker Artificial Analysis paints a tighter picture: Decision 1 and Jev 1.14 tie at 66 on its Decision Index, and Jev was faster (about 92 ms vs 211 ms per task) and cheaper per million tasks in that run.

Artificial Analysis shared that head-to-head on X:


Meet the challengers

"Everyone copied Jev" is the lazy summary. When I read the launch notes one by one, each team made a different bet. Here they are in the order they shipped.

Decision models grouped by where they run: hosted API only, hosted plus open weights, and open weights you run yourself, with the inputs each accepts

PostHog Jeeves: the one that thinks first (29 Sep)

The bet: think before you decide, and give the weights away.

  • The model: an open 9B model built on Qwen3.5-9B with a LoRA adapter and a pointer head. It credits Kev as its inspiration.
  • The twist: it writes a short reasoning chain for each question before scoring the options, then returns calibrated probabilities through a Jev-compatible API.
  • Score (self-published): 93.5% on the public JevBench tier, against 86.6% for Jev.
  • The price of thinking: roughly 0.3 seconds without reasoning, and a 3.3-second median (17.1 seconds at p90) with full reasoning chains.
  • Openness: Apache-2.0 weights, MIT code, and training code and data in the repo. You will need a CUDA GPU.

OpenAI Decisions API (announced 29 Sep, beta 6 Oct, GA 22 Oct)

The bet: you already have a giant model, so teach it to answer like a small one.

  • How it was built: it runs on GPT-6 Luna. OpenAI's API product lead has said it was not on the roadmap four weeks before release. The first version used Luna without extra training: constrained to structured answers, answering several questions at once and tuned to return the first decision faster. He credited Jev with inspiring it.
  • What you get: text and image input, preset options, true/false conditions and scores. Latency is reported at 100-180 ms, down from 1-3 seconds, and up to 10x faster than calling Luna through the Responses API. Input is $0.10 per million tokens and output is not charged.
  • Who tried it: early customers included Box, Datadog, Ramp and Speak. Speak's CTO said the time between actions fell from 1.4 seconds to 0.13 seconds.
  • How it stacks up: at its October launch, TechCrunch reported Jev ranking 3rd and OpenAI's model 14th on the Artificial Analysis Decision Index, so the pitch is speed and multimodality more than raw accuracy.
  • For scale: Vercel said nearly 13% of its paid teams tried Jev within a day of it reaching AI Gateway.

OpenAI's developer account announcing general availability (it quotes the earlier beta post):

Cloudflare Clef and Clef-flash (1 Oct)

The bet: the same API as Jev, open weights, and a way to fine-tune on your own traffic.

  • The models: the first models trained by the Workers AI team, at 27B (Clef) and 9B (Clef-flash), built on frozen Qwen backbones. Weights are Apache 2.0 on Hugging Face and both run on Workers AI.
  • Same API as Jev: switching means changing the endpoint and model name. Clef takes up to 64 questions per call, a 64K context (Jev's is 32K) and up to 4 images in the request.
  • Speed and price (vendor-reported): median latency of 209.3 ms for Clef and 38.8 ms for Clef-flash, against 524.1 ms for hosted Jev including network. Clef costs $0.24 per million input tokens and Clef-flash $0.09, versus Jev's $0.042.
  • Benchmarks (vendor-reported): a Clef model leads on 7 of 10 decision benchmarks, while Jev wins on When2Call and BRIGHT. On Cloudflare's Decision Index, Clef scores 61.21 to Jev's 57.91.
  • A real use: Cloudflare's threat intelligence team classifies domains with Clef and Browser Run in 2.2 seconds, versus 4.7 seconds for gpt-oss-120b, which returned only two categories.
  • The bigger idea: an RL fine-tuning service. AI Gateway captures your traffic, Workers AI runs rollouts, Containers act as sandboxes, a new Trainer updates the weights and bring-your-own-model redeploys the result. It starts with forward-deployed engineers.
  • Caveats: the numbers are vendor-reported, no calibration figures have been published for Clef, and it costs more per token than Jev. Clef-omni and cheaper, faster variants followed around 9 Oct.

Cloudflare's own launch post for Clef:

AWS Strands Decider 2B (1 Oct)

The bet: small enough for a laptop, with everything open.

  • The model: a roughly 2B open model on a Qwen3.5-2B torso. The text-generating head is removed and replaced by a pointer head of about 1M parameters that scores the answer options. Later checkpoints added Gemma 4 models up to 26B.
  • Where it runs: locally only, with no hosted API. About 106-115 ms median on an RTX 3090 and about 153 ms on an M3 MacBook for small tasks.
  • How it scores: 72.3% of 231 public JevBench tasks, with documented weaknesses on long multi-step documents and unfamiliar scoring rubrics. A rival open model, Mapika decider-2b, reportedly scored 76% on the same benchmark.
  • Openness: Apache 2.0, with weights, training data and scripts released.
  • The demo I like: before an agent calls a weather tool, Decider notices the location is missing, so the agent asks instead of guessing. AWS calls this "hybrid agents": a decision model chooses and an LLM reasons.

The AWS developer account introducing it:

Microsoft Decision-1 (9 Oct)

The bet: Jev's price, inside Azure, with your data kept in region.

  • The model: Qwen3.5-9B, post-trained by Microsoft for fast, single-pass decision scoring. It returns choices, scores and yes-probabilities with confidence.
  • Where it runs: Microsoft Foundry on Azure, with an option to keep processing inside a data zone. OpenRouter listed it the same day with a 32K context.
  • Price and inputs: $0.042 per million input tokens like Jev, with free output. Text or JSON only, no images.
  • Claims (Microsoft's own): top accuracy in a 36-benchmark comparison of nearly 150,000 questions, an 85 ms median, and 4.5x faster than the runner-up. One independent JevBench ranking shared on X put it sixth among API-offered decision models, two places behind Jev.
  • My read: the "already on Azure, keep my data where it is" option.

Microsoft Developer's launch thread (the benchmark figures in it are Microsoft's own):

Google Gemini Decision 1 (11 Nov)

The bet: send it anything, and ask a thousand questions at once.

  • Inputs: the most multimodal of the bunch: text, image, audio, video and PDF.
  • Scale per call: up to 1,000 questions per request, up to 255 multiple-choice options and a 256K context.
  • Speed and price (vendor): 150-250 ms end to end, $0.10 per million input tokens, no output charge.
  • Origin: a DeepMind prototype that a small team shipped about six weeks later.
  • Benchmarks: Google says it beats Jev 1.14 on 5 of 7 shared benchmarks. Artificial Analysis found a tie at 66 on its Decision Index, with Decision 1 averaging 211 ms per task against Jev's 92 ms and costing more per million tasks ($345 vs $156).

Sundar Pichai announced it on launch day:

What I take from the lineup

Hosted or open, text-only or multimodal, tiny local model or giant repurposed one: the real choice is about where your data and your agents already live. If you run on Azure, Cloudflare or Google, there is now a native option. If you want to own the weights, AWS, PostHog and Cloudflare each ship them. And if you just want the cheapest typed answer, the launch prices still favor Jev and Microsoft.

A cheat sheet matching situations to models, such as Azure with data residency to Microsoft-Decision-1 and audio or video input to Gemini Decision 1

Open-weight challengers at a glance

Model Released Base Size License Notable
Clef / Clef-flash (Cloudflare) 1 Oct Frozen Qwen backbones 27B / 9B Apache 2.0 Jev-compatible API, RL fine-tuning service
Strands Decider 2B (AWS) 1 Oct Qwen3.5-2B ~2B Apache 2.0 Pointer head, runs locally, training data released
Jeeves (PostHog) 29 Sep Qwen3.5-9B 9B Apache 2.0 weights, MIT code Reasons before it decides

Side by side

Here are the five hosted options in one table. The open-weight ones are in the table just above.

Jev (TypeSafe) Clef / Clef-flash (Cloudflare) Decisions API (OpenAI) Gemini Decision 1 (Google) Microsoft-Decision-1
Released 15 Sep 1 Oct Announced 29 Sep, GA 22 Oct 11 Nov 9 Oct
Base Purpose-built, RLCD Open models, 27B and 9B GPT-6 Luna, adapted Not disclosed Qwen3.5-9B, post-trained
Inputs Text, JSON; images from 1.14 Text, images Text, images, JSON Text, image, audio, video, PDF Text, JSON (no images)
Questions per call Not stated Up to 64 Up to 400 Up to 1,000 Not stated
Price per 1M input tokens $0.042 $0.24 (Clef), $0.09 at launch (Clef-flash) $0.10 ($0.01 cached) $0.10 $0.042
Reported latency 70-500 ms (vendor) 38.8 ms median, Clef-flash (vendor) ~150-180 ms (launch coverage) 150-250 ms (vendor) 85 ms median (vendor)
Openness Hosted, closed Open weights, Apache 2.0 Hosted Hosted Hosted on Azure

Bar chart of launch input prices per million tokens, from $0.042 for Jev and Microsoft-Decision-1 to $0.24 for Clef

Speed is messier, because everyone measures it differently:

Bar chart of reported median latency per decision, from Clef-flash at 38.8 ms to PostHog Jeeves at about 300 ms without reasoning

Prices and latencies come mostly from launch posts and change quickly, so check current docs. If I were choosing, I would also weigh the open options (Laya, Von, NanoJev): they give up some accuracy on hard cases in exchange for local control, zero per-call cost and serving you can tune yourself.


What it changed for harnesses

The pattern I see in working setups is almost a mantra: code builds the options, the decision model picks, code acts. The model is a stateless judge with no memory, tools or text output, so the harness supplies everything else. These are the seats where it shows up:

  • Router: choose which model or agent handles a request.
  • Gate: score a tool call, such as a bash command, before it runs. LangChain's AutoModeMiddleware packages this.
  • Selector: pick the next tool, skill or browser action from a menu rebuilt after every step.
  • Verifier: check whether the agent really finished, at the claimed "done" point rather than every turn.
  • Triage: send low-confidence cases to a human or a larger model.

The harness loop: code builds options, the decision model picks with a probability, code acts, plus five seats: router, gate, selector, verifier and triage

Lessons I would borrow from people who have built these:

  • Always include a "none of these" option, because the model will otherwise pick something.
  • Put facts it cannot know into the state. One builder reports that adding the current date moved an answer from 0.35 to 0.94.
  • Batch independent questions in one call, but run dependent ones first.
  • Decide what "unsure" means before you ship.
  • Benchmark against a dumb baseline, such as a constant answer. It is humbling and useful.

Of all of these, deciding what "unsure" means is the one I would settle first. Builders have shared a few starting points:

A probability bar from 0 to 1 marking reported thresholds: 0.35 to 0.65 ask a human, below 0.5 escalate, 0.85 and up before irreversible actions

Field note: a concrete example from Cloudflare's AI Engineer talk: in its build-failure demo, decision models handle tool selection and log extraction while a large reasoning model only diagnoses. That cut six LLM calls to one and about 40K tokens to 1.8K. It is a vendor demo, but it shows the division of labor I find most sensible.


Where I stay skeptical

I like this idea a lot, which is exactly why I want to be careful with it.

  • TypeSafe's launch claims were 20-200x faster and 40-400x cheaper, best case 193.6x and 444.6x across four of its own workflows. Several analysts note these are the high end of selected evaluations.
  • An academic evaluation (Deußer et al., cited in the DICE paper) tested Jev on 37 datasets: strong zero-shot performance and well-calibrated choice probabilities, but boolean outputs may need per-task threshold tuning and fine-grained classification and rubric judging remain hard.
  • A widely viewed Jev-as-context-compaction demo (a repost claimed a session fell from nearly 1M tokens to 86K) was measured by Teknium on Nous's compaction eval. At the same budget it tied simple recency. The criticism targeted the compaction strategy more than Jev, but the lesson stays with me: a demo that looks identical whether or not the scoring works proves nothing.
  • One Vercel engineer's swap of an OpenAI model behind a command-safety classifier for Jev was reported as 5-18x faster with better accuracy. I only found this in a secondary write-up, so verify before repeating it.

Cards contrasting the headline 193.6x faster and 444.6x cheaper claim with three independent checks


Open-source repos worth a look

I have not checked star counts or licenses (the DataCamp roundup lists none for most), so please verify before depending on any of them.

Project Repo Approach
Laya github.com/NandhaKishorM/laya ModernBERT-large (mmBERT for multilingual), 322M-421M params, one forward pass
Nimble github.com/bespokelabsai/nimble Qwen3.5-9B with rank-16 LoRA, training recipe released
Kev github.com/jaredpalmer/kev Qwen3.5 + LoRA + pointer head, 0.8B/4B/9B, Jev-compatible local server
SemIf github.com/TheoLeeCJ/SemIf No fine-tuning: reads option logits from an existing open LLM
Rizzo Flow github.com/Rizzo-AI-Academy/rizzo-flow Local-first, llama.cpp serving, 1.7B-4B models
Von github.com/wfzyx/von ~395M bidirectional ModernBERT, calibrated
NanoJev github.com/TianyuCodings/NanoJev Qwen3-0.6B with decision heads, small trainable replica
Clef Hugging Face Cloudflare/clef Apache 2.0 weights, also on Workers AI

The same list by size, which is the first thing I check before trying one locally:

Open decision models on a log scale of parameters, from Laya at 322M to Clef at 27B, split into laptop-friendly and GPU territory

Harness and framework support: langchain-typesafe (LangChain), TypeSafeModel (Pydantic AI), Asymptote-Labs/agent-beacon (cross-harness agent memory), VersusControl/devops-ai-guidelines (agent eval tutorial), and Google's "Getting started with Discriminative models" codelab.


How to start without fooling yourself

  1. Pick one bounded fork in your harness: a router, a gate or a done-check.
  2. Build a small golden set of real cases with labels you trust.
  3. Run it through a decision model, your current LLM and a trivial baseline.
  4. Define an escalation threshold and send everything below it to a larger model or a person.
  5. Measure end-to-end latency, cost and cache behavior, not only single-call numbers.
  6. Keep irreversible actions behind deterministic code. A typed probability is evidence, not authority.

The takeaway: a reasoning model plans and writes, a decision model judges at every bounded fork, and plain code holds the authority to act.

If you have wired a decision model into your own harness, I would love to hear where it helped and where it did not. Tell me in the comments.


Sources

Top comments (7)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The "decide what unsure means before you ship" lesson is the one most teams will skip and regret — a typed probability is only useful once you've committed to what 0.6 does. Your skepticism section is the honest part: the Jev compaction demo tying simple recency shows a benchmark that can't distinguish a good scorer from a dumb baseline. For a gate on irreversible actions, do you set the bar at the 0.85 you chart, or higher once real money is on the line?

Collapse
 
amananandrai profile image
amananandrai •

Thanks, and good catch. The 0.85 on the chart is a starting point builders have reported, not a number I validated, so I wouldn't copy it for anything with real money behind it. I'd go higher as the cost of a wrong "yes" grows, but I'd pick the number from data: take a labeled golden set of your own cases, plot precision against the threshold, and choose the point where a false approve costs less than the extra human reviews you'd pay for. That also means a separate threshold per action type, since boolean outputs can need per-task tuning.

For truly irreversible actions I'd go further than a bar. The probability only decides whether the action goes to a human or a larger model. Deterministic checks (allowlists, amount limits, confirmations) still hold the final say, and I'd re-check calibration whenever the model version or the traffic changes.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The routing framing is the key distinction — the probability decides who decides, not whether the action happens, which keeps the model out of the irreversible seat entirely. Per-action thresholds from a golden set is the honest way to set the bar; a single number across action types would just hide the cost asymmetry. Do you re-check calibration on a schedule, or only when the model version or traffic shifts?

Collapse
 
max_quimby profile image
Max Quimby •

Really like that you framed the headline numbers as a hypothesis to test rather than a result — that restraint is rare in this space. From running agent loops in production, my experience is that the per-call price was never the thing that hurt most. It was the tax around the call: parsing the answer, the recovery path for malformed output, and every gate waiting in line behind the last one. Pulling routing and relevance checks out to a typed decision model kills all three at once, which is the real win.

The part I'd push on is which decisions are safe to demote. Routing and relevance are low-regret — a wrong answer just costs a reroute. But "is this bash command safe?" and "is the agent actually done?" are asymmetric: a confidently wrong typed answer there is worse than a slow LLM, because you've lost the hedging you get from reading a reasoned reply. Did you find the decision models expose a calibrated probability you can threshold on, or is it effectively a hard label?

Collapse
 
amananandrai profile image
amananandrai •

Thanks Max, you're right that the cost around the call hurts more than the per-call price. On your question: they return probabilities, not just hard labels. Jev's noul primitive gives the probability that a yes/no statement is true, and choice and score outputs carry probabilities too. TypeSafe says it trains for calibration (RLCD), and an independent evaluation on 37 datasets found the choice probabilities well calibrated. The same evaluation found that boolean outputs may need per-task threshold tuning, so I wouldn't trust one global cutoff. Calibration is also uneven across vendors: Cloudflare hasn't published any for Clef. I haven't run these on production gates myself, so this comes from the published evidence.

I agree on the asymmetry. For bash-safety and "is it done?" checks, I'd treat the probability as evidence, not authority. I'd set the threshold from a labeled golden set, send anything below it to a larger model or a person, and keep irreversible actions behind deterministic code.

Collapse
 
glenallen profile image
Glen Allen •

Choosing between a reasoning model and a decision model should depend on the complexity and risk of the task, not just the cost per call. At IT Path Solutions, we see this separation as a useful approach to building efficient AI workflows: use specialized models for bounded decisions, keep deterministic rules for critical actions, and escalate uncertain cases when needed.

Collapse
 
gautamtalksdev profile image
Gautam K •

Agreed, a lot of LLM calls are really a classification or an if statement. Using the model to find the rule once, then writing the rule in code, saves money and makes behaviour predictable.