DEV Community

Cover image for The AI agent stack, layer by layer (and the one that gets skipped)
Mark Fussell
Mark Fussell

Posted on

The AI agent stack, layer by layer (and the one that gets skipped)

Draw the AI agent stack teams actually run in production and you get six layers and three verticals:

  • Layers, top to bottom: model, framework, tools and MCP, memory, durable runtime, compute and sandboxes
  • Verticals, across all of them: identity and access, guardrails, observability and evals

The AI agent stack: six layers and three verticals, with the durable runtime marked as often missing

Most of the budget and attention goes to the top three. The layer I most often find missing is the runtime between the framework and the compute it runs on, usually stood in for by a checkpointer and a retry decorator. It's also the layer that decides what a crash costs.

Here's what each layer owns, and a few opinions along the way.

The layers

1. Model

Reasoning and language. You will swap it, so keep the provider choice in configuration rather than in agent code.

2. Framework

The programming model: a loop, a graph or a crew. Pick the one your team reads most easily. Frameworks are converging on the same handful of patterns, so readability matters more than features.

3. Tools and MCP

How the agent touches the world. MCP describes the tools and leaves access decisions to whoever runs them, which is why identity and guardrails have to be verticals rather than an afterthought.

4. Memory

Conversation history, long-term facts and retrieved context. Memory is the layer most often confused with the one below it.

Memory is what the agent knows. The durable runtime is what the agent did.

Memory is what the agent knows. Facts it can recall.
The durable runtime is what the agent did. Steps it must not repeat.

5. Durable runtime (the skipped layer)

Records each LLM call and tool call as it completes, so a crash resumes at the next step instead of starting over.

Without a durable runtime, a crash after step 3 repeats steps 1 to 3. With one, the run resumes at step 4.

A checkpointer is not durable execution. Saving graph state between nodes doesn't record what a tool already did. When the retry decorator fires, the refund gets issued twice and the email goes out again.

A side benefit: a recorded run is also an eval you can replay, step by step, with the same inputs.

6. Compute and sandboxes

Kubernetes, VMs, serverless, or a sandbox for untrusted code. Runs and isolates the process, then throws it away.

Sandboxes make the durability problem sharper. They're short-lived by design: they time out, get recycled and are torn down after each task. That's the right call for isolation, and it means the run can't live there.

"Runtime" is the most overloaded word in the stack

Three things called runtime: the loop, the host and the durable runtime

Read a few agent-stack diagrams and "runtime" means at least three different things:

  1. The loop. The harness that runs reason, act, observe inside a process.
  2. The host. Where sessions are deployed, scaled and isolated.
  3. The durable runtime. What lets a run outlive the process it started in.

The word gets overloaded because it comes from language runtimes, where it means "the thing that executes code", and every product in the agent stack executes something. So everyone claims it.

The first two get an agent running. Only the third keeps it running reliably in production.

The verticals

Identity and access, guardrails, and observability and evals run across every layer

Identity, guardrails, and observability and evals aren't layers. Every layer needs to know who is calling, what it's allowed to do, what happened, and whether it was any good. Treat them as boxes in the stack and you get a different answer at every level.

  • Identity and access. Which agent is calling and what it may call. An identity per agent, deny by default per tool.
  • Guardrails. What the agent may say and do. Input and output checks, policy on tool calls, a human for risky steps.
  • Observability and evals. Every step's input, output, duration and cost, exported as OpenTelemetry to the backend you already run. Evals score those traces, before release and in production.

We've been here before

In the web services era, reliable messaging and transactions got their own specs, because a connection protocol alone was never going to make distributed work reliable. Agents are the same problem with more expensive steps.

It's good to see the Agentic AI Foundation describing an open agentic stack. AGENTS.md, MCP, A2A, gateways and sandboxes give the industry common ground for how agents connect. How agents survive failure is the natural next layer to bring into that conversation.

Close the gap

The same stack with the durable runtime filled in

Put durable execution under the framework you chose, and a crash stops costing repeat work.

For what a durable agent's runtime records, and why, see Durable Agents.

What's in your stack that isn't on this diagram? Tell me in the comments.

Top comments (1)

Collapse
 
petrj profile image
PetrJ •

Nice breakdown. One thing I’d push back on a bit: sandboxes don’t have to be throwaway. We build WBX Sandbox (Firecracker microVMs for agents), and ours go to sleep when idle and keep their files, so the next step of a run can pick up the same environment hours later. It doesn’t replace a durable runtime – it still won’t tell you whether that email was already sent – but it makes the compute layer a lot less of a reset button. Curious whether you see that as part of the compute layer or creeping into durability.