DEV Community

Cover image for Durable execution for local LLM workflows: what I’m learning
Khushighosh
Khushighosh

Posted on

Durable execution for local LLM workflows: what I’m learning

Most LLM demos assume the process stays alive. Kill it midway and the plan, the tool results, and the context all go with it. I've been spending time with a few tools that change that, and this is what I've picked up.

Durable execution changes how you think about failure

Temporal records every step of a workflow in an event history. If a worker dies, another one replays that history and continues from the last completed step. Finished work doesn't run twice.

For LLM workflows this maps well:

  • Model and tool calls belong in activities. They're slow, non-deterministic, and can fail, which is what activities are for.
  • Workflow code has to stay deterministic. Replay depends on it, so randomness, network calls, and timestamps stay out of it.
  • Retries become policy, not code. Backoff and timeouts are configuration instead of nested try/catch blocks.

Local models make failure cheap to test

Running a model through Ollama keeps the loop on your own machine. No rate limits, no API bill, and killing a process mid-run costs nothing. That makes it easy to test the unhappy path often, which is where most of the real bugs are.

The trade-off is that small local models are less predictable than hosted ones. Durable execution helps here because a bad step can be retried or inspected without redoing the whole run.

You can't trust what you can't see

Retries and resumes are invisible by default, and invisible failures are the hardest to debug. Sentry's tracing is a good fit, since a trace per step shows which call was slow, which one retried, and where a run picked back up.

A small React dashboard on top of the workflow history makes this easier still. Seeing a run's attempts as a timeline beats reading logs.

Open questions I'm still working through

  • How much state can sit in workflow history before replay gets slow?
  • Which failures should resume, and which should stop the run?
  • Does durability change how you write prompts and tool definitions?

I'll share more as I find answers.

Top comments (4)

Collapse
 
max_quimby profile image
Max Quimby •

The deterministic-workflow / non-deterministic-activity split is the right mental model, and the subtle part is that an LLM call being non-deterministic is why it has to be an activity: you record the model's output as the activity result so that on replay you get the recorded tokens back, not a fresh (and different) generation. Replay a model call directly and your history diverges from reality.

On your open question of which failures should resume vs stop — the line we draw is side effects, not error type. A pure read or a deterministic transform is always safe to resume. Anything with an external side effect (writing a file, posting, sending) needs an idempotency key or it stops for a human, because replay will happily run it twice. Durable execution makes the happy path free but it makes exactly-once your problem, not the framework's.

The state-in-history / replay-speed worry is real too — we started snapshotting large intermediate payloads to object storage and keeping only a reference in history. Curious whether you've hit that ceiling yet with local models.

Collapse
 
khushighosh profile image
Khushighosh •

Thanks, Max. Splitting by side effects instead of error type is a much cleaner rule, and I’m adopting it. I haven’t hit the history ceiling yet, but snapshotting to object storage with a reference is where I’d start. Appreciate the detail.

Collapse
 
paullovelace profile image
Paul Lovelace •

The local-model point is the one I'd actually use — killing a process mid-run costs nothing with Ollama, so you finally test the unhappy path as often as the happy one. The open question I'd steal is which failures should resume vs stop — that's the decision I keep getting wrong, because resuming a bad tool call feels safe until it loops. How are you drawing that line in practice, policy or gut?

Collapse
 
khushighosh profile image
Khushighosh •

Policy, mostly, though it started as gut. The rule I’m settling on is yours: reads and transforms resume freely, and anything with an external side effect needs an idempotency key or stops. For the looping problem, I cap attempts per step and mark certain failures, like malformed tool arguments, as non-retryable, so a bad call fails fast instead of replaying. Still tuning where that line sits.