DEV Community

Cover image for Designing MCP tools for a stateful app: what I learned building an AI-editable video timeline
shansjl
shansjl

Posted on Edited on

Designing MCP tools for a stateful app: what I learned building an AI-editable video timeline

Most MCP servers wrap something stateless: search an API, read a file, query a DB. I built one for something very stateful — a multitrack video project that a human is editing at the same time. This post is about the design choices that made that workable. The project is CutVoke, an MIT-licensed, local-first video editor.

The problem: AI video ends in an MP4

Agent-driven video pipelines usually render a file and stop. When a caption is wrong or a cut is late, you re-run the pipeline or rebuild everything in a human editor. I wanted the agent's output to be an editable project.

Architecture in one picture

how it works

  • A Python engine owns a persistent project (tracks, clips, captions, keyframes, effects), stored locally.
  • Four front doors share it: a React editor (cutvoke serve, loopback only), an MCP server (cutvoke mcp --data …, stdio), an HTTP API, and a CLI.
  • FFmpeg renders previews and exports. Desktop builds bundle it.
  • There is no model inside. The agent is whatever you already use.

Lesson 1: give the agent a lease, not a lock-free free-for-all

When a person is dragging clips and an agent is running a 15-step edit, you need a rule for who writes. CutVoke uses an editing lease: the agent calls edit_lock, renews it during long jobs, and releases it on success or failure. The UI shows that an agent is editing. Commits also carry expectedRevision, so a stale plan is rejected instead of silently clobbering human work.

Lesson 2: dry runs and batches beat "be careful" prompts

Instead of telling the model to be careful, make carefulness cheap. Supported commands can be dry-run with command_preview. Related changes go into edit.batch, which is one undo step for the human. A stable commandId makes retries idempotent.

Lesson 3: let the agent see pixels, not JSON

A timeline JSON can be perfectly valid and still look terrible: captions off-screen, a title over a face, a black frame at a cut. preview_frames returns real rendered PNGs (up to 6 timestamps per call), captions and titles included. The guide tells agents to check frames around cuts, transitions and caption intervals — and to first check whether the client can actually read tool images at all.

Lesson 4: keep the tool catalog small and lazy

A video editor has a lot of commands. Dumping every schema into context hurts smaller models. Discovery is layered: editor_help (onboarding + recipes) → commands_list({"includeSchemas": false, "query": "caption"}) → editor_help({"topic":"command","command":"caption.patch"}) for the exact payload schema.

Lesson 5: objects, not pixels, for the things people fix most

The most common post-AI fix is text. Captions are individual objects: project_lookup finds a line by its text, caption.patch changes words, timing or style. Transcript-based cutting works the same way: locate a sentence, audition it, preview the impact of deleting it, confirm, undo in one step.

Try it

  • Downloads (prerelease, unsigned): https://github.com/frankShan001/CutVoke/releases/tag/v0.3.5
  • From source: pip install -e ., build the editor, cutvoke doctor --json, then python skills/cutvoke-editor/scripts/demo.py --output .artifacts/first-edit builds a 12-second edit over real stdio MCP.
  • The README has two complete edits with source footage and editable projects.

It's early and I'd value feedback on the agent API more than anything: if you've built tools for stateful apps, what would you do differently?

⭐ Repo: https://github.com/frankShan001/CutVoke

Top comments (20)

Collapse
 
demovexa profile image
DemoVexa •

Since you asked what I'd do differently: I'd make project_lookup return a stable caption ID and have caption.patch accept only that ID, never the text. Agents often fix a repeated line, like "Click Save", and a text match can land on the wrong one without any error. The expectedRevision check catches stale plans, but not a correct plan aimed at the wrong object. I'd also let preview_frames return a before and after pair for the same timestamp. A single frame tells the agent the result looks fine, while a pair shows what the edit actually changed. The lease plus one undo step per batch is the part I'd copy straight away. It keeps the human in charge without slowing the agent down.

Collapse
 
frankshan001 profile image
shansjl •

Thanks — that's a real failure mode, and you're right that expectedRevision doesn't catch a correct plan aimed at the wrong object. Today the write path is already ID-only: project_lookup can filter by text, time, or entityId, but every hit includes a stable id, and caption.patch (also caption.update / caption.remove) requires captionId and never matches on text. The remaining gap is the lookup itself: a repeated line like "Click Save" just comes back as a list, with no uniqueness error, so an agent can still pick the wrong id — I'll file that. I'll also file a same-timestamp before/after pair for preview_frames; right now it only returns current-state PNGs (up to 6 times), and command_preview is a structural dry-run that doesn't render.

Collapse
 
demovexa profile image
DemoVexa •

Good to know the write path is already ID only, that covers the worst case. For the lookup gap, one cheap option is returning the match count plus a bit of the neighbouring caption text with each hit, so an agent can tell two "Click Save" lines apart by context instead of by position. You could also let the agent pass an expected count and fail when the lookup finds more than that. That turns a silent wrong pick into an error it has to resolve before it patches anything. In practice, do agents tend to call project_lookup by text first, or do they usually come in with a timestamp?

Thread Thread
 
frankshan001 profile image
shansjl •

Your expected-count idea is close to what I ended up writing in PR #35 (github.com/frankShan001/CutVoke/pu...). project_lookup now returns matchCount and an ambiguous flag after all filters are applied. There's also an opt-in requireUnique that returns AMBIGUOUS_MATCH when more than one item matches, even with limit=1. Returning neighbouring caption text with each hit is a good next step. I haven't done that yet, and I'll add it to issue #28.

On text vs timestamp: CutVoke has no telemetry, so I can't answer from data. Most requests quote the line ("fix the 'Click Save' caption"), so the agent usually searches by text first. The updated Skill tells it to narrow with a time filter when the user gives one, or ask, instead of taking the first hit.

Thread Thread
 
demovexa profile image
DemoVexa •

Nice, matchCount plus an opt-in requireUnique is a cleaner version of what I had in mind, and keeping it opt-in means existing calls don't change behaviour. Telling the agent to narrow by time or ask, instead of taking the first hit, fits how people actually phrase these requests, since a quoted line is often all they give. One case I'd test: the same caption text on two tracks at the same moment, like a callout and a subtitle both saying "Click Save". Does the time filter separate those, or would it need a track filter as well?

Thread Thread
 
frankshan001 profile image
shansjl •

Good case, and the honest answer is no, the time filter won't separate them. Captions in CutVoke are one flat list on the sequence, not per track, so a callout and a subtitle that both say "Click Save" at the same moment both come back from textContains + atSeconds. With PR #35 the agent at least sees matchCount: 2 and ambiguous: true instead of silently patching the first hit. To tell them apart today it can ask for fields like y, fontSize or background (the subtitle usually sits low, the callout higher and styled differently), and then ask the user which one if that's still unclear. I opened your scenario as a good first issue so it gets a test and a line in the agent guidance: github.com/frankShan001/CutVoke/is.... A real track or role field for captions is the bigger fix, and I've left that open for discussion there.

Thread Thread
 
demovexa profile image
DemoVexa •

Thanks for opening that as an issue. Asking for y, fontSize or background works as a stopgap, but I'd be careful with the agent inferring the role from position, since some templates put callouts low on the frame too. A role field like "subtitle" or "callout", set when the caption is created, would make the intent explicit and give project_lookup something stable to filter on. Until that exists, showing the user both matches with their position and style seems safer than letting the agent guess. Would you set the role at creation time, or derive it from the style preset a caption uses?

Thread Thread
 
frankshan001 profile image
shansjl •

Agreed, position is a hint, not an answer, and showing the user both matches beats letting the agent guess. I'd set the role when the caption is created and let a style preset only supply a default. Presets are about looks, and people restyle captions all the time; if the role came from the preset, changing a callout's font could quietly turn it into a subtitle. So: explicit role on create, defaulted from the preset when the agent doesn't pass one, and left alone when the style changes. I added your point to the issue so whoever picks it up sees the trade-off: github.com/frankShan001/CutVoke/is...

Thread Thread
 
demovexa profile image
DemoVexa •

That split makes sense to me. Looks and meaning change at different rates, so tying the role to the preset would break exactly when someone restyles a callout. Defaulting from the preset on create and then leaving it alone gives the agent a sane value without making style edits risky. One thing I'd add is letting project_lookup filter by role directly, so the agent can ask for the subtitle at a given time instead of matching on text. When a caption gets duplicated, would the copy keep the original's role, or fall back to the preset default?

Thread Thread
 
frankshan001 profile image
shansjl •

Good call on filtering project_lookup by role. That's what makes the field pay off: "the subtitle at 0:42" becomes a query instead of a text match. For duplicates I'd keep the original's role. People usually duplicate a caption to reuse one of the same kind, so a copied subtitle should stay a subtitle; falling back to the preset default would quietly change its meaning, which is exactly what we're trying to avoid. If someone wants the copy to be a callout, that's an explicit role change, like editing any other field. I added both points to the issue: github.com/frankShan001/CutVoke/is...

Thread Thread
 
demovexa profile image
DemoVexa •

Keeping the original's role on duplicate is the right call, and treating a switch to callout as an explicit edit keeps it honest. Thanks for adding both to the issue. One more edge case that might deserve a line there: splitting and merging. Splitting a subtitle in two should keep the role on both halves, but merging a subtitle with a callout is where it gets awkward. Would you block that merge, or make the agent pick a role?

Thread Thread
 
frankshan001 profile image
shansjl •

Split is the easy one: both halves keep the original's role. For a merge across roles I'd refuse by default and return a conflict that lists both roles, so the agent has to pass an explicit role or ask the user. Quietly picking one would be the same silent meaning change we're avoiding. Same-role merges just keep the role. Added this to the issue as well (#46).

Thread Thread
 
demovexa profile image
DemoVexa •

Refusing a cross-role merge by default and returning the conflict is the right call. It keeps one rule all the way through: the agent never changes what a caption means without saying so. Thanks for writing it up as #46. I think that covers the role field end to end, from lookup to duplicate, split and merge. I'll keep an eye on the repo as it lands.

Thread Thread
 
frankshan001 profile image
shansjl •

Thanks, this thread shaped the design more than you'd think. I'll post an update in #46 when the role field lands.

Collapse
 
fredabila profile image
Fred Abila •

The lease + expectedRevision + edit.batch-as-one-undo-step is the right shape for a stateful editor MCP — it treats the agent as a guest on a project a human still owns. One thing I'd add to the "see pixels" path: let preview_frames also return a short audio snippet or a peak around each cut, because a clean still can hide a click or a dropped syllable. That fits how you already treat captions as objects rather than a burned-in layer. Disclosure: I build Vidmoat, a hosted AI video editor where you (or Claude/Cursor over MCP) edit a real timeline by chat and then render an MP4 — complementary when people want that same inspectable-project idea without running the local engine. Nice that the tool catalog stays lazy and that transcript cuts can be auditioned before they land.

Collapse
 
frankshan001 profile image
shansjl •

You're right — a clean still can hide a click or a dropped syllable, and preview_frames today doesn't help with that. It only returns rendered PNGs (up to 6 timestamps, captions included). The human editor can already audition a transcript range before transcript.delete, and an agent can dry-run the impact with command_preview, but neither gives the agent a short audio snippet or a peak around the cut; audio.analyze is a whole-file loudness/BPM read, not a cut-local listen. I'll file this so the "see pixels" path can also return sound. And thanks for the note about Vidmoat — local vs hosted does feel complementary, as you said.

Collapse
 
awesomevideo profile image
AwesomeVideo •

I'm the maker of AwesomeVideo, a hosted captioning MCP server, so I've hit some of the same walls from the other side (stateless jobs, long renders). Two things that helped us, in case they map to your export path:

  1. Don't make the agent hold a call open for a whole render. Our tools wait inside the call for up to about 45 seconds, then return a job id with step, progress and pollAfterSeconds. Models poll much more politely when the server tells them when to come back; without that hint they either spin or give up early.
  2. Make retries resumable, not just idempotent. We save transcription in 30-second chunks, so a retry after a timeout only redoes what's missing. For a timeline export, caching rendered segments between cuts that didn't change might give you the same effect, since agents tend to re-export after every small fix.

The lease plus expectedRevision design is nicer than anything we have for a human editing at the same time. Curious whether you've seen agents actually renew the lease during long jobs, or whether they mostly let it lapse.

Collapse
 
frankshan001 profile image
shansjl •

Thanks — both export notes are useful. Long renders already use export.enqueue (it returns a jobId immediately) and export_job (a non-blocking poll with status plus a progress object: phase, fraction, processed time); the guide tells agents to poll about once a second with a bounded timeout. We don't yet return pollAfterSeconds, and retries are idempotent via commandId but not resumable — we also don't cache unchanged segments between cuts — so I filed those as #31 and #32. On the lease: I don't have telemetry (the editor is local-first and doesn't phone home), so I can't quote a renew-vs-lapse rate. The API supports edit_lock action=renew, TTL is clamped to 5–300 seconds, and nothing auto-renews; our demos usually acquire a two-minute lease and release it without renewing. The guide tells agents to renew before expiry and to drop the lease before a long export; if they lapse, it expires on its own, which is why I'm adding a human take-over control in #26 (draft PR #27 is the first slice, not merged).

Collapse
 
kolmopdf profile image
KolmoPDF •

The expectedRevision guard paired with transactional batches (edit.batch as one human undo step) is the cleanest way to prevent agent drift.

One operational edge case with lease locks is agent failure. When a model hits a tool error or context limit midway through, a global project lease leaves the UI locked until the TTL expires. Adding an explicit "break lease" button in the human UI, paired with a diff view of any uncommitted fragments, keeps the operator in control.

Scoping leases to individual track IDs rather than the entire project also helps. An agent can spend twenty seconds generating and previewing captions on Track 1 while a human trims background music on Track 2 without lock contention.

Collapse
 
frankshan001 profile image
shansjl •

Thanks, both points are fair. Today the lease is project-wide and the agent picks a TTL (clamped to 5–300 seconds), so if the agent dies it unlocks on its own, but the human can be stuck read-only for up to a few minutes. There's no break-lease button yet, and there should be. The agent guide tells agents never to steal a lease, but the person at the keyboard should always be able to. Track-scoped leases are a nice idea for exactly the captions-vs-music case. I opened an issue for both: github.com/frankShan001/CutVoke/is...