Most "AI fixes your issues" demos are a session: you run a CLI, it edits files, you look at the diff. I wanted the opposite: a standing service that sits next to a repository, picks up issues labeled bug, docs or chore, and opens a pull request — and that I can leave running without watching my LLM bill.
That's fly-swe, built on mini-swe-agent. Here's how it's put together, and what it cost on a real TypeScript monorepo.
One Machine, one process pair, one SQLite file
The whole thing is a FastAPI webhook receiver and a worker, run together by honcho on a single shared-cpu-1x Fly Machine with a 1 GB volume. Webhooks are verified (HMAC, timing-safe, fail-closed on an empty secret), de-duplicated by delivery ID, and written to a SQLite queue in WAL mode on the volume. The worker polls it.
No Postgres, no Redis, no broker. If the Machine restarts mid-task, orphan recovery picks the task back up: nothing lost, nothing run twice. The infra costs around $6/month.
The LLM never touches git
This is the invariant I care most about. The coding agent edits files and leaves them in the working tree. Then it emits a completion signal, and everything after that belongs to the harness: git add, commit, push and PR. The agent's shell gets a tokenless remote, no push credential, no gh, and an env allowlist with no secrets.
A side effect: when the target repo's AGENTS.md says "commit and open a PR", the agent follows it for style and tests but not for git, so nothing goes sideways.
Three budgets, enforced in code
-
max_iterations: a hard cap on agent steps -
max_cost_per_task_usd: per task -
max_daily_cost_usd: per repo per day
cost_tracking: strict goes one step further: if a step can't be priced, it isn't run.
When a run hits a limit, it doesn't just die. It force-pushes its partial work to a WIP branch, and the next trigger resumes from there, along with a summary of what was already tried. That summary costs zero LLM tokens.
"How I tested" is what the harness saw, not what the model said
After the agent's loop ends, the harness re-runs the test command the agent says it used, in the same environment. The PR's "How I tested" checkbox reflects that run. Anything the agent couldn't run (say, a suite that needs a database the sandbox lacks) is listed separately, labeled as the agent's report, and never checked off.
If verification fails, the harness says so on the issue and retries automatically. That happened on 3 of the 10 issues below.
Numbers from a real repo
10 consecutive issues (bugs and docs) on a TypeScript monorepo with an SDK and a docs site, using GLM-5.3:
- median cost per issue: ~$0.31; worst case $1.85 (225 iterations, two verification retries)
- typical run: ~80 iterations
- all 10 PRs were reviewed by a human before merge. fly-swe never merges.
What it isn't (yet)
Execution isn't sandboxed: the agent's commands run as a subprocess in the same container, mitigated by the env allowlist and the absence of credentials (SECURITY.md). A per-task sandbox is next on the roadmap. Ephemeral Fly Machines / Sprites are an obvious candidate.
GitHub is the only forge today, but it sits behind a provider plugin with a conformance suite. GitLab or Gitea would be a good first contribution.
Deploy in ~10 minutes: Quickstart · AGPL-3.0
Top comments (9)
Update, a week later (v1.4.0): the "honest test report" now gets a second opinion.
In the post, anything the agent couldn't run in the sandbox (say, a suite that needs a database) was listed separately and never checked off. Honest, but a human still had to go and look at CI. The $6 Machine isn't getting bigger, so I moved the full run to the forge.
v1.4.0 adds an opt-in CI monitor (
ci_monitor.enabled, or per repo):With the monitor on, the prompt tells the agent the full suite runs in CI. So in the sandbox it runs only the fast, targeted checks and lists the rest as left to CI.
Two smaller ones since the post:
protected_pathsis enforced by the harness before commit and push, not just asked for in the prompt.CI sits behind the provider plugin like everything else: GitHub is the first to implement the new port. And if you were eyeing the Gitea contribution, there's now a PRD for it.
Release notes: github.com/stateway-io/fly-swe/rel...
The tokenless-remote invariant is the part I'd copy immediately. We had exactly the failure mode you're preventing: an AGENTS.md that told the agent to "commit and open a PR" led to it learning a git alias mid-run and pushing from inside the model's shell. Splitting style-instructions from the git boundary via the env allowlist is much cleaner than prompt-engineering around it.
Question on the SQLite/WAL queue: when orphan recovery picks a task back up after a Machine restart, how do you guard the LLM side of idempotency? The git work is safe to redo, but the tokens for a half-finished 75-iteration attempt aren't — do you cap resumed attempts at the same max_cost_per_task_usd, or does a resumed task get a fresh budget?
That alias story is exactly the failure mode that convinced me prompts aren't a boundary. If the shell has a credential, sooner or later the model finds a way to use it, so it's easier to make sure there's nothing to find.
On the LLM side: the tokens themselves can't be saved, so what fly-swe persists is the work they produced, so the next attempt can continue instead of starting over.
resume.summary_on_block.resume.max_attempts.On the budget: each attempt gets its own
max_cost_per_task_usd, by design. The per-attempt cap bounds how far a single attempt can wander, the attempt cap bounds the total, andmax_daily_cost_usdsits above both.The case you describe, a Machine dying mid-run, is the one exception: there's no chance to write a summary or push a WIP, so the in-flight attempt's progress is lost. Orphan recovery puts the task back in the queue (at most
max_task_retriestimes), and it resumes from the last persisted state: the previous WIP and summary if there was one, otherwise from scratch.If you think a resumed task should carry over what's left of the previous attempt's budget instead of getting a fresh one, feel free to open an issue at github.com/stateway-io/fly-swe/issues. I'd be happy to discuss it there.
Having the outer harness independently re-run the test command is the cleanest invariant here. Most agent loops run off the rails because the model grades its own work, confusing a truncated stdout or a silent exit with a passing suite. Separating execution from verification also explains your cost distribution. The jump from the thirty-one cent median to the dollar-eighty-five ceiling is almost entirely retry search. Once verification is deterministic, spending that budget on bounded retries becomes an actual calculation rather than an open-ended gamble.
Thanks, that's exactly the reason the harness owns verification. A model reading its own test output will rationalize a truncated log or a non-zero exit, and a separate re-run can't do that.
You're right about the outlier, and the issue thread shows it. The $1.85 run was three attempts at roughly the cost of a normal run each. Every time the re-run failed, the harness posted the failing command and its output on the issue and started a new attempt with that output as context. The third attempt passed. 225 iterations split across three attempts is about 75 each, which is right around the median.
One nuance, though: in that run, the first verification failure was the environment (a module the test runner couldn't resolve), not the fix itself. That's the weak spot of bounded retries: an attempt can be spent on something the agent can't fix from inside the sandbox. That's why retries are capped at 3 and the per-task cost cap still applies. The worst case is a known number, not an open-ended one.
Better environment diagnosis before spending an attempt is on my list. A per-task sandbox with the right toolchain preinstalled should eliminate most of that class of failure.
The payback math usually hides an aggressive assumption on replacement costs. A 3-year or 5-year hardware amortization assumes the alternative API pricing stays flat. Frontier inference prices historically compress 50 percent or more year over year. If managed token rates halve over the next eighteen months, the displaced spend falls from $32,000 to $16,000, which pushes the breakeven horizon past the useful life of the node.
The other quiet tax is peak concurrency variance. Developer traffic clusters during business hours, so sizing hardware for three in the afternoon means carrying single-digit utilization across the full 168-hour week. You either queue the team during peak crunch or pay continuous power and depreciation on idle silicon overnight.
Hi. There's no hardware payback in this setup: fly-swe doesn't run inference. The $6/month Machine is a shared CPU with no GPU, and it only hosts the webhook receiver, the queue and the agent's shell. The model is a hosted API called through LiteLLM and billed per token.
That actually turns your first point around. Since there's no node to amortize, falling token prices go straight into a lower cost per issue. Nothing gets stranded, and if prices halve, the ~$0.31 median roughly halves with them.
On concurrency: the worker takes one task at a time from the SQLite queue. A burst of labeled issues at 3 p.m. simply waits its turn, and the idle cost overnight is the same $6/month Machine. For an issue-to-PR bot that's the right trade, because nobody is blocked waiting on it the way they would be on an interactive coding assistant.
That said, the economics you describe are real for teams self-hosting the model. fly-swe can point at a self-hosted endpoint via
api_base, and at that point your payback math is exactly the right question to ask.The three budgets are the part I would copy. Per task cost, iteration count and a daily ceiling cover different failures, and strict cost tracking means the agent stops instead of quietly spending more than anyone agreed to.
Keeping the model away from git entirely, with a tokenless remote, is also a good call. The agent can propose changes, but pushing is done by code you wrote and can read.
I appreciated the honest report. Retries on 3 of the 10 issues and one $1.85 outlier is more useful than a clean success rate, because it tells a buyer what a bad day costs. Deduping webhooks by delivery ID is another small choice that avoids paying twice for the same issue.
At Ascendra Ventures we write cost ceilings like these into a fixed scope before building an agent for a client. Did resuming from the WIP branch change how often the harder issues finished?
Thanks, glad the test report was useful! One suggestion, though: don’t copy the three budgets. Run fly-swe as it is, or fork it if a client’s setup needs something different. That way the caps, the retries and the cost tracking stay in sync, and you don’t have to re-learn why each one is there. 🙂
On WIP resume, the honest answer is that I don’t have enough data for a rate yet. The 3 retries in that sample were verification retries, which is a related but separate path. The fix is already committed on the work branch, and the next attempt is told to build on it, with the failing command and its output as context.
The one clean block-and-resume case I have on the same repo was one of the larger issues: lifecycle webhooks that were documented but never dispatched. The first attempt hit the task timeout at 63 iterations and about $0.43, and its partial work went to the WIP branch. The resumed attempt picked it up from there and opened the PR in 58 iterations for $0.22, so $0.65 in total. One data point isn’t a rate, but it’s the behavior I was hoping for: the second attempt didn’t rediscover the codebase from scratch, and it cost about half as much as the first.
One design detail that matters here: a timeout or budget block doesn’t requeue itself. The partial work waits on the branch, and a new trigger (a comment on the issue is enough) resumes it. That way a human decides whether an issue that already ran out of time or money deserves more. Only transient provider errors and failed verifications retry on their own, and both count against the same attempt cap and the daily budget.
Ask me again in a couple of months and I should have a real number.