We run a small pay-per-render web app: you upload a selfie, pay once, and a few minutes later you get a 15-second meme video with your face in it. It is one Cloudflare Worker, Stripe Checkout, R2 for files, and a paid video-generation job per order.
That last part is what this post is about. Every order starts two paid jobs (a main pass and a short segment pass that gets spliced in at the end). Together they cost a little over one US dollar per order. That is fine for a customer order. It is not fine for the orders we create ourselves.
Three kinds of orders
Once the site was live, orders came from three places:
| Kind | Who creates it | Pays? | Starts a render? |
|---|---|---|---|
| real | customers via Stripe Checkout | yes | yes |
| qa | a headless browser check that opens checkout every day and stops there | no | no (stays awaiting_payment) |
| test | us, with an admin token that skips Stripe to test the whole flow | no | yes |
The qa orders were harmless: they never get past checkout, and they carry an @example.com address so reports can filter them out.
The test orders were the problem. In launch week they were genuinely useful. Our honest "how long does it take" numbers come from them: 5:11 and 5:20 for one face, 6:57 for all five seats. But the admin path called the same start() as a paid order, so every "let's just check the order page after this deploy" also bought a real render. With scheduled jobs and post-deploy checks that each like to "test the whole flow", that adds up quietly.
The guard
The fix sits on the server, in the two places that start a render for an order. Simplified:
// Internal test orders never start a paid render unless a human approved that one run.
const paidRunApproved = (req) => req.headers.get('x-allow-paid-run') === 'approved-by-owner';
async function dryRun(env, tpl, order) {
order.dry = true; // reports and the order page can tell it apart
order.status = 'done'; // same terminal state as a real order
order.video = tpl.exampleVideo // the template's published example clip
order.error = order.video ? null : 'dry run (no example video)';
await putOrder(env, order);
}
// 1) new admin test order (skips Stripe)
if (isAdmin) {
order.status = 'paid'; order.test = true;
if (!paidRunApproved(req)) { await dryRun(env, tpl, order); return json({ url: `/o/${order.id}`, dry: true }); }
await start(env, tpl, order);
}
// 2) re-using a credit from a multi-video pack
if (order.test && !paidRunApproved(req)) { await dryRun(env, tpl, order); return json({ url: `/o/${order.id}`, dry: true }); }
Thirteen lines in the real diff. A few decisions behind them:
Guard where the money is spent, not where the tests live. We could have told every test script to "please don't render". That works until the next script, or the next person, forgets. The Worker is the only place that can start a render, so the rule lives there and every caller gets it for free.
Make the expensive path opt-in, and make the opt-in loud. A boolean like ?real=1 is easy to paste into a script and forget. A header whose value reads like a sentence (approved-by-owner) is hard to add by accident and easy to grep for in logs and code review. Our rule is that it is only sent after the owner says yes to that single run, for example when the generation pipeline itself changed.
Keep the state machine identical. A dry order ends in done with a playable video URL, exactly like a real one. So the order page, the video player, the download button and the video_ready analytics event still run end to end. Only the render is skipped. The video is the example clip already shown on the homepage, so nobody can mistake it for a fresh result.
Flag it. dry: true on the order lets the daily order report count these separately from real and paid orders, so a dry run can never inflate the numbers we judge the site by.
Find every entry point. The first version only guarded new test orders. The second path, re-using a credit from a 3- or 5-video pack, also starts a render, and a test pack would have rendered on every reuse. If your app has "redo", "retry" or "use another credit" buttons, each one is an entry point.
What dry runs don't test
The render itself, obviously. We cover that from the other side:
- real orders are monitored: any order that ends in
failed, or a splice step that logs an error, raises an alert; - when the pipeline changes, one paid test is run on purpose, with the header, after an explicit OK;
- the timings we publish come from those deliberate runs, not from dry ones.
A short checklist if you sell AI output per order
- List every code path that starts a paid job (new order, retry, redo, pack credit, admin tools).
- Put one guard in front of all of them, on the server.
- Default to the cheap path for anything that is not a real paid order; make the paid path an explicit, greppable opt-in.
- Finish dry runs in the same terminal state as real orders, so the rest of the flow stays covered.
- Flag them, and filter them out of revenue and conversion reports.
The app this came from is Tiny Desk AI Video, a fan-made parody tool for the "Outstanding" desk-concert meme (not affiliated with NPR, Tiny Desk or the performers). The guard went live on November 15, 2026.
Top comments (0)