DEV Community

Cover image for Jira said Done. It wasn't, and the bug showed up later. So I made Jira check first.
Wageesha Tripathi
Wageesha Tripathi

Posted on

Jira said Done. It wasn't, and the bug showed up later. So I made Jira check first.

Exposes how AI and tired devs game CI green checks

Picture this. The ticket moved to Done on Friday. The pull request was merged, CI was green, everyone moved on.

On Monday the checkout total was wrong.

The test that should have caught it was still there. It just didn't check anything anymore. In the last commit, labelled "fix flaky test", one line had changed:

- expect(total(cart)).toBe(90);
+ expect(total(cart)).toBeDefined();
Enter fullscreen mode Exit fullscreen mode

The old line checked that the total is exactly 90. The new one only checks that there is some total. 0 passes. 900 passes. So CI went green, and Jira said Done.

Nobody lied. A developer in a hurry, or an AI agent told to "make the tests pass", took the shortest road to green. In a code review it looks like a one-line tidy-up, so nobody stops it.

The same trick, again and again

I first noticed this while building proof-of-done, a small tool that stops an AI agent from saying "done" before the tests really pass. The quickest way to a green CI was rarely fixing the code. It was one of these:

  • Skipping a test: it.skip(...), @pytest.mark.skip, @Disabled
  • Deleting a test and not replacing it
  • Loosening a check: toBe(90) becomes toBeDefined(), assert.equal(a, b) becomes assert.ok(a)

CI can't see any of this. A skipped test doesn't fail. A deleted test doesn't fail. A loosened check passes even more easily. All three make CI more green.

So I asked one question: before a ticket moves to Done, do its tests still prove what they proved before?

So I made Jira check first

Proof of Done for Jira is a free, open-source Jira app. When someone moves a work item to Done, it reads the pull requests behind it (on GitHub or Bitbucket) and checks four things:

  1. every pull request is merged
  2. CI ran and passed
  3. code changes came with test changes
  4. no test was skipped, deleted or loosened to get green

If something is wrong, Done is blocked, and it shows exactly why: the file, the line, and the test before and after.

Proof of Done on DEMO-2: test/cart.test.js line 19, the exact check before and the loose check after, and why it matters

It also shows the commit history, and marks the exact commit where CI turned green right after a test was changed:

Test timeline on DEMO-4: the first commit failed CI, the next one loosened the test and CI passed. The app marks it: CI turned green here, after the tests were changed to pass

Watch the 4-minute demo on YouTube

▶ Watch the 4-minute demo on YouTube

How it knows a test got weaker

It reads the pull request diff line by line. The main rule is simple: a loose check is a problem only when it replaces an exact check about the same thing.

  • expect(user).toBeDefined() in a brand-new test: fine.
  • expect(total(cart)).toBe(90) removed and expect(total(cart)).toBeDefined() added in its place: caught.

It also knows the honest changes that look similar, so it doesn't block good work:

  • tests moved to a new file (the same test names show up again)
  • an expected value changed, like 200 to 201 (still an exact check)
  • a test renamed, or a check made stronger

It works for JavaScript, Python, Java and Go tests. I tested it on 113 small examples: it caught all 66 shortcuts and blocked none of the 47 honest changes. On 100 real merged pull requests from three of my own repositories, it blocked none. The benchmark is in the repo, so anyone can run it with npm run benchmark.

What else you get

  • Create a task to fix this: one click turns the block into a sub-task, with the file, the line and the pull request already written in.
  • A dashboard for team leads: what's ready, what's blocked, and every catch in the project.
  • Override with a reason: if a block is wrong, a project admin can let it through, and the reason stays on the ticket.
  • A receipt on Done: a comment with exactly what was checked, for anyone who asks later.

The Proof of Done dashboard for a project: 4 checked, 1 ready, 2 blocked now, 3 tests changed to pass, 1 let through with a reason

Try it in 5 minutes

You need a Jira site where you are an admin. Nothing to download.

  1. Install Proof of Done for Jira.
  2. Create a Jira project with the key DEMO and four work items: DEMO-1 to DEMO-4. It reads a ready-made demo repository, so you don't connect anything.
  3. Open Jira settings → Apps → Proof of Done settings, pick DEMO under Turn it on for a project, and press Apply.
  4. Move each work item to Done.

All four demo pull requests are merged with green CI. Only DEMO-1 is really done. The other three get blocked, each with its reason.

Use it on your own project

  1. In the same settings page, add a line like SHOP=your-name/your-repo (or SHOP=bitbucket:workspace/repo for Bitbucket). SHOP is your Jira project key, the letters before the number in SHOP-12.
  2. For a private repository, paste a read-only token. It is stored encrypted.
  3. Turn it on for your project and press Apply.
  4. Put the ticket key in the pull request title or branch name, like SHOP-12: fix checkout total.

Proof of Done settings: one line per Jira project, KEY=repository

Work with no code, like a design or a meeting, just gets the label no-code and goes to Done.

Ask it in Rovo

Rovo is Atlassian's AI chat inside Jira. The app adds its own Rovo agent, called Proof of Done, so anyone on the team can just ask whether a ticket is really done.

Setup: nothing extra. Once the app is installed and turned on for a project (the steps above), the agent is there. You need a Jira site with Rovo, which comes with the Standard, Premium and Enterprise plans. If the agent doesn't show up right after installing, give it a few minutes.

How to use it:

  1. In Jira, click Ask Rovo at the top right. The chat opens on the side.
  2. Type @Proof and pick Proof of Done from the list of agents.
  3. Ask in plain words:
    • "Is DEMO-2 really done?" → "DEMO-2 is not done yet", with the file, the line, the test before and after, and what to fix.
    • "Is DEMO-1 really done?" → yes: the pull request is merged, CI passed and the tests were kept.
    • "Which commit turned CI green for DEMO-4?" → commit a03b524, right after the test was loosened.
    • "What is blocking this from Done?" while you are on any ticket.

Rovo chat in Jira: asking the Proof of Done agent "Is DEMO-2 really done?" and getting the reason it is not

It answers only about tickets the person asking can see in Jira.

Ask it from your own AI agent (MCP)

The same check works from Claude Code or any other MCP client:

  1. A Jira admin turns it on once: Atlassian Administration → your site → Connected apps → Proof of Done for Jira → Tools → Enable tools.
  2. Add it to your agent. In Claude Code:
claude mcp add --transport http proof-of-done https://mcp.atlassian.com/forge/487a9b05-d2bc-4e8f-98ce-2c533e95676f
Enter fullscreen mode Exit fullscreen mode
  1. Sign in with your Atlassian account when asked, then ask your agent "Is SHOP-12 really done?".

Check before merge on Bitbucket

A second small app runs the same check on the Bitbucket pull request itself, before anyone presses Merge:

  1. Install it from the Bitbucket install link.
  2. In Bitbucket: Workspace settings → Custom merge checks, turn them on.
  3. In your repository: Repository settings → Custom merge checks → Add → Proof of Done.

On Bitbucket Premium the check can be made required, so Merge stays locked until it passes.

The full setup guide for all of this is in the README.

Links

Your turn

Have you seen a test quietly get weaker just to make CI green? What did the change look like? I'd love to add real examples to the benchmark.

Top comments (14)

Collapse
 
iictp profile image
IICTP •

The toBe(90) to toBeDefined() example is the one that sticks with me — it's such a small edit and CI stays green the whole time, so nothing downstream ever notices. I also appreciated that the check tells apart a genuinely weakened assertion from honest changes like a test moving files or a value going from 200 to 201. Have you seen teams use the override-with-a-reason much in practice, or does the block itself mostly get people to fix the test instead?

Collapse
 
wst2002rk profile image
Wageesha Tripathi • • Edited

Thank you! That tiny edit with CI staying green is exactly the case I wanted to catch. So far the block mostly gets the test fixed, because the panel shows the exact line before and after, and putting it back is usually a one-line change. The override is for the rare change that is on purpose. In the demo, DEMO-2 changes rounding on purpose, so an admin lets it through with "Rounding down approved by finance, see FIN-12", and that reason stays on the ticket with their name. A new commit cancels it, so it can't quietly cover later changes. Both are in the demo video.

You clearly think about this a lot, so if you ever spin up the DEMO project, I'd be really curious what you would change. Your take would genuinely help.

Collapse
 
iictp profile image
IICTP •

That makes a lot of sense — showing the exact line before and after makes fixing the test the easier path, so the override stays rare instead of becoming a shortcut. The DEMO-2 example is a nice touch too, especially the reason staying on the ticket with the admin's name and a new commit cancelling it. Do teams ever push back on that expiry, or does the one-line fix usually win them over once they see the panel?

Collapse
 
aheadkit profile image
AheadKit •

expect(total(cart)).toBeDefined() passing on a total of 0 is the funniest and saddest line I've read this week. A test that passes when the cart is empty and passes when the cart costs 900 — that's not a test, that's a mood. The Friday-to-Monday shape of it is so familiar too: green CI, ticket closed, everyone relaxed, and then the checkout total is wrong on Monday morning. Your four checks are the right shape, but honestly the one I'd add first is the simplest: if the diff touches a test file, somebody human has to read the diff. No tooling needed, just a rule nobody can click past.

Collapse
 
maxbuildsdev profile image
Max Kaya •

The assertion-diff check is the right instinct, but text-diffing assertions has a blind spot: a test can keep toBe(90) verbatim and still stop proving anything once the thing it exercises gets replaced — the component boundary moves behind a mock, or a fixture/seed changes so the number is now produced by the stub. The line looks identical in the diff and the coverage is unchanged.

The check I'd trust more there is a scoped mutation run on the files the PR touched (stryker on the changed modules, or mutmut for pytest), which catches loosened assertions and mocks in one step, though it's minutes of CI rather than a parse. Two questions: does the app compare assertions structurally (AST, so toBe(90) → toBe(total) is caught even if the string is reformatted), and does anything stop a JS snapshot test from being re-baselined in the same commit, since that produces a big green diff reviewers skim past?

Collapse
 
paullovelace profile image
Paul Lovelace •

The receipt on Done is the bit I'd steal first. A blocked ticket is useful, but a paper trail saying exactly what was checked, on which line, is what stops the same argument happening again on Monday morning. Haven't tried the Jira app myself, but that part feels like it'd survive contact with a real team.

Collapse
 
halfcycle profile image
Halfcycle •

toBe(90) to toBeDefined() is the cleanest example of this I have seen, and "fix flaky test" is exactly the label it hides under.

One check we leaned on while building Halfcycle: when a change edits a test, run the new version against code that is known to be wrong (the bug the test was written for, or a quick mutation like returning 0). If it still passes, the test stopped testing something. That catches the weakened assertion, not only the deleted or skipped one.

Does proof-of-done look at how strong an assertion is, or mainly at tests that were removed or skipped?

Collapse
 
hitensethiya profile image
Hiten Sethiya •

the toBe(90) to toBeDefined swap is nasty bc in review it reads like cleanup. what happens when the test really was flaky and someone needs to ship a hotfix friday, is there an override that leaves a trail or does Done just stay blocked

Collapse
 
wst2002rk profile image
Wageesha Tripathi • • Edited

Thank you, Hiten, and yes, that Friday hotfix is the real test. A project admin can press "Wrong block? Override", write a reason, and Done goes through. The reason and the admin's name stay on the ticket, and a new commit to the pull request cancels the override, so the trail always matches the code that shipped. There's also a "Create a task to fix this" button, so the flaky test becomes the next piece of work instead of being forgotten.

Funny enough, your Friday hotfix is exactly the scenario I'd love someone to throw at it. If you ever play with the DEMO project, tell me whether the override felt right or got in the way.

Collapse
 
sandipan_chakraborty_c557 profile image
Sandipan Chakraborty •

Love this idea! A smart solution to a real problem developers face. Great work!

Collapse
 
wst2002rk profile image
Wageesha Tripathi • • Edited

Thank you so much, Sandipan! If you'd like to see it in action, here's the 4-minute demo on YouTube.

Collapse
 
sandipan_chakraborty_c557 profile image
Sandipan Chakraborty •

Of course! I’d love to watch it. Thanks for sharing, Wageesha!

Collapse
 
varun_bansal_4728b5726b35 profile image
Varun Bansal •

This idea is solving a real problem.

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

Thanks a lot, Varun! That was the whole goal: Done should mean the tests still prove something.