DEV Community

Cover image for We Renamed One Button and Broke 63 Tests: A Test Maintenance
vivek
vivek

Posted on

We Renamed One Button and Broke 63 Tests: A Test Maintenance

Here's the diff that started it:

- <button id="submit-btn" class="btn btn-primary">Submit</button>
+ <button data-action="confirm" class="btn btn-brand">Confirm</button>
Enter fullscreen mode Exit fullscreen mode

A designer wanted consistent naming across the design system. A frontend developer made the change in five minutes. It looked harmless, and visually it was.

By the time the pipeline finished, the QA lead, Anjali, was staring at a wall of red:

FAILED  checkout.spec.ts      > places an order
FAILED  profile.spec.ts       > updates address
FAILED  onboarding.spec.ts    > completes wizard
FAILED  invoices.spec.ts      > downloads invoice
... (59 more)

63 failed, 112 passed, 175 total
Enter fullscreen mode Exit fullscreen mode

Not one of those 63 failures was a bug. The product worked perfectly. The tests were broken.

This case study follows what happened next: the real cost of test maintenance, the decision framework the team used, and a practical plan to cut maintenance work without losing coverage. Reading time: roughly 6 minutes.

The Background

Detail Value
Product Multi-tenant HR platform
Team 14 engineers, 2 QA engineers
Test suite 175 Selenium/Playwright E2E tests
Maintenance load ~35% of QA time spent fixing tests

Anjali kept a simple spreadsheet tracking where her team's hours went. When she finally totaled it for the quarter, one line stood out:

Writing NEW tests:               22%
Investigating real bugs:         27%
Triaging false failures:         16%
Fixing broken locators/waits:    35%
Enter fullscreen mode Exit fullscreen mode

More than a third of the QA team's capacity went to keeping existing tests alive rather than creating new value. That's the hidden tax of locator-based automation.

Why Locator-Based Tests Are Fragile

Traditional scripted tests find elements through selectors. A typical page object looks like this:

// pages/CheckoutPage.ts
export class CheckoutPage {
  constructor(private page: Page) {}

  submitButton = '#submit-btn';
  couponInput  = 'input[name="coupon_code"]';
  totalLabel   = 'div.summary > span.total-amount';

  async placeOrder() {
    await this.page.click(this.submitButton);
    await this.page.waitForSelector('.order-confirmation');
  }
}
Enter fullscreen mode Exit fullscreen mode

Every one of those strings is a contract with the DOM, and nobody told the DOM it signed one. Here's how the fragility scales:

1 UI change  x  N tests that touch that element  =  N broken tests
Enter fullscreen mode Exit fullscreen mode

If your "Submit" button appears in 63 tests (login, checkout, profile, onboarding), a single rename is 63 failures. Page objects reduce the edit effort to one file, but they don't eliminate the detection and triage effort: someone still has to open each failure and figure out whether it's real.

The Decision Framework

Anjali presented three options to the team. Here's the comparison she used:

Option Upfront cost Ongoing cost Risk
A. Keep patching locators None High and growing Team burnout
B. Add data-testid everywhere Medium (devs must adopt) Medium (needs discipline) Drift when teams forget
C. Move to intent-based, self-healing tests Medium (pilot + migration) Low Tool evaluation needed

Option B is genuinely good practice, and you should do it regardless. Stable test IDs make any suite more resilient:

// Stable, explicit hooks for tests
<button data-testid="checkout-confirm" className="btn btn-brand">
  Confirm
</button>
Enter fullscreen mode Exit fullscreen mode

But the team noticed a problem with relying on it alone: it requires every developer to remember, on every change, forever. The suite's stability depended on human consistency.

So they went with a hybrid: adopt data-testid conventions and pilot a tool that doesn't depend on a single hard-coded locator in the first place.

The Pilot

They picked their 20 most maintenance-heavy tests and ran them through an AI-driven QA platform. This is where BotGauge's AI testing platform entered the picture. The pitch that made them try it: tests are written as plain-English scenarios, and the platform finds elements using multi-modal AI rather than a brittle chain of CSS selectors, with self-healing when the UI shifts. You can run a similar pilot on your own flakiest tests to judge it for yourself.

The same checkout scenario, expressed by intent:

Scenario: Customer places an order
  Given I am signed in as a customer with items in my cart
  When I go to checkout
  And I apply the coupon "WELCOME10"
  And I confirm the order
  Then I should see an order confirmation
  And the total should reflect a 10% discount
Enter fullscreen mode Exit fullscreen mode

Notice what's missing: no #submit-btn, no div.summary > span.total-amount, no explicit waits. The step "confirm the order" doesn't care whether the button says Submit or Confirm, or what its ID is.

The "rename test"

To validate the claim, the team did the most honest experiment possible. They reproduced the original incident on a branch:

git checkout -b experiment/rename-submit-button
# apply the exact same diff that broke 63 tests last month
git apply rename-button.patch
npm run build && ./scripts/run-pilot-suite.sh
Enter fullscreen mode Exit fullscreen mode

Results on the 20 migrated tests (illustrative):

Legacy scripted suite (same 20 tests):  14 failed, 6 passed
Intent-based suite (20 tests):           0 failed, 20 passed
Enter fullscreen mode Exit fullscreen mode

That experiment, more than any vendor deck, is what convinced the skeptical senior engineers. The lesson is to replay your own worst incident against any tool you evaluate.

Rolling It Out Safely

They avoided a big-bang rewrite. Their sequence:

Sprint 1  Instrument: tag each test with maintenance cost (edits per month).
Sprint 2  Migrate the 20 costliest tests. Run old and new side by side.
Sprint 3  Compare: false failures, runtime, and bugs caught by each.
Sprint 4  Retire migrated legacy tests. Plan the next 40.
Ongoing   New tests are written intent-first. Legacy shrinks on its own.
Enter fullscreen mode Exit fullscreen mode

A tiny script helped them pick what to migrate first:

# maintenance_cost.py
import subprocess, collections

# Count how often each spec file was edited in the last 90 days
log = subprocess.check_output(
    ["git", "log", "--since=90.days", "--name-only", "--pretty=format:", "tests/"],
    text=True,
)
counts = collections.Counter(line for line in log.splitlines() if line.strip())

for path, edits in counts.most_common(10):
    print(f"{edits:>3} edits  {path}")
Enter fullscreen mode Exit fullscreen mode

Files at the top of that list are your highest-ROI migration candidates: they're the ones costing you the most to keep alive.

The Results After One Quarter

Metric Before After
QA time spent fixing broken tests 35% 8%
QA time spent writing new tests 22% 41%
False failures per week 18 3
Time from UI change to trusted CI result Half a day Same pipeline run
Designer-requested UI tweaks blocked by "it'll break tests" Several None

That last row mattered more than the team expected. Engineers had been quietly avoiding small UI improvements because of the test fallout. Removing that friction improved the product, not just the test suite.

Takeaways

  1. Track maintenance as a first-class metric. If you don't measure it, 35% of your QA capacity can vanish unnoticed.
  2. Adopt data-testid regardless. It's cheap, it helps any tool, and it documents testable surfaces.
  3. Don't depend on human discipline alone. Systems that need everyone to remember something forever will eventually fail.
  4. Replay your worst incident. Any evaluation of a testing tool should include reproducing the change that hurt you most.
  5. Migrate by pain, not by alphabet. Start with the tests that cost the most, not the ones that are easiest.

Try This Today

Run the maintenance_cost.py script on your own test folder. If the top five files account for most of your edits, you've found your pilot candidates. Then ask the question this team asked: "If a designer renamed our most-used button tomorrow, how many of our tests would fail?"

If the honest answer is "a lot," that's your sign.

What's the most absurd reason one of your tests has ever failed? Tell me in the comments.

Top comments (0)