Startup Testing Dilemma: E2E or Not E2E?

In a fast-growing startup I'm part of, a question sparked intense debates among senior developers: with all the setup and maintenance time it cost, is E2E testing really worth our time?

We all prioritize fast delivery, and figuring out how to keep that momentum in long term was the crux of our chats

Quick note: The other perspective would be to bypass E2E, let a few bugs sneak by QA, and patched things up when they showed in production.

The E2E Test Paradox in Startups

End-to-end (E2E) tests are often seen as the gold standard in quality assurance. They replicate real user actions and make sure everything in the application runs smoothly. But in a startup environment, they can be a double-edged sword.

Imagine investing hours crafting a detailed E2E test suite for a feature. The next week, based on new insights, that feature undergoes a revamp or is replaced entirely. Now, not only do you need to rewrite the feature, but you also have the unenviable task of overhauling the test suite. This cycle can be both mentally exhausting and time-consuming.

To counter the challenges posed by E2E testing and its associated maintenance costs, we explored an alternative: visual testing.

🧿 Visual Testing: Agile’s Best Friend

In the dynamic landscape of startups, where changes are frequent, visual testing emerges as the optimal solution. Here's why:

1. Trust in Every Change: With visual testing, changes become transparent. Approve expected updates and reject unintended alterations.

2. Minimal Effort, Maximum Coverage: Efficiently use one line of code to visually verify a full page's layout and content, eliminating the need for extensive scripts.

3. Cut the Maintenance Chain: Even if your product evolves constantly, no need to update the test code. Simply approve or reject the updated screenshots.

⚒️ Our Go-To Tool

We've been using Argos for visual testing. It's a dev tool that allows to review the changes a pull request brings. It's easy to onboard and it slides right into our workflow.

// Homepage test file. No maintenance needed! 
import { test } from "@playwright/test";
import { argosScreenshot } from "@argos-ci/playwright";

test('homepage', async ({ page }) => {
  await page.goto("http://localhost:3000");
  await argosScreenshot(page, "homepage");
});

An example of Argos build, where the dev can review the PR's visual changes.

⚖️ Balancing Act: E2E and Visual Testing

While visual tests shine in adaptability, the value of E2E tests remains unmatched for mission-critical functionalities. The key is balance. For core features with less frequent changes, E2E tests guarantee stability. For others that are in flux, visual tests provide adequate coverage without the maintenance headache.

Conclusion

In the fast-paced world of startup development, agility is critical. While robust testing remains essential, the type of testing needs to align with the product's stage and volatility. Visual testing provides the flexibility startups need without compromising on quality, making it a crucial part of any developer's toolkit today.

Have you faced similar testing dilemmas in your startup? How did you navigate them? Share your experiences in the comments.