Separate or project repository to store tests? Which, why, and when?

Oryginally published at: https://kodziak.com/blog/separate-or-project-repository-to-store-tests-which-why-and-when

While starting to work with end-to-end tests, one of the first questions is where to store them. Alongside source code or in separate repository? Two solutions, both slightly different. Each of the approaches has their pros and cons – but which to choose?

In this short article, I've tried to describe them both. They're different, and from your perspective you've to choose which one to use, which isn't simple at all. I believe that this could help you a bit.

Project repository

Store tests alongside source code keeps everything in the same place. You've both app-related and tests-related stuff like tasks, workflows etc. in the same spot. It causes, that the developers could execute everything much easier than searching some commands in another repository. It closes the gap between the developers and test engineers (SDET's). There aren't any barriers.

When the tests fail, it would fail on the programmer branch, so it'd be his job to maintain them. That's the mental shift when things are together. You wouldn't be responsible for bringing back the tests to love after every failure. Or find the person which going to fix them or the development is blocked.

The point is, that storing everything in the same repository, there aren't any other “strange” place where some code checks the developer's job and makes it easier to maintain.

What's more, when the newest feature change the application flow and breaks tests, you don't have problem with different versions of tests and app. They're synced because of the same repository. No concern with merges and only one thing which left would be to maintain them on the same branch as the feature is and push a complete working solution. The time when tests aren't compatible with the newest application version shouldn't exist.

Example structure

<application-repository>/<root>/app

<application-repository>/<root>/tests

Separate repository

When the test code is in “another” project, and it usually breaks — the developers probably won't care about it. It's going to be your job to maintain the tests and unstuck the blocker.

Unfortunately, when the things are separated, it would reduce the collaboration between team members. If your organization have clear distinction between the developers and test engineers, a separate repository will widen that gap. One wouldn't work on each other solutions. That's really not a situation you should go. The most beneficial would be working together on both, the application code and tests, as a team.

Imagine a situation. From the developer's perspective. You've created a feature which breaks some of them and want's to merge it, fast. Obvious is, the tests would fail, so you've to set up to another repository, maintain the tests and prepare request, or just merge the feature without properly working tests. It would create a gap because it's impossible to merge both, new versions of the code at the same time.

In that scenario, each of the repositories going to have workflow according to its needs – which is good. It will separate the contexts, so the person which going to start working on one of the projects will have only things related to that. The application has tasks related to build and running the application, and in tests will contain jobs to set up the application under tests and execute them. But it has downsides too. It's just annoying to run the whole flow on local machines. You'll need to execute commands in different places, which bring up the complexity.

Of course, I didn't say that only the test's repository has to contain tests – nah. The application should have their own. The developers should write unit tests alongside the source code and execute them in pipelines.

Separating the repositories keeps more clean environment. Repositories contain only things it's concerned about. It's make easier to navigate on the daily basis. But there's a downside to this approach. You'll have more moving parts to your workflow, which makes things more complicated along the way. For example, with new developers, setup everything it's going to be a little more difficult in front of a situation, where everything is in one place.

Example structure

<application-repository>/

<tests-repository>/

Conclusion

Project repository

• Increase ownership through the team,
• Closes the gap between the developers and test engineers,
• No problem with merging, you'll always have synced versions between tests and application,
• Everything app and tests related going to be in the same place.

Separate repository

• Decrease ownership through the team,
• Widen the gap between the developers and test engineers,
• Could be a situation where tests aren't working on newest application version,
• Cleaner environment, only things concerned about are in the repository.

Reading the article and points above, it seems like a single repository is a straight winner. But it's just how it works for me, most of the time. It doesn't mean, that it's the best solution for your organization. It always depends – on your project architecture, company structure, how many people you have and how you're working as a test engineers.

The best would be to talk to your teammates and decide as a team which could work better for you. Work together on that topic. I assumed that you have in-house test engineers. If not, then, probably you should choose a separate repository for the tests.