Build automated test suite with 150 tests and 80% code coverage in 30 minutes

Hey fellow devs! 👋

I know, I know, the title seems quite ambitious, right? But bear with me here, as we embark on a journey to create a suite of 150 automated integration tests in just 30 minutes! 🚀

In this blog post, we're going to walk you through the process of setting up an automated test suite with Pythagora. That is an open source tool that creates automated integration tests by analysing server activity without you having to write a single line of code.

The app we'll create this test suite for is an open-sourced e-commerce app, which you can find and clone from this GitHub repo: https://github.com/mohamedsamara/mern-ecommerce.

So, grab your favorite drink, sit back, and let's dive right into this together! 🤓

Oh, and for everyone who likes watching videos instead of reading, I've recorded myself building this test suite here:

Setup

Setting up Pythagora is super easy. All you need to do is run npm i pythagora and you're good to go. After that, you'll have access to two main commands for working with Pythagora.

• The first command is for capturing tests:

npx pythagora --init-command "yarn start"

• The second command is for running the captured tests:

npx pythagora --init-command "yarn start" --mode test

In both of these commands, you can replace "yarn start" with the command you typically use to run your server. And that's it! 🙌

Now that we've got the setup out of the way, let's move on to the exciting part – capturing and running our tests! 🧪 🚀

Capturing Tests

Now that we've got Pythagora set up, it's time to start capturing tests. All you need to do is run the capture command which will start your server (running with "yarn start" in my case) wrapped by Pythagora. This allows Pythagora to monitor all server activity happening when an API request hits an endpoint on the server. When this occurs, Pythagora captures the activity and creates a single integration test from it. 🕵️‍♂️

What's great about Pythagora is that it doesn't just assert the API response, but also asserts server activity such as database queries (for both Mongo and Redis) and requests to third-party APIs.

To get Pythagora to capture tests, all you have to do is make API requests to your server. I did this by interacting with the React app in my browser, but you can use your preferred method, such as Postman or cURL.

Keep in mind that to achieve 100% code coverage, you'll need to test all features in your app. Due to certain limitations with Pythagora (eg. handling random variables), you might not reach 100% coverage, but with this particular repo, we easily managed to achieve 80% coverage.

Running Tests

Once you've captured the tests you want to see in action, running them is a piece of cake. 🍰 All you need to do is add --mode test to the capture command. This will run all the captured tests located in the pythagora_tests folder at the root of your repository. When the tests are finished, you'll see a summary of the code coverage covered by the captured tests.

In my case, I achieved 50% coverage within the first 10 minutes of capturing. This is because a part of that coverage is the code that's triggered when the server is spun up.

If you're curious about the inner workings of Pythagora, I recommend checking out my Pythagora tech deep dive video but what's important to mention is that you can run tests from any environment and any machine regardless of what database that environment is connected to. In fact, during testing, your local database is never touched or used. Instead, Pythagora creates a temporary pythagoraDb database, which it uses to restore the data that was present at the time when the test was recorded. It restores the data before each test is executed and cleans it up after the test is done.

Iterating and Increasing the Code Coverage

Ensuring you've tested all existing features can be challenging, especially when it's difficult to remember them all. That's why Pythagora integrated nyc code coverage reporting. This report allows you to see all lines of code that are covered by tests. To obtain the report, simply add --full-code-coverage-report argument to the end of the test command.

npx pythagora --init-command "yarn start" --mode test --full-code-coverage-report

Then, open the file pythagora_tests/lcov-report/index.html, and you'll see all the files in your repo along with the code coverage for each of them.

Your next step is to open the files with lower code coverage, look for the red lines which indicate that they are not covered by tests, and make the API requests that you know will trigger them.

This becomes an iterative process that you can repeat as long as it takes to cover all your code with tests. Just keep Pythagora capture running and make requests. By following this approach, I was able to achieve 80% code coverage for the e-commerce repo in just 30 minutes. 🎯

Negative Tests

We're all know that having 100% code coverage doesn't necessarily mean that your test suite covers all the edge cases that could cause issues in production. This is where negative tests come into play! Negative tests are designed to create unfavorable conditions that trigger errors, helping you identify scenarios that, if not handled correctly, could break your server.

Pythagora uses GPT-4 to generate negative tests for your entire test suite with just a single command. Just add --generate_negative_tests to the end of the test command.

npx pythagora --init-command "yarn start" --mode test --generate_negative_tests

In the case of our e-commerce app example, I was able to generate 50 negative tests in a matter of seconds by running this command. 🛡️

Negative tests feature is not yet publicly available but to gain early access to this feature before its public release, you can add your email here.

Conclusion

And there you have it, folks! In this blog post, we explored creating an automated test suite with 150 tests and 80% code coverage in just 30 minutes with Pythagora.

As you might've noticed, I'm one of the creators of Pythagora and since we're keeping it open sourced, your support means a lot to continue development and improvement. If you find Pythagora valuable and want to show your appreciation, please consider starring the Pythagora Github repository. Your support helps the project grow and reach more developers like you.

We hope you enjoyed this journey and that you're now equipped with the new testing tool! Happy coding and testing, everyone! 🚀🎉