Testing Pyramid

Hey developers and CEOs, this article is for you.

Im going to explain you five different types of software testing that you need to know as a software developer or a CEO of an IT company.

You have probably heard of the testing pyramid. Chances are you probably haven't thought much about why the pyramid is a pyramid to start off with.

Unit Tests

Tests at the bottom of the pyramid should make up the majority of the tests in your test suite. This is where you should be focusing most of your efforts on. As we move up the pyramid, the tests become more complex and slower and they take more time to maintain. At the base of our pyramid, we have the unit tests.
Now, most developers are aware of what unit tests are and the importance of them. Developers are the one who are writing unit tests and they are responsible for that part of quality assurance. Developer writes unit tests for all the methods and functions in the code, to ensure that the program is working correctly at the lowest level.
You may wonder on the number of unit tests you have to write. That really depends on what the goal is for your testing strategy. You should aim for and test every single line of your code in your methods. If for some reason something is preventing you, then that should be a sign that your function is doing too much or you haven't written it with testability in mind.
100% testing coverage means to have test coverage of every single line of code.

Unit tests are not enough to ensure that your software is working and looking as expected. You should not assume anything if it is working at the lowest level, that it's all going to work when you put it together. That is often not the case. For that reason we introduce next level of testing in the testing pyramid.

API Automated Tests

The next level up in our testing pyramid is what we call API tests.
This is where we test each endpoint in isolation, to cover expected and unexpected behaviour. Error messages, response body, json schema, http status codes, etc.
API tests are very fast, and we should when scenario allows us aim for writing end to end tests. To ensure that feature workflows are working exactly as we expect them.

You would run them in dev environment which should be a mirror of your prod environment.

Unit tests and API tests are all generally run as part of the build process, before a release, after each git push to your working branch, after each git push to master branch or you have them scheduled to run at certain time. Depending on how your process is defined.

Frontend Automated Tests

If you're writing a web application, then these will typically be in the form of automated UI tests.
And we generally use tools such as selenium, playwright and such, to mimic user action in the UI through a web browser. The goal here is to test that everything is working as expected end-to-end, is clickable, typeable and such. These tests typically include a mix between functional testing, such as making sure that you can signup/login or a form is populated correctly, And acceptance testing, which makes sure that your application is meeting business requirements. I tend to write end-to-end tests in style what is called a Gherkin language, which follows the given-when-then pattern. Frameworks such as Cucumber or any other BDD framework allow us to execute code in this format while still having the tests non-technical-human-eye-readable.
These type of tests are significantly slower than unit and API tests and can take really really really long time to run, and typically they're not run on every single git push. Imagine you have hundreds of these tests, they can take several hours to run, so you generally need to run them overnight.

Frontend automated tests are not ideal if you want to be releasing very often in a startup style. Most QA teams split up their tests into multiple groups tagging scenarios by critical group that they can run before each deployment.
For example, most essential tests can be tagged as @smoke and you can setup to run only subset of those tests tagged with that tag after given action. This would save you time, and you would run only what you need. While full suite of tests you could schedule overnight.

It is important to note that these tests need all the components working together, and therefore they're typically run on an environment such as QA or UAT. It can take a while to have a stable set of end-to-end tests. Subtle things such as your application taking a little longer to load, or someone accidentally breaking QA or UAT environment can cause your tests to break, and therefore you generally need someone working full time on your automation tests. Maintenance is absolutely crucial, in both API and Frontend tests. Especially in Frontend tests, as they are most dependent and most fragile.

Manual Tests

As cherry on top of our pyramid, we have manual tests. These are the tests that either need human eye, human exploratory mind, or they are not worth the time in trying to automate them. Manual testing even though unfairly not respected enough play important part in the last final piece of functional and visual side of the software application.

There are certain bugs you can only find by executing manual testing.
Manual testing should be executed in various phases. While feature is developing, you want to test new things, but you also want to make sure you are executing entire suite of tests you have from before, to ensure that new code did not break existing features. That is also called regression testing.

When you find a bug in your application, it is always better to find it lower down the pyramid than near the top. Each bug found in manual testing phase makes you to run application, reproduce the issue and search for logs and try and work out where exactly your application failed. Compare that to finding a bug in your unit tests and you will be given a stack trace that shows you the exact line where the problem occurred.