First published on techpilot.dev
TL;DR In a perfect world, we would have automated tests for everything, but in reality, there's always a compromise between code quality and costs.
I've been lucky enough to start a fair amount of greenfield projects in the past years. Looking back, I feel like writing tests had always saved time in the long run but were rather cumbersome when working on UI and smaller projects.
It goes without saying that for some apps (i.e. life-critical, mission-critical) an unit test coverage of 80%+ should be required. But, despite what most of the literature says, in some cases, unit tests are actually slowing us down. In which case, how do we decide if it's worth the trouble and keep a balance between quality and costs?
Here are a couple of things to consider.
- Debugging process Remember when you had that one awkward bug, and it took days only to reliably reproduce it? Unit tests allow you to control the state of your components in great detail, making it easier to find and fix such issues.
- Reusability and life span If you wrapped up some code in a reusable library, chances are you'll improve it frequently in the beginning. Unit tests can help you ensure no breaking changes sneak through each iteration.
- Complexity When you can't reason about a system in the wild, it helps to frame it in isolation. Unit tests do just that.
- Third-party dependencies Software requires the perfect alignment of many moving parts, some of them not under your control. When third parties act unexpectedly, it becomes almost impossible to debug your system. Mocking dependencies in your unit tests solves this.
- Costs If you think you don't have enough time to write unit tests it automatically means you're ready to trade quality for it. This might work in some scenarios (e. g. proof of concepts), but you must be aware of the implications.
- Team size You'll find that as you make your code testable, you also increase its intrinsic quality. Also, unit tests are the closest thing to written specs, and while you can manage without any formal requirements while working on a part-time project by yourself, larger, heterogeneous teams would have a hard time doing so.
Unit testing is time-consuming, especially if you have a poorly designed codebase. You can decide to skip it, and it might be the right thing to do, depending on the context, but you'll have to trade code quality in return.
Top comments (26)
I’ve found it’s two fold: shorten feed back loop and it encourages good design practice to make your code testable. I’ve also found, however, they aren’t extremely valuable when your code has quite a few external dependencies (api client wrappers, AWS, GCP). Testing for regressions and business logic makes sense, but not everything needs to be unit-tested and shouldn't be. There should be deliberate value from that test resulting in saved developer time. 100% test coverage is a constraint on time, tightly couples the implementation to the test and usually results in tests that don’t tell you anything valuable. Integration tests are more intensive and do require a bit of time to run, but provide a bit more feedback. Like everything in tech, I’ve started to view them as tools. Right one for the right case to ensure I’m confident in the code I’m shipping.
I usually aim for 80% coverage from unit tests alone. 100% is definitely not achievable without wasting time for real. Problem with integration tests is that, before you know it, people start using it to test business logic, just because in a way, is more convenient (not easier or less verbose, just more convenient: no mocks/stubs, flows that translate 100% to user flows, tools that are more familiar, blackboxed etc)
In any case, I liked your comment because you're pointing out that in the end all of them are tools and each project is different.
Unit Testing actually helps me writing my code. It's like a rubberduck in a certain way. And there is no need for TDD do get this advantage out of it. On top of it there is a lot of code in the backend that is no fun to test manually at all. Therefore I can see why people skip it on the frontend if everything seems to work after the hot reload.
And high maintenance is a myth in most cases.
Couldn't agree more. Strict TDD is a myth as well, at least the way it's presented by the book: fully writing test cases before everything.
I'm inclined to think nobody does that. Instead, you get a synergy between the test and the code and they sort of drive each other. When the code is immature, the tests are as well. They beautifully grow together.
And yet so many (I mean all of them) tutorials articles have 0 test in it. Devs hear one thing but see another. This is a sad state of affairs.
I think this is an important aspect. It applies to schools as well, at least in my case. You learn how to write code 3+ years, but learn how to test it one semester, and even then, very little hands on practice, only, theory. At the end of it, no wonder the benefits are not clear and you fail to form a healthy habit of testing.
If things are hard to reason about in isolation, they are impossible to reason about as a system. Unit tests tell us at an atomic level what works and under what assumptions it works given a large number of fast running assumptions. System tests tell us that something doesn't work, somewhere based off a very narrow, slow running set of assumptions.
Anyone still arguing against unit tests is either a complete genius programmer or a dangerous developer. I've met very few genius level developers in my time...and they all prefer unit tests.
Based on your analogy, all successful projects delivered on time are either built by complete geniuses or fully covered by unit test cases. Well, in the past 20 years of my journey in insurance, corporate banking, supply chain, retail data science and so on about roughly 80% of projects didn’t bother too much about unit test cases and are highly successful. Fact is most of them are built by normal devs who know how to consciously write code. They don’t consider themselves as geniuses, including myself.
You can be a dangerous developer and deliver on time. "On time" isn't the only marker for Software. There's also quality. You can deliver utter tripe on time, every time then need to spend 60% of your time doing rework.
How do you know the developers in my team are dangerous? Fact of the matter is, none of them are dangerous. They are rather smart. We know how to get mileage with ITs, E2Es and regressions and shadow the need to write 1000s of useless unit tests. We delivered solid products and not just us. I have seen an entire generation do that. Quality can be achieved by different tactics. Anyone who is saying there is only one way to achieve quality is unfit.
I never said there was only one way to achieve quality. I said that by not unit testing, it is much harder to ensure quality. How do I know your developers are dangerous? I don't. I know that they can't possibly cover all testing scenarios in integration or end-to-end because that is an infinite number of tests for most applications. So either they understand the code and interactions so well that their base level and interaction assumptions are all correct most of the time or they are fooling themselves and they've been lucky. Writing proper unit and integration tests takes away that mental load and ensures that there are no assumptions that are waiting to be tested.
I agree on the fact that there has to be UTs to cover anything that has medium level complexity or higher in the codebase. But what I have seen in the recent years is devs are writing what we used to call as "checksum" test cases. They write test cases that test, for example, a function x() {} calls functions a() and then b() and then c() and assigns so and so value to a variable. What they are essentially doing is writing a hascode protection for x(){}. In this scenario, when another dev wants to refactor x() they have to rewrite the entire test case. So the problem here is, unit tests written are not testing the output of x(), rather they are forcing x() to be written in a specific way. Tests should focus on what, not how. Thats where unfortunately many projects are failing the overall goal. Even in our current system (6 years old), we have 25+ angular apps in a mono-repo and a whopping 10,000+ UI unit tests. We continue to see considerable amount of defects. When we finally ran a stat on how effective the existing UTs are, we found they only prevented about 8% of defects during development. If I were a business owner and spending so much money on testing, I would expect that % to be much higher. On the other hand, we found a solid reliability on our E2Es and ITs. E2Es and ITs are harder to write, but they provided us the best ROI so far.
Your example isn't clear but either it's...
A method which calls three others and assigns a value outside the unit which is just testing that value was set on a mock. The unit test is describing that the unit passed responsibility of assigning a value elsewhere. We just need to check the value is correct.
A method which calls three others and assigns an internal value which is a unit test that checks a unit has a certain value after x() is called. The unit test is describing that calling x() mutates the state of the unit.
A method which calls three others and returns a value which is a unit test to check the return value. The unit test is describing that a value is expected when this method is called.
In either of these you could run 1000 tests in a split second. You will never reach that kind of coverage on E2E and if you do it will cost you a hugely unreasonable time investment. All of these tests focus on aunit as a black box and therefore a What and not a How so not sure what your complaint on that front is.
It sounds like your E2Es are providing the best value because your units or your unit tests, likely both, are badly designed.
For the most part, they are a waste. They have very low ROI. I have seen numerous young devs getting obsessed with unit tests and TDD and end of the day it only satisfied their desire, not much in stabilizing the system. On the other hand, integration and E2Es have provided best bang for the buck. If devs are so relying on unit tests, it’s a sign that they are incapable of writing solid code, or prone to write complex logic for simple problems or just simply craving for the dev community buss and a sense of fake self pride. This is what I have seen after working at Enterprise Architecture capacity across 5 organizations in last 10 years, with some systems having more than 200 micro services and 50+ UI apps and monorepos having close to 10,000 test cases running under multiple suites. Learn to write quality code, think of all possible control flows while writing code. If you are spending more than 10% of your development time in writing or maintaining unit tests, you are already lost. If a developer is writing code that causes consistent bugs, he is not a good developer. As simple as that.
I usually start with integration tests and only break down to the unit level if necessary.
Integration tests are clearly important. However, unit tests are easier and faster to write. You don't have to reason about other subsystems, you don't have to manage artefacts (e.g. clear the database) and so on. With enough unit tests, you can have less integration tests, testing only what they're suppose to test: the integration (not business logic)
I think it depends on the tech stack. Working with frameworks like Laravel or Adonis.js writing integration tests is a joy.
Integration test with JS are also easy to write both in frontend and backend. I find writing unit tests a bit more effort as you have to mock a lot of components. My personal opinion is aiming for more of Integration and E2E tests in frontend/backend both. Inspired by this article.
In my experience, unit tests are crucial for pieces of code that can be tested as an independent unit. I believe as soon as you start to mock all sorts of dependencies in order to write a unit test, you are not testing the truth anymore.
With modern, a lot more reliable testing tools, I've seen huge benefits in writing more end-to-end tests and focus on unit tests only where it makes sense.
A single unit test covers a small area of the code whereas a single end-to-end test covers a larger area of the code and with that, has the potential to catch more bugs.
It takes time to set up a fully automated end-to-end test environment, but I've seen it worthwhile many times over the past few years.
To visualize my thoughts:
Diagram source: excalidraw.com/#json=5186401523990...
Problem with end-to-end tests is that you can't easily test outliners because often you don't control all the ends. Covering all cases is many times hard or even impossible (you want to test what happens with your (micro)service if another service it depends on fails in a certain way)
Also, they are much more expensive.
In the end, both unit, end-to-end and integration testing need to coexist, but in my experience, having a solid unit tested base, can save you lots of trouble/code higher in the pyramid.
Also, you're right, when unit testing, you're not testing the whole system, you're testing the unit, the function or class and assume that if all units work, the system works as well. This assumption can be wrong, that's why integration, end-to-end and manual testing exists parallel.
Ultimately, for me, cost and time are the main reasons I fancy the unit tests first approach, although I agree it's also a matter of team/organization/project/tech stack etc
Well said. We follow a very similar strategy to produce an effective and yet defect free product that also meets the timelines. For really complex functions, especially the ones that have multiple permutations of inputs and outputs, we write unit tests. For example, lets say we have a complex math function that resolves inputs from 15 different sensors and produces > 10 different variations of outputs. Manual testing in this scenario is tedious and discouraged. For simple scenarios, we use ITs, E2Es and regression to cover a large area. Best of both worlds.
It took me a really long time to grasp how I should do unit tests. I finally understood them as a mean of testing that I'm writing the correct code, and not testing that the code is returning the correct result. The latter is done by integration tests.
I learned it the hard way: I worked at a startup that had 0 automated tests and saw its products fail on production. Then I started looking for how I should do automated tests. The easiest ones seemed integration tests, as they test that the result of the code is correct, plus I didn't have to worry about this weird thing called Mocking. Then, my integration tests grew bigger and bigger, and started taking more and more minutes to finish running. And they failed randomly.
Today, I am going through a refactoring process and writing unit tests now that I have experienced all this first-hand - and for the last few days, I have already caught a lot of hidden semantic bugs here and there (even typos!). It all really became easier when I understood how mocks work.
To recap:
That's my opinion!
One issue where people don't write test is "How do you know if a test works in a unit test?" and "I don't get paid to write test". I write tests when the client doesn't want a major breaking changes in the UI before going to the next phase. If 1 functionality gets added/modified, it affects that test cases. I'd simply wait for his/her final decision not to change any feature or test will not be written.
I'd advice for starters that when writing a test, always assume the language specifications will go wrong(even if you're writing typescript, it will always go wrong without type validation first in production).
The challenge with unit testing, is having effective unit testing.
Many groups use it as a box checking exercise. Basically using the code as a design to implement the testing, this results in significant effort for zero benefit.
The challenge is educating developers to implement from the use case perspective over the code perspective.