(Sorry, lazily reused the same graphic above.)
This article is my "attempt" to define the benefits of proper levels of testing. I had a larger project in the past that showed clear benefits from developer-level Unit and Integration Testing.
I will update the contents here from time-to-time as I get more concrete numbers.
Script Parser and Engine
~120,000+ LINES-OF-CODE
58 BUGS found before reaching production BY TESTING
1 BUG found once the code was in production BY USERS
This means ...
1 BUG per ~2,034 LINES-OF-CODE
From December 2016 until April 2017, about five months, I wrote the 120,000+ lines of code that became a parser and engine for a "user" scripting language. This language replicated (in JavaScript) one created on a mainframe with almost three decades of fine-tuning and optimization.
While writing this code, I began writing Unit and Integration Tests. The integration tests being written were designed to tests the parts of this system against each other (for example, could I pass in a script to the parser and get an appropriate Active Syntax Tree back). I was not using Test-Driven Development (TDD).
I also kept track of the number of bugs in the original code that did not make it to production ... 58 bugs found
Once the code was released into production, there were two (2) issues and one (1) bug located by users running scripts against this code. The issues were due to incorrect specifications between what I was actually designing and what was needed.
Testing Checklist
~700 LINES-OF-CODE
5 BUGS found before reaching production BY TESTING
0 BUGS found once the code was in production BY USERS
This means ...
1 BUG per ~140 LINES-OF-CODE
This application was actually the catalyst for this article. I designed an application for developers, designed to reinforce the testing ideals we were trying to propagate to the client's teams.
Proper testing, in this case, allowed me to see a refactor of the logging methodologies that made the code easier to read and understand, as well as simpler to test.
Conclusions
This article is simply my "attempt" to define the benefits of proper testing levels. Having had a larger project in the past that showed clear benefits from developer-level Unit and Integration Testing, this seemed like a clear direction to go.
I will update the contents here from time-to-time as I get more concrete numbers.
Top comments (3)
Love the scientific focus. We need to be better at tracking stuff like this because not all code has the same level of risk.
The "scientific focus" is what I was going for. This article will be one I update as I gather more information on various projects (may even include data passed on to me via reliable sources).
If we could get tooling to indicate what line or method an error occurred, maybe something to look at FIX commits, it might be possible to automate some of this, but it requires process changes and discipline. It's an interesting idea.