Introduction
If you go to Google and search for "measuring software developer productivity" you will find a whole lot of nothing. Seriously -- nothing.
Well, okay, not exactly nothing. You'll get a ton of links. But almost all of the links you find will talk about how measuring the productivity of software developers can't be done effectively. Some people will even argue that it shouldn't be attempted at all. Some others will describe techniques to measure developer productivity, that, well, everyone else knows don't really work.
There have been many valiant attempts to measure developer productivity, but all seem to end in less than successful territory. (We all know to laugh at "Lines of Code" as a productivity measure). Virtually any objective measurement you can apply to the software development process can be "gamed" into submission.
There are just too many variables to account for everything that goes into the development process. Measuring any one of them, or even a combination of them, simply cannot begin to capture effectively everything that is involved. You can't pick a surrogate thing to measure, but because of human nature, developers will react in ways that will skew the measurement.
Many studies have been done on this topic, with little to show for it. Every software development manager who has to fill out a developer's evaluation or determine who gets bonuses has struggled with this. Many a leading expert and guru has thought about and researched this thorny topic.
Experts Agree
And not one of the experts has come up with a good idea or method to measure individual productivity. There are plenty of good and effective ideas on measuring team and project productivity, but there is a severe if not complete shortage of ways to measure individual performance and productivity.
For instance, Robert D. Austin wrote an entire book on this subject called Measuring and Managing Performance in Organizations and came to the conclusion that unless you can measure 100% of something -- that is, every aspect of it -- then there is no point in measuring at all. If you attempt to measure less than 100% of the things involved in a given activity, you will get what he refers to as "dysfunction", or what you and I might call gaming the system. And since software development is an incredibly complex and creative endeavor, it's virtually impossible to measure every aspect of it. Often it quite literally involves sitting and staring at a computer screen. How do you measure the productivity of that?
Here's Martin Fowler's conclusion in an article he entitles " CannotMeasureProductivity":
"This is somewhere I think we have to admit to our ignorance."
Neil McAllister of InfoWorld writes in an article entitled "The Futility of Developer Productivity Metrics"
"For most other companies, however, it might be best simply to forget about the idea of measuring developer productivity and rely instead on tried and true methods. That's right: A highly effective, productive developer workforce is often the result of high-quality, effective management. Unfortunately, nobody has developed a metric for that yet. Coincidence?"
"It's rather hard to measure programmer productivity; almost any metric you can come up with (lines of debugged code, function points, number of command-line arguments) is trivial to game, and it's very hard to get concrete data on large projects because it's very rare for two programmers to be told to do the same thing."
Steve McConnell, a widely regarded academic, author, and consultant says:
"So while I see the value of measuring individual performance in research settings, I think its difficult to find cases in which the effort is justified on real projects."
Like I said, it's hard to find a leading expert making the case for objective measurements, and easy to find plenty that say you shouldn't even try.
So I don't think that it's a tough leap to agree that there is no objective way to measure productivity in software developers. I know this is controversial, but it seems quite clear to me that software is more art than science, and measuring productivity in art is, well, simply not done. One does not tell Michelangelo that it took him too long to paint the Sistine Chapel because he should have been painting 1.4 square feet per day instead of 1.2. That would be preposterous.
So, What Are We to Do?
Anyone who is charged with managing and leading developers is left in a quandary. We want to be fair, but if everything is subjective, how can we be?
The fact that there is no objective measure of developer productivity doesn't mean you can't measure it. It just means that you have to measure it subjectively . And we do that, don't we. You can look around your shop and just know who the productive developers are. There are probably developers you know that simply crank out good code faster and fix bugs more efficiently than the rest. For ease of discussion, I'll give this developer the name "Jason".
And just how do you know Jason is more productive and more valuable? You just do. You can just tell. You know it when you see it. But if you stop and think about it, Jason has some characteristics that you can figure out and write down.
What I mean is that if you stop and examine Jason and his work, there are things that you can say about him that describe what he is all about. They are subjective, sure, but you can write them down and give an "objective" measure to these subjective things.
A Modest Proposal
So yeah, what I am proposing is a way to measure developer productivity and value via a subjective set of measurements. Think of it as a way to qualify the notion of "you just know that Jason is productive".
Here is the list:
Productivity
- Does the developer get a reasonable amount of work done in a given period of time?
- Is the developer's velocity on bug fixing sufficient?
Engagement
- Is the developer dedicated to his/her craft?
- Is the developer committed to delivering software on time?
- Is the developer dedicated to company success?
Attention to Quality
- To what degree does the developer's code work as designed?
- Does the developer thoroughly test code and believe it to be correct before checking it in?
- Do a minimal number of bugs get reported against his/her code?
- Does the developer write unit tests for all new code?
- Does the developer follow the Boy Scout Rule and leave a module cleaner than it was before he or she worked on it?
Code Base Knowledge and Management
- To what degree does the developer understand the code base assigned to him/her?
- Does the developer take responsibility for his/her team's code base, improving it at every opportunity?
- Adherence to coding guidelines and techniques
- Does developer's code routinely meet coding standards?
- Do code reviews reveal a minimum of problems and discrepancies?
- Does the developer use Dependency Injection to ensure decoupled code?
Learning and Skills
- Is the developer constantly learning and improving his/her skills?
- Does the developer show a passion for the craft of software development?
Personal Responsibility
- Does the developer first assume that the error lies within his or her code?
- Does the developer understand that he or she is solely responsible for their code working correctly?
- Does the developer take pride in their code, ensuring it is clean, tested, easy to read, and easy to maintain?
Again, these seven items are all subjective measurements. All the questions require the evaluator to make subjective judgments. But a good lead/manager is presumably in their position because of their good judgment about such things, and so they should be able to make fair and accurate assessments about Jason's productivity using these "measurements".
Most of the questions ask for a yes or no answer, but all can spark a discussion with management or even the developer being evaluated. You could use these measures on a formal evaluation form. Whatever you do with them, it's probably better than the "just know it when I see it" approach you are probably taking now.
Conclusion
In the end, any evaluation, judgment, or other assessment of the productivity or value of a single developer will be subjective. No one has yet found an objective way to measure individual productivity in our excessively complex profession. But I am saying that it is possible to qualify and make judgments about a specific set of criteria as a small and humble step towards being able to do the thing that our industry has so far found very difficult.
My suggestion is that we simply accept reality and embrace the subjectivity -- and lack of objectivity -- in measuring developer performance.
Top comments (5)
I don't think it is this black and white. Loc are too easily dismissed in this industry together with a lot of other metrics in favor of something even worse: opinions and gut feelings. Sure, Loc are not a great metric but they are better than measuring nothing. And it's not the only metric. You can look at number of commits, size of those commits, etc. Frequency of commits per time unit, tickets closed, test coverage, etc. Together they tell a story; especially when you track them over time. It's usually not the whole story and in the end it is just numbers. But they tend to even out over time and big changes tell you something just happened.
One glance at the github statistics of any project will usually tell you who are the movers and shakers and who are definitely not having a lot of impact. When metrics vary by magnitudes between developers, it's pretty hard to ignore. In my experience the numbers can vary dramatically between developers doing similar work in one project. 10x is pretty rare, but 3-5x within one project is pretty common and it is usually not temporary.
With those kind of differences, effectively 70-80% of all the code is being written by a handful of people. Amount of code varies of course with technologies used, quality of that work, etc. But it varies particularly with uncertainty and confidence. Somebody that is struggling will not get a lot done and the quality of what they get done might not be great either. If somebody is struggling a lot, their impact is going to be pretty low; or even negative. It's going to show in the numbers.
That doesn't necessarily mean you have a slacker but you do have an issue with somebody needing a little help in the form of coaching, pairing or training, or a gentle nudge to a different type of role.
I agree with the approach, but I would modify the terminology. Measurement evokes a sense of objectivity, which the introduction describes as difficult to impossible to apply. I'd call your rubric more of an evaluation. It's subjective but also transparent. If done well, it describes what you value as a manger and an organization and can be used to have constructive discussions with subordinates. At the same time it requires a degree of involvement with them that LOC and commit-counters might prefer to avoid due to a lack of time, interest, and/or people skills.
LOC are objective, easy, and make for pretty progress charts but they are too easily gamed and encourage the wrong goals
My personal experience with LOC came after a large refactor that eliminated bugs and improved reliability, but cut LOC by 30%. In the minds of LOC counters, I had just reduced the value of the product and was "rewarded" accordingly. So, full disclosure, I'm not a fan and In my experience they are worse than nothing when they encourage bad code over good. I've also worked with "movers and shakers" who just churned the code, not really advanced it.
You can't really apply process-based metrics to something until you have a repeatable process with consistent units of input and output.
There are truly objective metrics out there now, which take into account the contextual difference every developer works in, that analyse revisions from multiple dimensions such as it's Volume, Complexity and Interrelatedness altogether.
As Jilles said, it's true that gut feelings and opinions are part of the process with other metrics but now there are measures that eliminate that altogether. It's possible to directly compare productivity (one metric) and maintainability (another metric from a quality perspective) of code provided by a developer, which starts to make comparison of developer effort on every project fair and allows everyone to move forward together to improve the rate at which code is delivered and simultaneously advance it.
They're useful as they give you an indication of where to look, especially if you're going to be working on different projects with many teams wouldn't you say?
Anyway the metrics I was talking about earlier and I've come across so far is provided by BlueOptima (blueoptima.com).
Without a doubt, this topic raises many doubts and questions, mainly because variables of a quantitative and qualitative nature are mixed. At Scope we started asking ourselves how complicated would it be to measure how much time it takes for a developer, depending on their skill level, to learn the documentation of a specific framework? It would be pernicious on our part to try to generate a single answer to this question.
We at Scope -where we provide metrics and visualizations so that a team leader can know the complete workflow of their teams-, we know all this casuistry and, therefore, we have developed an algorithm to generate qualitative value for the work done by a software developer. Through this algorithm, which you can read in this article, we weigh variables that directly affect the quality of the code of a repository, through the Pull-Requests.
Our speed metrics give us appropriate indicators on the entire flow and linked to the quality of the tasks measured through the algorithm, conclusions can be drawn on how well a developer is working and what exactly the workflow is, where they are bottlenecks, etc.
Furthermore, it is not only a matter of measuring individual productivity, as they have mentioned in some answers, but also measuring the progress of projects over time. That is why at Scope we provide metrics both at the developer level and at the repository level.
We try to create an approach based on data and metrics to measure in a way as close to reality as people work in software development.
Love this post @nickhodges . Recently when I start with a new teams, I run a workshop to agree how we will measure individual "productivity"/"performance". We go through the usual things, LOC, num of PRs, bugs, before we settle on a binary metric of contributing or not form via 360 feedback - especially when considering the logistic overhead and "policey" feeling of monitoring individuals.
Team productivity is easier as it includes the other disciplines which influence your work (product, analysts, design etc.), can look at number of bugs, how soon bugs fixed, DORA, etc. But I would say the best metric is tied to the company goal/vision - how many customers have you helped with X - fairer as different disciplines can be involved, and aligns individual with the company vision (what matters, results). Your team may have failed most of their projects, but one of their successes brings in most of the profits and customers.
Before looking at performance, it may be worth keeping in mind "hygiene":
After those things are addressed you can better spend your energies on measuring productivity or figuring out new ways to do it. As a "by-product" of the above points on good "hygiene" you'll find real improvements in your teams and individuals performance.