Technical Debt - The Unseen and Surprising

TL;DR;

• We accrue technical debt in different, often unseen and innocent ways:
  • 'Unexpected' - created by unpredictable or aggressive change.
  • 'Unintended' - created by something missed at the design/development phase.
  • 'Tools and Tech' - opportunity cost of not leveraging faster, more effective tooling and technologies.
  • 'Team changes' - codebase incoherence due to number of contributors (who may no longer be part of the team).
• As with all work - get it into the backlog so it's visible.
• Weigh up not only the cost of clearing technical debt, but also the value gained from clearing quickly.
• If you can't 'sell' the benefits of a refactoring - you probably shouldn't do it.

---

'Technical Debt' (Tech Debt) is explained nicely by Wikipedia:

Technical debt (also known as design debt or code debt, but can be also related to other technical endeavors) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer.

Many roll with the financial metaphor and discuss 'interest' (time, effort) to clear the debt.

Tech debt is recognition that "we're intentionally cutting corners here". This may sound bad - but there's several great reasons to do take on tech debt - to name a few:

1. Speed to market - especially if a new product where scale and reliability are less important than early user feedback.
2. Beta testing - similar to the above - but normally to a closed group. While we may not ship the product to the public - we're willing to 'roll forward' with warts-and-all. This is very typical in Scrum teams where Products are continuously incremented.
3. Deadlines - the real kind. While generally rare - they do exist. Contractual or legal obligations may force corners to be cut.

I believe these are well understood - and often agreeable. However, I keep seeing 'Technical Debt' getting used for very different types of problems. They need different approaches to solve.

Traditional Tech Debt

Let's get the common and easy one out of the way. When knowingly cutting corners, define up-front when you expect/want to pay that debt off. Similar to how you'd take out a loan.

For example:

• "We know this won't scale" - "When we hit a sign up rate of X/Y, we'll get this into the next Sprint"
• "We want to get this out ASAP, with some known bugs" - "We'll clean these up next Sprint"
• "To meet contratual agreements - we'll do X, but we must spend the next Y Sprints cleaning up"

Generally speaking - the more 'interest' you need to pay, it's best done quickly.

Unexpected Tech Debt

Increasingly common - especially with the rise of more highly distributed, service-orientated systems. Essentially the system becomes a victim of it's own success and/or isn't refactored quickly enough to meet rapidly-changing business demands.

For example:

• Explosion of usage (e.g. increased user counts, taking an API public) means that the system is now operating way beyond it's original design. For example: Data stored may not be partitioned effectively given the new usage patterns.
• What seemed sensible at the time... Wasn't. While moving fast - things might get implemented in the heat of the moment - and look fine. The holistic view of the system is missed - and a key refactoring should have taken place to accommodate the new demand. For example: Addition of new markets introducing cross-cutting, leaky models into an architecture.

These ones are very tricky and frankly - often painful for the development team. They could start to manifest as performance issues. Or, the 'awesome team that made this business' suddenly starting to look slow and like a bug factory.

A typical phrase I use when working with a team experiencing these is:

"Knowing what you know now - how would you build it?"

If the solution is architecturally very different - you've likely got some unexpected tech debt.

As soon as these are identified - get them in the backlog. Make sure the Product Owners are extremely aware of the problem and how it's only going to get worse. Sitting down and devising some immediate (albeit hacky) mitigations while figuring out a long-term fix is often needed. Give the Product Owner options to work with.

Unintended Tech Debt

If 'unexpected' is the tech debt created in the future, 'unintended' is created in the past. This is essentially "we built with the best of intentions, but we missed something".

Again, this is becoming increasingly common - especially with the glorification of phrases like "move fast and break things".

I like to throw different ones around like:

• "Don't break the till" (or 'cash register' for our friends speaking American English 😊). If people can't pay, you don't get paid.
• "Don't break the door" - No signups, no users - not to mention reputation damage. Ever seen a busted door during a extremely successful marketing campaign? Not pretty.

There are certain areas where unintended tech debt can be very damaging - so extra care/attention is required. I'm not advocating "Big Design Up Front" (BDUF), but "care and attention to details when and where it matters".

This kind of tech debt can vary in severity based on what's affected. However, normally these come as a nasty shock to the team and can result in high-priority fixes. Not only do these lower the profile of the development team, but also ruin plans in progress. Not fun for anyone involved.

The key here is quality design and testing. Spend enough time in the design/modelling phase. What's your testing plan? Do you have adequate time to perform exploratory testing? What else can be shifted-left?

Tech and Tools

It's worth highlighting these because:

Tools and technologies evolve

What was good yesterday, may be superseded tomorrow by something 100x better. Not leveraging that can introduce debt due to opportunity cost etc.

To help overcome this - I think best thing is a healthy culture of curiosity, learning and sharing among the developers. We can't learn everything so spread the load and get talking about the highlights!

Make sure you and your team has the freedom to learn, play and discover what's new and upcoming so they can be ready to run with it.

Platforms change, as do their costs

I remember having to rent fixed kit with a fixed cost (👴). We now live in incredible times with a veritable feast of PaaS options on the table. These are constantly evolving - potentially offering (significant) cost savings both in usage and (often more importantly) saved developer time and cognitive load.

Similar to the above - a healthy culture creating awareness is key. If you're in an organisation large enough to have Architects around - this should absolutely be one of their core objectives to be on top of this.

Team Changes

We work in teams with multiple Developers. There's churn. A lot of opinions end up in the code. You end up with code that's "kind of OK", but over time has become increasingly awkward or downright dangerous to work with. Yes, we know "it needs a refactor, and better tests". But, few are willing to do it given the risks it might introduce.

Whenever I am working with teams and hear grumbles of "oh no, we need to go near that", I'll start to ask what their understanding of the problem(s) are, how they intend to fix it, and risks.

If it feels like a spike is needed to work out answers, then we'll do that.

Once we're armed with plan and possible solution(s) - I'll then ask:

• "What is your estimate if we don't take time to fix this?"
• "... and what if had fixed it, per this plan?

If the plan means smaller, more confident estimates - we likely commit to it. Especially if we know that there's a lot of change in that area coming up. Having several 5-pointers turn into a series of 2's by taking the time to 'finally fix that thing' not only feels good from a planning point-of-view, but general pride in the codebase.

'Dislike' Isn't 'Debt'

This is the one that tends to lose me most friends 😉 It's important to call it out though, because it's doing us harm.

When adding tech debt to the backlog - it needs to be based on 'this is causing objective pain' - not "I don't like it" or "this would be cool".

This will often manifest as "The Great Refactoring", only to discover one or more of the following:

1. It's in a part of the codebase that's barely touched - and no further changes needed on the horizon.
2. The developer flew solo - often creating more of the 'incoherence' problem. They dip in, changing patterns/conventions in place in just that bit - may look good for them, but the wider team now struggles with it.
3. It was an opportunity to introduce {coolest pattern/technology of the day} - often adding complexity rather than taking it away.
4. The Product Owner doesn't really 'get' it, and trusted the developer in that 'we just need it'.

This is tricky to pick up. It can be difficult to judge if a refactor adds enough value. It's important that we try though because these low/no/negative value-add refactorings cost time, energy but more importantly - trust.

As professionals we need to work with our Product Owners and the rest of the team, and be conscious of the investment (our time, opportunity cost, increases in complexity) we're proposing.

A good way of getting a sense of it is to simply do the maths! Take your daily salary - plus some for that of your pair. Guesstimate time you're going to spend on it - what would be required to get that return on investment back? This can be a real eye-opening experience - especially when you consider that some people go down the rabbit hole for entire Sprints or more.

In my experience - there's two common warning signs:

1. 'Tech Debt' Sprints appearing on plans.
2. Developer X (often a senior) doing a 'thing' for days/weeks that no-one else on the team can really explain. But "they know what they're doing".

Arguably - local leadership could/should pick this up, but we're all human. Everyone has good intentions.

So, be conscious of your investments. If you can't 'sell it' to your Product Owner (as you should), skip it (or spend more time figuring out the benefits).

To close...

Not all tech debt is created equal, much (most?) of it is unintentional. This unintentional debt often has very negative side effects when it becomes apparent. There's several warning signs we can look out for. Once discovered, get it into your backlog to review and resolve.

Understanding the cost of both paying off the debt, and delay in paying of the debt are key for prioritising.

Work with your Product Owner - if you can't 'sell' a refactoring, you probably shouldn't be doing it. Present options/trade-offs. Negotiate.

Now go create.

Happy hacking!