Uncertainty

We live in the world marked by four characteristics:

1. Uncertainty
2. Ambiguity
3. Incompleteness
4. Confusion

From the scientific/philosophical point of view, these traits are well documented and supported by Heisenberg's Uncertainty Principle (covering the uncertainty part), Wittgenstein's Tractatus Logico-Philosophicus (the ambiguity part), Gödel's incompleteness theorems (the incompleteness aspect), and the Second Law of Thermodynamics (the confusion caused by relentless entropy).

In this article, I am going to focus on uncertainty. To begin, we must realize that not everything in our world is at the same level of uncertainty (as the inverted pyramid depicted below illustrates):

The inverted pyramid of uncertainty is roughly split into 5 levels: from complete certainty (in the realm of mathematics) all the way up to completely irreducible uncertainty (in the realm of religion, spirituality and philosophy).

The remaining three levels in-between stretch the notion of risk by attempting to reduce uncertainty onto something that would be, to a lesser or larger degree, predictable and controllable.

We see that uncertainty increases as we move from physics to chemistry to biology, and then to social constructs such as economics, history, psychology, etc. With uncertainty the risk also increases.

Traditional fields of engineering could be placed somewhere between mathematics and chemistry. As such, engineering is leaning more toward lower areas of uncertainty.

Uncertainty in software

One unfortunate mistake that was made back in the early days of nascent software engineering was to bundle it together with traditional branches of engineering. Moving from that position, it was assumed that software engineering also tends to reside in the lower uncertainty regions. And the same methodology used for building bridges, cars etc. was knee-jerk applied to software engineering.

Today, almost 70 years later, as we've accumulated a lot of knowledge and experience in software development, we see that software engineering is a different beast. In terms of uncertainty, software engineering is more akin to the levels of uncertainty found in biology, economics and other social constructs:

It is obvious, then, that blindly applying time-tested principles, processes and practices that are prevalent in traditional branches of engineering, is detrimental to the success of software development. Those traditional methodologies are absolutely not applicable to the issues software development teams are faced with while working. Many aspects of not-yet-built products in traditional engineering are completely absent from the software products. It is therefore necessary to invent a different approach to building software.

How to deal with uncertainty in software engineering?

Since we have established that Big Plan Upfront (i.e. envisioning the 'end state') does not result in reliable software delivery, we must adjust our sights. While in traditional engineering (such as civic engineering) it makes perfect sense to first envision 'end state' in as many details as possible, such approach in software engineering is not reliable.

Engineering crews can quite easily envision a bridge that connects one river bank to the opposite river bank. It is relatively straightforward to then produce a phenomenally detailed blueprint of that bridge, which will also include very accurate estimates of the time and cost needed to build it. The level of uncertainty at that point is quite low/manageable.

No such thing is possible in the world of building software functionality that will automate some business process. There are simply too many unknowns upfront (high levels of uncertainty).

The only way to reliably proceed with building software products is to keep things relatively vague. Instead of jumping in head first and doing the precision work, teams are advised to first spend some time doing a bit of approximation work.

That approximation is, and for the most part remains, quite vague. Remaining vague makes sense because we are anticipating bumps in the road and must be prepared to turn on a dime. Expectations change, markets shift, government regulations change, there are crises looming (pandemic etc.) Charging ahead with detailed projections and hardening them into highly precise shipping code is a recipe for disaster.

The above shift in the approach tends to be quite confusing to many software development teams. Most of the developers have been educated and trained with the traditional engineering mindset as a guiding principle, and this vagueness and ambiguity truly go against every fibre in their being. It takes a lot of re-orientation before the culture shift becomes the norm; we find ourselves at this time right in the middle of this tectonic shift.

One way to illustrate this shift is to use a metaphor (while keeping in mind that all metaphors tend to stretch thin very quickly, and also tend to require a lot of suspension of disbelief). But hey, it's worth a try. On with the analogy:

Suppose, then, that we have a customer whose burning ambition is to become a musician. They have no musical training, they don't know anything about music, but for some reason would really love to take the plunge and see if they can become a good musician.

They hire us to help them achieve that goal.

If we've been trained to start the project from the envisioned 'end state', we may suggest to our client that the first thing they should do is go and purchase a musical instrument. Usually the first thing that comes to mind is a piano, or maybe a guitar etc.

We consult with our client, and learn that they have no preferences. They just want to start on something that would be easy for them, although they have no idea what instrument could that be.

The onus is now on us, the consulting agency, to guide them. A misguided move would be to advise them to splurge on a quality piano. Why would that be misguided? Because of the looming uncertainty. Our client may or may not end up liking the career of a piano player. And they may not be certain about that until significant time and energy has already been spent.

A better approach (which takes into consideration the inherent uncertainty of our client's situation), would be to advise them to start with something super modest. Perhaps something like...

One string cigar box guitar (super cheap!)

A one string cigar box guitar is a Very Small Thing (VST -- courtesy of Ron Jeffries).

"Haha!", I can only hear some engineers. "You can't be serious?!?"

Yes, I am very serious. With a bare bones minimum of investment in time and money (an empty cigar box, a stick and a piece of string), we could 'build' a lowly musical instrument in no time flat. So at the very least, our client is not out a considerable sum of money, before they even know in which direction they are going.

Furthermore, without having to wait for more than an hour or so, our client can now take a plunge and pluck that musical instrument. Get their feet wet, see how it feels for size, get some instant gratification.

The moment we get there (and it didn't really take long) we solicit feedback from our client. Do they like it? How does it feel? Getting excited? Or they hate it?

Suppose they like it a bit, and are intrigued. But now they feel like they'd like to try an electric guitar. They happen to have some old fashioned radio box which could be used as an amplifier, so they are itching to hear how would that sound.

Do we accommodate their request by suggesting they go out and buy electric guitar? Not if we care about our client, and if our job is to minimize their risk of getting into some uncertain transaction. Electric guitars are quite expensive, and who knows, our client may in the end decide they dislike the instrument.

Instead, we engage in yet another VST:

One string electric guitar (home made, super cheap!)

This time we stretch the budget a bit and go for a simple electric guitar pickup. We screw it into a plank of wood, stretch the string over it, plug it into the old radio, and voila! our client can now have a foretaste of an electric guitar.

Suppose they don't like the buzzing experience of the electric guitar and would prefer to go into learning how to play acoustic guitar. What do we do next? We find them an inexpensive beat up acoustic guitar and put only one string on it:

One string guitar (finally a real guitar!)

Why only one string? Again, we like to stick with the VST philosophy. Keep things small (very small), keep them simple, and if there is any indication that it is safe to proceed, make the next step -- only be sure that the next step is also very small.

That's the way we deal with uncertainty. Neither we, nor our client, know exactly how will their pursuit of music career play out. It is important not to overwhelm the process, and that's why we prefer to keep things modest and small.

Our client may begin to like the feeling of pressing that sole string with their fingers and plucking it with another hand. They may even learn a few simple melodies while messing around. So the natural progression will then be to add one more string, let them play around with that. Then if they don't feel lost, add more strings. Get them into playing not only single note melodies, but also double stops, chords, strumming etc.

There eventually comes a point where the client can see much more clearly where would they actually want to go. They may decide to stay on the course of mastering the 6 string guitar, or who knows -- they may realize that they'd much rather focus on trying to master bass guitar. Or maybe some other string instrument. Or maybe stretch into keyboard instruments, or into percussion?

It is great if we can guide our client on that journey in such a way not to stress them out, and not to stretch their wallet either. If we do it like that, our client will appreciate our guidance, will call us again, and will recommend our services to others.

So what about vague software development?

Same as how we deal with uncertainty in the above hypothetical (and admittedly very contrived) example, when doing software development it is much more respectful to our paying clients to approach it with a healthy dose of modesty. Sticking to the VST (Very Small Thing) philosophy will never hurt, and will actually only help a lot.

In software development, it is very advisable to eschew grandiose 'end state'. We could try it (and many have indeed tried it), but in the end we often come out of it looking arrogant and even clueless.

Humility is a virtue, and taking our paying clients on a journey with us, small step by small step, while watching the bottom line expenditures very carefully, will demonstrate sincerity and respect to our clients. They will not end up being frustrated, and will gladly call us again and also recommend us to others.