Agile is Difficult Because of Difficulty

Here's a statistic which I found by hastily googling. According to the Standish Group 2015 Chaos Report,

• 19% of IT projects failed
• 52% were "challenged"
• 29% were successful

Those numbers haven't changed much from the previous years. What do "failed", "challenged", and "successful" mean in this context? I didn't dive deeply into that. What is an "IT project?" It almost certainly encompasses more than just software development. I'll trade precision for accuracy and say this:

• Some software development projects are successful, but not nearly enough.
• A project is more far more likely to face serious challenges than it is to succeed.
• Lots of projects fail. If a software project was surgery you wouldn't risk it unless you were certain to die otherwise. And then you'd say goodbye to everyone just in case.

Now consider the estimates found in this article which asserts that "the agile movement has failed:"

Depending on the source, it is said that 70–85% of the attempts to introduce agile project management fail.The remaining 15-30% are bobbing and rubbing against this conflict*

Do you see the similarity? Is it a coincidence that our response to failed software projects has a comparable risk of failure? Why does this happen?

Consider this Agile principle:

Continuous attention to technical excellence and good design enhances agility.

One problem that impedes or even dooms software development projects is a lack of technical excellence. That is, we write software that's buggy because we can't or don't test it, and then we can't implement requirements - planned or unplanned - because we can't maintain it.

Having experienced such difficulties we turn to Agile which tells us - correctly - that we must pay attention to technical excellence. (It might sound as if I'm finding fault with Agile, but in fact this entire post is just an observation. I'm not finding fault with anything.)

Here's the problem: Prior to our Agile transformation there was abundant information - books, blogs, videos, training - telling us how to pursue technical excellence. Still we faced challenges: We didn't know that such information existed. We knew but we didn't apply it. We learned but found that applying it is difficult, requiring persistence and discipline.

How does that our Agile transformation reduce that difficulty? For the most part it doesn't. Now we've got new books and training telling us, in effect, to read the books and practice the skills that we didn't before. (If we were learning and practicing before Agile, we're likely to adopt Agile more easily.)

My observation is so simple that it's tautological:

Adhering to a principle which directs us to do something difficult is exactly as difficult as what it directs us to do. We are exactly as likely to succeed or fail at following the principle as we are to succeed or fail at doing what the principle says, because they are one and the same.

By "difficult" I don't just mean that something is hard to understand, although that's one form of difficulty. I also mean that it is difficult for individuals, teams, and organizations to learn and do new things.

By "difficult" I don't mean impossible, either. Learning to develop software isn't impossible, only challenging. If we improve because Agile tells us to (nothing wrong with that) then the challenge of Agile is, in part, that of improving our development skills. Now Agile is challenging, too.

If we resist improving our skills - for example, we prefer creating mountains of spaghetti code with no unit tests, and we make the same mess over and over - we sabotage ourselves and our projects. We help them to fail. If we continue on that path during our Agile transformation, we sabotage that, and it is less likely to succeed for exactly the same reasons as our software projects. Neither are doomed. We simply contribute to the same succeeded/challenged/failed ratio that the industry experiences overall.

(It also doesn't help that while technical excellence is a critical factor that influences success or failure of projects, it's too easy to disregard it in favor of other details of Agile implementations like sprints and standups.)

While technical excellence is but one principle of Agile, following the other principles also takes effort. Succeeding at an implementation like Scrum or XP requires overcoming challenges. We can succeed, but that outcome is influenced by the very same factors that determined the success or failure of our projects before Scrum or XP.

I communicated with one individual who disagreed that Agile is difficult and told me that we must unlearn our old thinking and we'll only run into difficulty if we remain stuck in our old patterns. While we agree on what we must do, my observation is that unlearning our old thinking and getting unstuck from our old patterns is the difficulty.

What conclusion does this lead me to? Perhaps an obvious one:

The success or failure of software development projects without Agile depends heavily on whether we put effort into learning and applying knowledge and skills. In other words, we get out of it what we put into it.

Agile does not and cannot change that. Agile requires that we apply knowledge and skills that exist apart from Agile. Then Agile and its implementations supply additional knowledge and skills that we must learn and apply.

Think about that. If we try to adopt Agile and we haven't developed whatever habits or skills comprise "technical excellence", we could say that adopting Agile is at least twice as difficult because it encompasses both the old - which is critical - and the new. Not accounting for individual determination, would that not, on average, make an Agile transformation even less likely to succeed?

It would be a mistake to think that Agile or Scrum will make projects easier. Learning and applying skills are the opposite of easy. They require effort, just as the skills we applied before Agile required effort. That is why our attempts to succeed with Agile mirror our previous attempts to succeed without it.

Agile neither succeeds nor fails. It adds principles derived from experience to our knowledge, and learning to apply them adds to our skills. Then we succeed, fail, or fall somewhere in the middle according to the effort we put into it.

If we succeed, with or without Agile, it is because we accept and face the inherent difficulty of learning and improving. Nothing will remove it from our path. If we lack the determination to face those challenges or the wisdom to learn from the experience of others then nothing will change the outcome - not Agile and not whatever comes next.

*Still speculating - here are few reasons why Agile transformations might fail more often than IT projects:

• It's easier to abandon Scrum than it is to abandon a software project. When the Agile transformation fails the project is likely to continue in some form.
• Attempting Agile is a common response to software development challenges and failures, which means that Agile transformations start where there are already problems.

I also don't like using the words "succeed" or "fail" to describe things people do. They're rather all-or-nothing, and it's hard not to read as saying that people themselves are successes or failures. Our work doesn't define us and even apart from work I don't think it's healthy to classify ourselves or others that way. But we're talking about software projects. Calling something a failure can save months or years of wasted effort.