“If you’re incompetent, you can’t know you’re incompetent … The skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is.”

David Dunning [NYT]

Ever heard of the Dunning-Kruger-Effect? If not, you might want to change that – it’s a great scientific foundation to make fun of people who – for example – use buzzwords, obviously without knowing their meaning. It also gives you some academic ammunition when complaining about the technical incompetence of management, project lead or sales. By using the term and throwing it towards whoever you think is your opponent you can show your superiority. And isn’t it fun to insult people when they don’t even notice being insulted? Clearly a proof of your intellectual dominance, because you don’t fit into the group of incompetent people, do you? You are clever, reflected and open-minded enough not to fall to the traps of human psyche, aren’t you?

For those who don’t want to read through the linked Wikipedia-article, have this (very simplified) description:

The Dunning-Kruger-Effect describes a psychological phenomenon that people who are low-skilled in a certain topic are the most likely to think they know a lot and are skilled at least “above average”. This is due to the fact that their lack of knowledge on a subject also means they can’t recognize how much they don’t understand.

A low-skilled person can overcome these effects by education and increase of knowledge on the topic.

The other side of the effect is, that highly skilled people are the most likely to underestimate their own competence and tend to be less confident about their expertise.

There’s a lot of information and commentary about the Dunning-Kruger-effect on the internet, often in combination with fingerpointing towards extreme examples of the phenomenon (you can read “Trump” between the lines here). Most of us have seen people whose lack of experience and knowledge on the subject they presented made us feel uneasy while they themselves seemed pretty confident and happy.

And I agree that it’s fun to watch the effect on others in its full glory, but if you calm down and think about it, it can suddenly become very serious (I guess that’s the reason why the work of Prof. Dunning and Kruger was awarded by the Ig Nobel Prize for “first making people laugh, then think”).

The problem is – this is not a phenomenon reserved for others. And it’s not something only present in ridiculous and extreme examples.

Narratives

I used to think I know a bit about security and how a secure password should look like. Then I watched this great presentation by James Coplien – and was ashamed on my own ignorance (While I highly recommend to watch the presentation I will now spoiler one of its examples).

I thought the following was a pretty good and strong password:

27dfJl!#uzB

It has everything it needs, doesn’t it? In fact, the following password is some billion times more secure:

Black pony wednesday noon meal.

I was shocked when I realized that I fell to the Dunning-Kruger-effect in an area where I didn’t expect it and in a way I was not aware of. It wasn’t some ridiculous over-confidence, easily recognized in public. It was sneaky and therefore much more frightening. It made me think a lot about what other things might exist where I’m terribly wrong and was also the reason why I wanted to write this blog post.

How comes this? Why do we believe 27dfJl!#uzB is a strong password? James Coplien comes up with a good explanation (the last thing I’ll copy from him in this post, promised): It’s a good narrative. It sounds good. Passwords hard to guess by humans should be hard to guess by computers, right? That’s wrong. But it’s a good story. We love good stories.

We love things that sound good and – from the first glance – logical. We also love stories which either confirm what we wish – or what we are terribly afraid of (say hello to conspiracy theories or refugees being the source of random problems). And therefore we want to believe.

There are numerous examples of good narratives we are likely to believe but when we increase our knowledge about the subject we – at least – start to realize that the topic might not be as simple as we initially thought it was:

• Passwords hard to guess for humans are hard to guess for computers
• Stronger winters disprove the theory of global warmth
• People who work more get more done
• Children need to be in nursery as soon as possible to develop better social skills
• TDD leads to better design
• The customer doesn’t know what he really wants

Generalization

Another possible trap of the Dunning-Kruger-effect seems to be generalization – at least that’s what I think after reflecting and analyzing my own and the behaviour of other people I met during my career.

We often seem to believe that a subject we are quite skilled at is much wider than it actually is.

I’m dealing with Oracle databases for over 12 years now, but does that mean I am skilled in dealing with other RDBMS? Does it mean I can expect to be acting on the same level of expertise when suddenly dealing with MySQL or SQL Server?

I can’t. There are thousands of things I didn’t know when I started to dive into SQL Server development – things I didn’t know they even existed.

Try to create hierarchical data structures in Oracle – it will be a quite simple task given CONNECT BY PRIOR. Try to do the same with MySQL and you will have to grow knowledge about Closure Tables, Adjacency Lists and Modified Preorder Tree Traversal – things you didn’t even know they existed before.

It’s easy to fall to the Dunning-Kruger-effect by generalization. Your actual knowledge and expertise can be outdated, very specialized on a certain use-case, on a certain programming language or framework. Here are some questions, not to blame or fingerpoint but to make you and me think:

• When you have a lot of experience developing websites with wordpress, does it make you an experienced web developer or does it make you an experienced web developer for wordpress?
• When you hit expert level in Java 1.5 – does it mean you’re a Java expert or an expert for Java 1.5?
• When you master one specific programming language, does it make you an experienced developer or does it make you an experienced developer in that particular language?
• When you grew three children, does it make you an expert in parenting or an expert in parenting these specific three children?

To get expertise in a more general subject, there is no way around mastering several of its more specific sub-topics – what means we first have to learn that there are different sub-topics.

Defense

So how can we defend ourselves against overestimating our skills when we are unaware of our lack of skill and knowledge?

I’m far from experienced in this subject and very likely to fail in my conclusions, but after thinking and reflecting about the topic for quite a while I think I can at least write down what I’ll try to do and keep in mind for myself in 2018 (yes OF COURSE this is a reflective old-year-has-gone-new-year-should-be-better post, didn’t you notice?)

First of all I think humility is a skill highly undervalued for developers while – in my opinion – being one of the most important. I’ll try to improve it. To expect that the bug you’re confronted with is not someone else’s fault but that you might have done a mistake yourself will greatly help you with debugging in most cases. It also makes you much better company to your team members by the way. And to expect not being perfect is a great defense against the Dunning-Kruger-effect.

Most topics are not simply black and white. They come in all shades of gray, sometimes even different colors appear once you take a deep dive. In my experience discussions, statements and advices which simplify a topic down to “eat or die” are very likely to be victims of the Dunning-Kruger-effect. Dogma is a substitute for knowledge and a pretty strong warning signal for illusory superiority. Getting better in recognizing such situations and don’t become evangelized easily by an opinion or trend seems like a great goal to me as well as searching for simplification in your (my) own opinions.

Training to change your mind might also be very beneficial for personal growth and defense against being ignorant. It’s also related to humility again.

I remember one thing Pettr Holmström (Senior Software Architect at Vaadin) said during a presentation (can’t find the link): “If you create a piece of architecture – encourage others to challenge your decisions!” It impressed me deeply and I think it’s a very wise statement. If others challenge your opinion or suggestions – don’t be mad but welcome it as chance to find things you might have missed. Use the possibility to iron out your thoughts and explain why you think it should be done this way. And be open to change your mind.

I also want to increase the influence of contradictory opinions in 2018. If you are sitting in your nutshell you are easy prey to Dunning-Kruger-effect. Read what others write, challenge their opinions, widen your horizon by reading different, opposing comments on the same topic and therefore increase your overall knowledge. Don’t stay in your ivory tower – get in touch with others.

Management and Leadership

This is a rather difficult topic (especially given a good chance my own bosses will read this article) but I really want to write down my thoughts about Dunning-Kruger and its impacts on management, leadership and employee-employer relation in a development environment.

I’m not a leader, I’m not a manager and I’m not an entrepreneur. Therefore I’m obviously not qualified to speak from that end of the relationship. But being employee for nearly 20 years might be qualification enough to speak about my experiences, expectations and thoughts from the other end.

A survey of Monster.com from 2014 showed, that 84% of the questioned employees (US, Germany: 83%) thought that they could do a better job than their boss. Monster.com concluded, that therefore most employees don’t have respect for their bosses – which is a terrible foundation for productive work.

I believe that this situation has much to do with the Dunning-Kruger-effect – on both sides. On the one side there are the employees, probably very competent in their specialized field, pretty aware of every single technical mistake their managers make in chats, presentations and interviews. Knowing at most theoretically how a (or this particular) company works and what’s needed to be successful as a leader they are very likely to over-estimate their own management-skills significantly – especially for they most likely don’t have all information and its necessary context available.

The amount of pressure and psychic burden that comes with being responsible for the health of a company and therefore the income of much more people than just yourself is also very likely to be overseen or just not recognized due to lack of experience in that topic.

On the other side being a manager or leader often means that you don’t have the depth of knowledge on a specific (technical) topic to understand the amounts of dependencies and risks you don’t even know about. You are therefore in constant danger of the Dunning-Kruger-effect, especially because people are expecting you to make decisions – and might hestitate to correct your (probably low-skilled and wrong) opinion because you are their boss.

So how can we overcome this difficult situation which can lead to catastrophic impacts on our company? Let me suggest some ideas:

Humility, honesty and respect for each other is – in my opinion – key. Employees, be aware that you are very likely to suffer from Dunning-Kruger-effect and be confident in your manager doing his/her best. Managers, be confident in your experts because also due to the Dunning-Kruger-effect they are very likely to be much less confident in their own skills than they should and could be.

Dunning and Kruger state, the best weapon against illusory superiority is education and more knowledge about the subject. So a great way to overcome its dangers is to ask. Employees, ask for more context if you don’t understand a management decision. Managers, ask your employees why they are suggesting something or criticising a decision. Skilled people (on both sides) will be able to logically explain why they are doing and judging a certain thing the way they do. Unskilled people will struggle to do so.

For incompetence in a specific topic can hardly be noticed by the affected people, critique and debate are immensely important for a healthy, growing company. To be able to do so, managers and employees both need to learn not to take (work-related) criticism personally but “encourage others to challenge your decisions” and opinons. As a leader, you got the additional responsibility to create a suitable environment of honest and safe debate.

Do something

The last thing I want to improve to defend against over-estimation of my skills: Do something. Don’t stop with theoretical hypotheses – start doing it. Reality will most likely strike you if you’re wrong and you will – again – increase your knowledge on the subject, which is in the end the best defense against being incompetent and not knowing it.

I heartly invite you to contact me via twitter, e-Mail, comment or whatever about your experiences and thoughts on the topic.

I wish you all a great year 2018.

This post was first published on Clean Database Development