The Dunning-Kruger Effect In Tech

This is not the kind of post I usually write on my blog. This is more of a psychology lecture than a how-to tech tutorial. But it’s not completely irrelevant as well, because I’m going to talk about my experience with the Dunning-Kruger effect in tech that I’ve seen over the last decade.

I’ve always been interested in learning more about psychology and how the brain works. Because the field is so bloody interesting, and also, it helps me understand people, make out patterns, and make better decisions. But I never write about it, because decidedly, I’m not expert in the field, and the Dunning-Kruger effect is about understanding that very fact. But first, let’s understand what it is.

The Dunning-Kruger Effect

To put it simple, the Dunning-Kruger effect is one of the many types of cognitive bias in which people tend to believe they more smarter and capable of things than they really are. That’s probably the most smiply the effect can be explained. And if you’ve spent any time in the tech industry, you don’t want me to tell how common this is.

In his book The Dissent of Man, Charles Darwin wrote that “Ignorance more frequently begets confidence than does knowledge.” The Dunning-Kruger effect is more common than you might expect, and I’m very sure you have experienced this yourself. You might not want to believe that you are one of “those people,” but you most likely are. That’s because the Dunning-Kruger effect spares nobody.

There have been many studies done in the social psychology space around this. But the most prominent and the most discussed is the Dunning-Kruger effect, named after the researchers David Dunning and Justin Kruger. The study that they conducted by itself wasn’t really that big or wide. They performed four experiments and around 100 participants (or less) in each of them.

If you follow the Jimmy Kimmel show, you’d be familiar with the Lie Witness News segment, in which his crew go to popular locations and ask random people questions which are mostly made up, and see how they react. In one such episode, people were asked if the movie Godzilla is inventive to the people who survived the giant lizard attack in Tokyo in 1954. You’d be surprised with how people reacted.

The reason most people go with the flow is that they don’t want to appear dumb or clueless, especially in front of a camera. But in lying about it, they make themselves look dumber, especially in front of a camera. Do you see the irony? It’s natural to feel dumb when you admit that you don’t know something. But the reality is that when you admit you don’t know something, you’re opening up an opportunity for yourself to learn something new. But when you lie about it, and continue to do so, you’re not only making yourself appear dumb and ignorant, but you start developing the false confidence that you know stuff and can hold a conversation about it.

It’s not only depriving you of knowledge, but it’s also putting you on the path to dumbness, ignorance, and arrogance. You might have heard this from a lot of people before, that knowing the boundaries of your knowledge is the most important education you can have. If you are educated about what you know and what you don’t, and if are you are able to differentiate between the two, consider yourself smarter than most people and smarter than what you think you are.

How Is This Related To Tech?

Enough of the psychology lecture, you say? I got the hint that you’re not really interested in that. So let’s see how this is related to our tech industry. You might have heard and read that you need to “fake it till you make it.” We all do that, and it’s necessary at times. I did it when I was fresh out of college and new in the tech industry. I was surrounded by some really smart people, entrepreneurs kinds. And these people hold patents to their names, and I was this kid who thought he knew what he was doing.

So I did fake it, till I made it. But the problem is, some people don’t stop the faking even after making it. I see the Dunning-Kruger effect a lot in such people. It might not be right of me to talk about others, so let me tell you my story.

Early in my career, actually when I was still in my college, I learnt who to build websites using PHP, Java, and MySQL. This was a very common and popular stack (the LAMP stack) back in the days. And because I was able to write APIs for CRUD operations and design websites to perform a few UI tricks, I thought I had mastered software engineers. So in my CV, I started rating myself 7 or 8 out of 10 in the said technologies. Remember, I was fresh out of college without any industry experience.

Now, 8 years later, if you ask me to rate myself on those same technologies, I would not rate myself more than 4 or 5. It doesn’t mean that I’m a worse software engineer (or developer) or that I don’t have the same confidence in myself anymore. But it means that now I know what I know and what I don’t know.

I went through the same phase 5-6 years back when I switched from software engineering to data engineering (don’t get me started on those designations). I started with Apache Spark for a project, I was able to do a lot of stuff with RDDs in Java for that project, and thought again that it’s done, I’m an expert in big data. But as I started reading up on stuff and interviewing for data-related jobs, I again realised that I just don’t know what I don’t know.

That’s something I now say to everybody I know, that knowing what you know and realising that there’s a lot more that you don’t know is very important. If don’t know realise that, you are just making a fool of yourself, trust me.

Once I started realising this, I started reading up on this, and that’s when I first came across studies done on this. I thought, wait, I can do my own study on this. I don’t know if this is ethical or not, but whenever I interviewed people, I started asking one or two questions in the interview that I made up on the spot. This is of course related to the technology that the candidate is interviewing for. But I made sure that the answers to these questions never had any effect on the outcome of the interview. And I’m sure that I’ve been put through such questions myself in the interviews that I’ve given over the years.

But in those interviews, I realised that the Dunning-Kruger effect is actually real. I was taken aback by the answers the interviewees would give to such questions. There were maybe just a handful of people who admitted that they didn’t know what I was talking about or that there’s nothing of the sort that I was describing.

I have nothing against the fact “fake it till you make it.” It is in fact necessary sometimes. But you need to be candid about it to yourself before the fake you consumes the real you. It can become destructive. And I’m not against new engineers who rate themselves high on their CVs, because they are just inexperienced and don’t know what they don’t know.

But once you gain the experience and the knowledge, make sure you are cognizant of the fact that there’s a lot more to learn. Nobody, in my opinion, can be an expert in everything. You might be an expert in your field, but you can’t deny the fact that you don’t know it all.

So, the Dunning-Kruger effect is very much real in the tech industry. And if this news to you, maybe it’s time to evaluate yourself, just to make sure you’re not oblivious to the fact that you don’t know it all.

And if you like what you see here, or on my Medium blog and personal blog, and would like to see more of such helpful technical posts in the future, consider supporting me on Patreon and Github.