tl;dr Humility is an important quality in technical interviewers. Our data shows that interviewers who are strongly confident in their own abilities give less consistent interview scores. Interviewers who are aware of their own weaknesses (and of how noisy interviews can be) in contrast, give more consistent scores. We've developed an exercise to help train interviewers in this area.

Programming interviews are noisy. Two interviewers judging the same candidate will often reach decidedly different conclusions about the candidate's skill, even in the same specific area. This noise is a significant obstacle to interview accuracy. Reducing this noise is one of the primary goals when training technical interviewers.

Training Interviewers

We train a lot of interviewers at Triplebyte. We employ a team of 40 experienced engineers to conduct interviews with candidates as they go through our platform. When we train new members of this team, we focus on several things. We make sure that interviewers are strong and up-to-date in the areas they will be measuring (it's surprisingly hard, sometimes, to distinguish a candidate who gives an unusual answer because they are an expert in an area from someone who gives an unusual answer because they don't know what they are talking about). We make sure that interviewers have clear guidelines for what skills they are assessing (this is the best defense against pattern matching bias in interviewers). However, I now think it's equally important to train interviewers in humility ^[1].

Lack of recognition of your own weaknesses is a major source of interview noise. This is true because overconfident interviewers judge candidates too harshly. The field of software engineering is broad enough that no single engineer can master it all. However, we all convince ourselves that the areas that we have mastered are the most important. And we don't fully respect another engineer if they are weak in an area where we are strong (even if they are very strong in other areas). In interviews, this manifests as a bias against candidates whose technical strengths are dissimilar to those of their interviewers. We measure this at Triplebyte by having multiple interviewers observe and grade the same interview. The effect persists even when interviewers grade areas unrelated to their own strength, and even when they use structured grading rubrics. Interviewers just give lower scores to candidates who are not like them. This is noise. It makes interviews less accurate, and we need to reduce it.

The solution, we've found, is to train interviewers in humility. Interviewers who are aware of their own weaknesses (and aware of how noisy interviews can be) are less influenced by areas other than the ones they are supposed to be evaluating, and give more consistent scores.

An Exercise to Build Humility

So, how can you train interviewers to be humble? How can you make yourself more humble? The answer, I think, is to experience what a candidate goes through. Interviewing for a job is humbling. You get grilled. You have to remember things you've not thought about in years. Smart people point out embarrassing flaws in your logic and code. You never know quite as much as you thought you did. And almost everyone fails a good percentage of their interviews.

We've developed an exercise that we use to let our interviewers experience being a candidate. At first we tried simply asking interviewers to interview each other. This did not work, however, because they were not able to give honest feedback. If you interviewed your co-worker and ended up thinking that they were kind of bad, would you tell them this honestly? Most people in this situation wouldn't. We needed to get around this somehow.

The solution came unexpectedly. The exercise actually started while I was trying to hire for our interview team. Part of the evaluation process that I used was asking candidates to interview me (it got really meta), and, in order to make these interviews more interesting, I gave a mix of (my attempt at) good and bad answers. I immediately noticed how this got them to give honest (and humbling) feedback--even on my “good” answers. From this experience, we developed the following exercise to train all our interviewers in humility:

• Pair up with a co-worker, and have them ask you some of their favorite interview questions.
• Tell them in advance that you are going to intentionally answer some of the questions poorly (role-playing answers that a weak candidate might give).
• Then, as the interview progresses, do exactly this. About half the time give your best answer. The other half of the time give an intentionally poor answer.
• After the interview is over, ask your co-worker to critique your answers.

What this does is free your co-worker to be 100% honest. They don't know which parts of the interview were really you trying to perform well. Moreover, they are on the hook to notice the bad answers you gave. If you gave an intentionally poor answer and they don't “catch” it, they look a little bad. So, they will give an honest, detailed account of their perceptions.

Be careful with this exercise! I've done this a bunch, and it's deeply humbling. It almost always results in someone you respect pointing out things you're bad at. And it has some potential to create conflict. I think it should probably only be done inside teams with a good degree of internal trust (the danger is convincing team members that other team members are not very good). But the result is powerful. It highlights clearly both the extent to which strong engineers are weak in certain areas, and the extent to which interviewers jump to conclusions about what a candidate means. I think everyone who conducts interviewers should put themselves through this exercise.

Conclusion

It's important for technical interviewers to be humble. This creates a better experience for the candidate, and it also makes interviews more accurate. The best interviewers are aware of their own limitations, and have a healthy appreciation of how capricious the process can be. To get better at these things, interviewers need to spend more time as candidates, being interviewed themselves. It's hard to create this experience among co-workers, but we've come up with a (dangerous!) exercise that does a pretty good job.

I'd love it if people tried this exercise more broadly. I think it might be something that should become standard for interview teams at most companies. If you give it a try, email me at ammon@triplebyte.com and let me know how it went.

^[1] I've interviewed over 1000 people since starting Triplebyte. Some of the them probably don't feel that I was humble when I spoke with them. All I can say to this is... I'm sorry if I did a bad job interviewing you. Everything I write about here I apply to myself.↩