Your Hiring Process Is Probably Biased

This post is cross-posted from my blog, just like my last dev.to post in 2017!

Hiring is hard and important

Hiring is hard. This is true in any industry, but in the tech industry we're in the habit of handing out life-changing salaries to people who have bootstrapped their own careers. With our hiring choices, we have the power to reshape the financial reality for people from marginalised backgrounds or impoverished communities.

It's not uncommon for skilled developers to be neurodivergent, or to struggle with mental illness. For many people, mental health problems have been exacerbated by years of abusive treatment within the industry, or by burnout. As a hiring manager, you have a duty to care for the people you interact with. At a minimum, your hiring process should not actively hinder people's well-being and mental health. When successful, your hiring process is a critical step to building a high-performing engineering team and a great product. When done badly, you'll burn your reputation and you'll struggle to recruit, or you'll find yourself only recruiting people who resemble your current (likely homogeneous) team.

You are not entitled to good candidates

This post is for engineering managers, technical founders, engineers who aspire to become managers one day—for anyone with the responsibility of hiring software developers at a small to mid-sized company. It contains strong opinions, anecdotal examples of actual hiring processes I have endured, and quite a lot of advice for ways you might be able to improve your hiring.

Of course, my idea of a better hiring process might not be the same as yours. If you don't agree that diverse teams build better products, you've found yourself in the wrong place. Please move along. I'm not here to sell you on the importance of diversity, and you have bigger problems to sort out than your broken hiring process. You can try to build a diverse team within an organisation that doesn't value diversity, but it won't be sustainable and you'll risk recruiting wonderful engineers and then giving them a bad experience. Please don't do that. I've made that mistake, and I can assure you, it feels terrible. If anyone from HR or your executive team says the words, "It's a pipeline problem," to you, just run as quickly as you can into the arms of a new and better employer.

The goal is simple

Healthy hiring has a simple goal: to enable candidates to show you their best work, so that you can try to assess whether they'll be able to perform the job responsibilities, and so that they can assess whether they'd like to work with you.

Brittle == biased

Hiring can be conducted on a rolling basis—you have many openings to fill and a good runway to pay them—or what I like to think of as "high-stakes hiring"—you have one or two immediate openings, and must select the best candidate(s) from a pool of people who have applied during a fixed time period. The process is basically the same for both, but your feedback loop is longer for a high-stakes hiring event, since you won't be able to enact improvements to your process until your next hiring round. With rolling hiring, your process can evolve and improve over time as you learn from the feedback candidates provide to you.

Building a good hiring process follows the same cycle as building a good product: Build, measure, learn. But how do you make sure you're building the right thing? Who is your hiring process optimised for? One size doesn't fit all—not for socks, not for t-shirts, and especially not for assessing the diverse range of skills and capabilities that make for a good software engineer.

Most hiring processes for software engineers are divided up into a few rounds, or phases. There's some sort of intake procedure, a sifting or screening process, often performed by an algorithm or a low-level HR employee or support staff. There's likely to be one or more technical assessment rounds. There will almost certainly be some sort of interview delving into a candidate's past experience, and often an additional interview to evaluate "culture fit". These stages can drag on for weeks, even months, or can be pushed through in less than a week. Sometimes the timing is flexible, while the steps followed are not. Bias creeps in at each stage of the process, and it's easy to lose sight of the goal.

If your hiring process isn't flexible, you're defaulting to optimising for a certain type of candidate. Is that really what's best for your business? Is this a choice you've made, or a trap you've fallen into? Consider the wide range of people you might be excluding, and the valuable perspectives they could be bringing to your product development:

• People with mental illness (anxiety, depression)
• Neurodivergent people (autism spectrum, ADHD, etc.)
• People from nontraditional backgrounds, who may have job-relevant skills but, if they don't job-hop or interview a lot, may not yet have gained the specific interview skills you're expecting them to exercise
• People of lower socioeconomic status, for whom performing unpaid timely labour (completing your hiring process) may not be feasible
• People with carer responsibilities, more than full-time work, or other time and availability constraints
• People with disabilities, and if you haven't offered suitable accommodation, this may be illegal in your country

Sifting and screening

You probably don't have time to interview every candidate who applies for a job at your organisation. And you might not want to, even if you have time—copy & paste cover letters are common, and some folks clearly take a firehose approach to job applications, especially if your application software offers a 1-click "import my data from LinkedIn to apply" button. This initial screening, to hand-pick the candidates who might be a good fit based on their CV and cover letter, is frequently performed by people far removed from the rest of the hiring process. I can't tell you how many stories I've heard from hiring managers who received a stack of 40 CVs for an open role, filtered by their HR department, only to discover that everyone who wasn't a white man had been filtered out.

It's also a measurable, proven source of bias and discrimination in hiring. It's totally unnecessary, unhelpful, and you should skip it and do something better. And I say this as someone who almost always, 99 times out of 100, will get through this initial screening due to my penchant for writing a persuasive cover letter. This process is clearly biased in my favour and I'm still telling you straight-up that it is bad and toxic and you shouldn't do it.

This is not an advertisement for Applied, but I had the opportunity recently to use the Applied platform as a candidate and see for myself what their sifting process is like. It's fantastic. Candidates are presented with a slate of questions designed to reveal competencies relevant to the role. Once they've submitted their answers to these questions, candidate responses are anonymised and shuffled into random order, presented alongside responses from other candidates for scoring and review. The aggregate of anonymous scores is used as a cut-off to determine which candidates have scored well enough to proceed. During this initial sift, no one responsible for hiring is seeing candidates' names, educational history, or employment history. Candidates are evaluated solely on their responses to job-relevant questions, which (unsurprisingly) turns out is a far more accurate predictor for success.

In my experience as a candidate with Applied, I was asked questions about web performance, code reviewing broken Javascript, using elasticsearch for analytics and reporting, working in teams, and prioritising work. The questions were thoughtfully designed to reveal a lot about my thought proccess and approach in each of these areas, and I'm confident their team learned a lot more about my skills and knowledge than they would have learned from looking at my CV.

Writing skills are crucial to working well in teams remotely. Writing a cover letter is a skill unto itself, and not one which is generally useful during day-to-day work as an engineer. By asking candidates to provide thoughtful written responses to questions relevant to their work, you'll be able to evaluate competence at written communication; by doing this anonymously and in randomised order, you'll ensure that your evaluation is based on salient information instead of the irrelevant background noise to which you've been conditioned to pay attention.

Interviewing remotely

Once you've identified a few promising candidates from your pool, you'll contact them to schedule an interview. It's 2021 and the world is a hellscape, so unless you're lucky enough to be in New Zealand, South Korea, Taiwan, Australia, or a few other countries with competent governance, you'll be interviewing remotely over Zoom or Skype or some WebRTC-based video chat tool.

Interviewing remotely sucks. I love remote working, but if given the choice, I'd always choose an in-person interview. For many people, the technology for remote meetings just doesn’t yet compare with the fluidity of IRL conversation. In particular, unreliable upstream internet service can have huge impacts on how a candidate will perform in an interview.

So it sucks, but we have to do it, so we may as well make the best of it. Luckily, with some slight adjustments to your interviewing etiquette you can set your candidates at ease and give everyone a fair chance at success.

1. Start by making sure everyone can hear and see each other. This can feel a bit awkward, but normalise starting your call with a verbal confirmation that all parties can see and hear one another.
2. Outline the agenda in advance. You should include a detailed agenda, with the questions you plan to ask, in the calendar invitation for the interview. At the start of the call, review the agenda and outline how the interview will proceed. For example, "The purpose of this call is to get to know you a bit and to allow you to ask questions of us. We'll start by introducing ourselves, then we'll talk through the questions we shared in advance. We expect that to take around 20 minutes, which should leave 10 minutes for us to answer any questions you have."
3. Score responses to fixed questions. Ask the same questions of each candidate, as much as possible. Define your scoring system and your questions in advance. If you're hiring on a rolling basis, these questions will change over time. That's fine. But aim to maintain a consistent approach and ensure that it's well-documented.
4. Watch closely for visible signs of packet loss. You're a technical person. You can see and identify signs of poor connection, high latency, and packet loss. You need to adjust your communication style accordingly, by pausing and repeating yourself as-needed to counteract technical challenges.

Bad connections are inevitable, often outside of our control, and make video meetings an incredibly ineffective and inefficient way to communicate important information. We still do them, because it's nice to meet someone face-to-face and see how you interact with one another. But the limitations are substantial. Use video calls to augment your knowledge, but do not rely on them as the only means of collecting certain information about a candidate.

One of the best interviews I had was with a company who build a WebRTC-based video chat system. I spoke with two of their engineers, who were on it when it came to noticing packet loss and adjusting the conversation accordingly. Even though the connection wasn't great, we were able to have a delightful, comfortable conversation by working around the technical difficulties.

One of the worst interviews I had started with, "I'm XXXXX and this is XXXXX. Let's start with the back-end task. Can you share your screen?" For this interview, I requested an agenda and more information about the nature of the interview in advance, but was told my questions wouldn't be answered because it "wouldn't be fair." This was not a high-stakes hiring situation; the role had been open for months and many recruiters were working to fill it. If you're relying on your interviews to weed out candidates, or to pit them against one another, you've lost sight of the goal. To enable candidates to show you their best work, help them to come prepared, just as they would for a day at work. And for goodness sake, at least start your call with, "Hello, how are you?" and "Can you see and hear me okay?"

Assessing technical competence

Tech tests, coding challenges, technical assessment. Whatever you call it, this is where a lot of places fuck up.

Many companies use a take-home coding challenge as a second-round screening, to eliminate candidates from the funnel who don't show a particular set of skills in their solution. Congratulations, you're optimising for people with free time and comfortable socioeconomic status, and you're placing the burden of evaluation solely on the engineers on your team who are willing to score a coding challenge.

Some hiring managers look at side projects and open-source contributions as a way of assessing technical competence. Congratulations, you're optimising for people with free time, comfortable socioeconomic status, and who have not been victims of harrassment, abuse, or stalking.

Maybe you prefer to do a technical interview or some pair programming? Congratulations, you're optimising for people who have taken the time to hone their technical interviewing skills. These skills are not actually very related to the job you'll be asking them do. You're effectively eliminating candidates with anxiety, from nontraditional backgrounds, or with some forms of disability. Pair programming is particularly insidious, as it shares a name with a common practice you might engage in at work, but in an interview format it bears no resemblance to the other kind of pair programming you might be familiar with.

There are a lot of different ways to be a good programmer. A person usually only has to be good at one or two of them. Odds are good that you’re testing for skills you don't need. This may also be preventing you from assessing far more valuable skills that you’re not seeing. To truly assess someone's technical competence, don't lose sight of the goal. How can you enable a candidate to show you their best technical work? I don't know! And you don't either. The only person who will know how they work best is that specific candidate. This is where a brittle process will fall flat, losing you valuable candidates and poisoning your name if you torment people unnecessarily. Define the skills you're looking for in advance, and offer your candidates a slate of choices for demonstrating those skills.

Options for technical assessments

• Take-home tests

  • 👍 Working at home and independently is similar to an actual remote working environment. A well-designed take-home test can allow anxious candidates a good opportunity to show their best work.
  • 💰 Can be inaccessible for candidates with care duties and full-time jobs, so your take-home test shouldn't be a firm requirement. If you must have a candidate complete your take-home test, you need to pay them for their time.
  • 🤖 The challenge should be as closely related to your business domain as possible. I have seen challenges to build a vending machine (for an AI-type chat application), to program a robot to move in a certain way (for a fintech company), and to make a 3-card Monty game (for a company doing film analytics). Just.... why. Don't do this. Don't do any of this.
  • 🛠 Unless you're hiring a developer for a greenfield project, you're probably looking for someone who can work well in your existing codebase. So make sure your take-home test includes starter files! Bonus points if you use your actual codebase, or some sanitised portion of it.
  • 🔍 Requirements must be clearly defined. No, more clearly than that.
  • ⏰ Be mindful of your timing! Particularly in high-stakes hiring, if you're giving all of your candidates the same take-home test to do over the same time period, you're a real asshole if you make that time period "the weekend of Rosh Hashanah". This did happen to me. I don't think they have any Jews working for them. Probably not too coincidentally, there are also no women on the team this was hiring for.
  • 💞 The feedback you provide to candidates should be of comparable quality to what's shared in an internal code review. This means it should be kind, empathetic, and constructive. If you find yourself providing the same piece of feedback to everyone, that's a great indicator that your challenge requirements aren't clear enough.

• Technical interviews

  • ⚖️ More respectful of candidates' time, but require skills that are interview-specific, not very related to job duties
  • 🧩 Can be a good way to demonstrate technical competence in lieu of a take-home test, or to follow up on knowledge gaps potentially evidenced by take-home tests
  • 🧠 To get the best performance from candidates, allow them to prepare in advance. A good developer doesn’t go into a meeting unprepared. Why are you screening for people who excel at doing that? It’s far more interesting to see how a candidate can approach a problem if they’ve been given some time in advance to think it over and understand the context, particularly for design and architecture problems.
  • 💣 I've often heard the conventional wisdom that developers should practice their interview skills regularly by going on interviews for jobs we have no intention of taking, or applying for jobs even when we're happily employed elsewhere. While it's true that often these interviews, particularly systems design and other technical interviews, generally require practice to do well at, this advice is, frankly, absurd. Going to practice interviews is a waste of everyone's time, and privileges people who can afford to take time out of their jobs to do this regularly. And bad interview experiences can be psychologically damaging! Practice for the sake of practice doesn't come without risks. Moreover, the reason these types of interviews require practice is because they don't exercise skills actually used regularly on the job.

• Code review

  • 🔑 A trend I've been seeing more of lately is to include a code review phase of technical assessment, often as part of a take-home test or interview.
  • 🧶 I like this, but it relies on clear requirements to work well. If your take-home code review assessment says something like, "You're reviewing this code for a colleague," that raises more questions than it answers. Determining the appropriate tone and content for a code review is highly context-dependent, so you need to make sure candidates have that context.

• Trial work day

  • ⏳ One place I interviewed used a "practice onboarding day” in lieu of other technical assessment. It’s a good option if you’re in a high-risk hiring situation, where you need to give a fixed number of candidates a deep assessment over a short time period.
  • 📅 Sometimes this can be extended; you can offer up to a full week's worth of trial work if you're very nervous or unsure.
  • ⚡️ You must offer options; this option won't be accessible to people with full-time jobs if you're expecting them to do this during regular working hours.
  • 💸 Of course, you must pay them.

• Side projects or open-source contributions

  • 🐙 This is fine if the candidate offers up some code to look at, but bear in mind that it might not tell you what you need to know.
  • 🔒 It's reasonable to use this as a positive measure, but it's not acceptable to count an absence of side projects or OSS contributions against a candidate.

Feedback and fallout

A bad experience with your hiring process can ruin a person's week, or even month. You don't want that kind of ill-will towards your company floating around out there if you can avoid it. (And you can! It's not that hard to not be awful!) What feels like a good experience to one person might be anxiety-inducing or worse for others. People with mental illness, disabilities, or other constraints have developed strategies to be successful at work, but these strategies don't necessarily map to the different set of skills required to interview and apply for jobs. By making your hiring process as close to real-life work as possible, you'll gather more relevant information and stop excluding those candidates who may provide the most value to your organisation.

• Over-communicate, and communicate in multiple formats, to align expectations and overcome mismatches in communication style or technical issues.
• Provide valuable, constructive feedback to all of your candidates, and listen to their feedback for you.
• Offer options and accomodations during each of your hiring stages to enable candidates to show you their best.
• Share questions and problems in advance so that candidates can come prepared to dazzle you.

With a bit of effort, you can craft a more inclusive hiring process. Effort expended up-front to make your process more pleasant for everyone will pay dividends in diverse teams of people who feel valued and supported, and who are empowered to do their best work for you. Good luck!