Conducting Great Interviews with an Analytics Mindset


By Tyler McCabe, Director of Operations, Analytics Guild

There’s perhaps no greater opportunity to fall prey to confirmation bias than in the process of conducting job interviews. That’s a disconcerting fact when you consider that hiring the right person for a job is among the most business-critical decisions any manager will face.

Recently, I found myself in the position of conducting interviews to fill a critical position at a local nonprofit. I began preparing by listing skills and attributes I thought a great candidate would have, everything from the more concrete “knows bookkeeping basics” to the harder-to-pin “responds well to surprises.” Like anyone would, I ended up with a list so large it seemed impossible to act on. How could I assess whether or not a person contained so many qualities—both hard and soft skills—inside just a few hour-long conversations? 

Adding to this conundrum, the more interview questions I wrote based on the qualities I wanted in this candidate, the more I could see myself falling into the familiar trap of confirmation bias. From Decisive by Chip and Dan Heath: “When we want something to be true, we gather information that supports our desire.” It’s incredibly easy to go into an interview having prepared an hour’s worth of questions that will lead you to all the answers you want to find. Here are some examples:

  • Do you think you will thrive in a position with very little oversight?
  • What kinds of experiences have you had managing others?
  • If you were hired, how would you begin your own self-training for the portion of the job you know the least about?

Each of these questions contains their own answer, which is a bit like throwing up the kiddy bumpers on a bowling lane. The interviewee may bounce around a lot, but there’s virtually no way not to hit a few pins. The insidious thing is that as an interviewer, I want the interviewee to succeed; there’s an emotional boost I get from seeing someone perform well. And at the end of the day, I want to perform well by filling this position well. You can see how easy it would be to traverse the whole hiring process this way, hiring the wrong person and feeling very satisfied at the time of hire that all the right care had been taken.

So I began to research further, and the truth is that interviews are statistically unreliable predictors of success. From Decisive:

Research has found that interviews are less predictive of job performance than work samples, job-knowledge tests, and peer ratings of past job performance. Even a simple intelligence test is substantially more predictive than an interview…. The interviews seem to correlate with nothing other than, well, the ability to interview.

The Heaths go on to say that we tend to rely on the interview as a method of assessment because “we all think we’re good at interviewing.” Our tendency to fall into confirmation bias creates a loop of positive reinforcement: we guide people to display the traits we want to see, and we reward them for displaying the traits as much as we reward ourselves for “rooting them out.”  

Yet surely there is something to salvage here. Most people (myself included) would find it very difficult to hire someone without conducting an interview at all. To overcome my own biases, I began with a mental shift: what if an interview were more like an exam with a quantifiable score, an opportunity to collect data? This shift reframes the entire challenge of interview preparation.

For a moment, I put aside my questions and tried to create an environment of testing instead of predicting. The nonprofit didn’t have a budget to do a trial run with an employee, so that was out of the question. But I could give an actual exam assessing their ability to read spreadsheets and perform financial analysis. I could give a writing test to assess their grammar, diction, and control of style. I returned to my list of skills and attributes and began to see how I could give interviewees all kinds of “trial runs” in the skills I wanted to assess without simply taking their word for it in a conversation I had rigged. 

Though the data I was searching for was largely qualitative, I could still apply an analytics mindset when searching it out and compiling it. I created those two tests mentioned above with scoring systems as well as a minimum score required to earn a face-to-face interview. Incorporating numerical data into a hiring process not only makes your assessment stronger, but also frees you up to spend your precious interview time focused on the harder-to-pin traits you’re looking for. Assessing everything you can with a score system, you spend your interview getting to know the candidate’s quality of mind.

After I had created tests, I finally returned to my interview questions. The challenge was now to collect qualitative data, so I elevated a few qualities from my initial brainstorm that were more important than all others (introspection, flexibility responding to surprises, calm leadership, and mental organization) and put together questions that would give interviewees opportunities to display or tell stories about these qualities or not, creating ample space for self-assessment and individual determination of where the conversation would wander. For example:

  • Tell me about the most difficult work relationship you’ve had before, and the best.
  • Given that there are so many components of this job, which parts are you most excited about? Which do you want more training on?
  • Tell me about an experience where you had to take someone else’s direction, even though you disagreed with it.
  • Describe an especially fulfilling day at work for you.
  • Tell me about a role model you have for your life—who are they and what about them are you trying to emulate?
  • What do you think are the challenges this organization will be facing in the next decade?

By leading interviewees to tell open-ended stories about their life and professional experiences, I could assess their level of introspection, flexibility, and calm, and I could assess their ability to organize large amounts of information into clean, logical narratives.

A balanced analytical approach is about knowing how far to go with quantification and how far to trust yourself with qualification, being thoughtful on both fronts—especially when hybrid approaches like the one above work for your context. Very large companies like Google or Amazon tend to leverage more fully quantitative tools in HR, which contain some exciting possibilities. But regardless of our particular resources, we should always be thoughtful about integrating softer statistical tools like estimation and probability, and using intuition after we’ve tackled our natural biases.

Next time you’re in the middle of the hiring process, try creating skill tests and critically overhauling your interview prep to allow interviewees to self-reflect, spreading out the types of data you feature in your decision-making.

And, ultimately, you can always apply an analytics mindset to help you re-frame your tasks at hand:

  1. Identify what you can possibly collect
  2. Prioritize according to what is most important
  3. Sample the information
  4. Compile that information
  5. Assess it according to what is needed

When we marry the right quantitative and qualitative data, we create powerful methods for creating the right company cultures and generating the business we desire.