class: center, middle # Individual Interviewer --- class: center, middle # Disclaimer Beyond my general disclaimer, these are personal notes that may be out of date. Read generously. --- class: middle, center # Selection --- class: center, middle We are selecting people into the organization who can effectively achieve our goals. Whether that means executing within our current processes or disrupting our current structure, we need to focus on selecting people who have a *problem-solving process* that will achieve results in our organization. --- How well you select candidates is typically evaluated through a diagnostic tool called a "confusion matrix": -- .center[] -- There is a tradeoff between high precision and high recall, and every company has to decide which to prioritize. --- class: left, middle ## Selection Factors A [major study and meta-analysis](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172.1733&rep=rep1&type=pdf) has some useful ideas for what works best for interviews: - General Mental Ability: while IQ tests are not legal to administer due to bias, a good lesson here is to focus on *how* candidates solve problems instead of whether they know the "password" to a brainteaser. - Structured Interviewing: Use consistent questions and pre-determined clear criteria with detailed feedback to reduce the propensity for interviewer bias, and create the opportunity to assess question performance over time. --- class: middle # Structured Interviewing --- class: left, middle ## Phases The rest of this talk will orient around the three phases of an interview: - Before - During - After --- class: left, middle ## Before: Prerequisites A hiring manager should pre-commit and articulate the following: - What does our ideal candidate look like? What skills should they have? What problems should they be able to solve? What is the comparative advantage of our organization and given tradeoffs between different skills, which ones are higher priorities? Everyone wants to hire the unicorn who is good at everything, but who is the non-unicorn who would excel in the role we have open? - What is the role of the interviewer? Given a hiring loop with multiple interviewers, who is evaluating which dimensions and attempting to acquire what signal? --- class: left, middle ## Before: Scripted Question If there's one thing you should do, it's to use a scripted question and rubric - while this won't *eliminate* bias, it does help to reduce and compartmentalize it - after all, your evaluations shouldn't depend on whether you're grumpy right before lunch. Walk into your interview with the following in hand: 1. A scripted statement of the problem. 2. A reference solution of the problem. 3. Common mistakes, errors, and paths people take through the problem. 4. A scoring rubric for the problem. --- class: left, middle ## Before: Practice The second most important thing is to practice. This will help you smooth out things like confusing problem statements that may impact candidate performance without providing any signal. - Run your question on a few people before you go live. - Have someone run it on you. - Have someone shadow you running the question. Interviews tend to be pretty noisy, which can make practice awkward, so make sure to establish a rapport with your partner. --- class: left, middle ### 2-4-6 Game -- It turns out that you want to ensure your interviewing properly classifies both positive *and* negative samples. You can achieve this and mitigate some of the awkwardness of practicing by having the candidate flip a coin and then use an artificial limitation to hinder performance. --- class: left, middle ## During You now have (most likely) 60 minutes to arrive at a concrete decision, which means you want to have the candidate spend as much time as possible on the *interesting* parts of the interview. Imagine the candidates walking in and facing a completely blank map - you have an opportunity to fill in *some* of it, so that the candidate will spend more time in the sections that you are actually trying to assess for. --- class: left, middle ## During: Structure Preamble I like to start by eliminating as much meta-ambiguity as I possibly can, by explicitly describing the structure of the entire interview: it eliminates guessing about remaining time, shifting requirements, and surprise anxiety. 1. Describe the overall structure of your interview (major phases, time allocated), and the high level of what you'll be assessing for. 2. Give a very short background description of yourself, to help the candidate understand their audience (and to also help them think of questions to ask later). "I'll spend two minutes telling you about myself, 45 minutes on a technical evaluation of your programming skills with a laptop during which I'll be taking notes, and I'll leave 10 minutes at the end for you to ask me any questions you might have." --- class: left, middle ## During: Evaluation Preamble Before starting a particular evaluative question, I also like to eliminate as much ambiguity as possible by describing the evaluation structure: - Are there multiple parts, and how long is each part expected to take? - What matters and what doesn't? (E.g. - "Attempt to write working code, but you won't fail if you miss a semicolon on the whiteboard, and don't worry about degenerate inputs") --- class: left, middle ## During: Evaluation During the evaluation, you should be focused on *bifurcating* the candidate pool as well as possible with the time you have remaining. Is the candidate currently trending towards "yes" or "no"? In the time that remains, what do you need to hear in order to reach a concrete decision one way or another? --- class: left, top ## During: Notes and Details During the evaluation, take extensive notes, and force yourself to capture *details*. If you want to say "the candidate was a good problem solver", force yourself to dig deeper than that - what did they do that made you come to that conclusion? And have it written down so you will have a clearer record of what happened. -- This also applies to answers the candidates provide. "I built product X" is not a useful answer. Dig in. Did you come up with the idea? Did you design it? Did you receive a spec and implement it? Did you do customer research? How did you choose which technologies to use? Was this built with a team or solo? etc. You want to understand the *process* that led to outcomes. Ultimately, new hires do not bring their previous outcomes to your organization, they bring their problem solving processes. --- class: left, middle ## After: Feedback When writing your feedback, cover a few major areas: 1. Use your rubric to assist you with a structured decision. 2. Provide top-level relevant insights ("strong problem solver") backed up with useful concrete details. 3. Include general notes after light curation to remove excess noise. --- class: left, middle ## After: Logging The one final step I recommend after interviews is to *log* your decisions. If you keep track of: - What you recommended - What you predict about the candidate Then for some candidates, you'll be able to revisit the issue 6 months later and get some concrete data and how your interview performs.