Structured Interviewing

A few weeks ago, I wrote a post on applied interviewing. Mark Tsimelzon, founder and President of Coral8, replied that Coral8 is a strong advocate of structured interviewing and has had tremendous success hiring the best and brightest by a well-defined, consistently applied hiring methodology. Mark kindly offered to serve as this blog's first guest writer and the post below is a wonderful "how to" structured interviewing guide. Thanks to Mark!

A Practical Guide to Structured Interviewing ============================================

All entrepreneurs agree that building a strong team is extremely important for their startup's success, but many first-time founders are not sure how to achieve it, and are baffled by the interviewing process. The number of questions and choices is indeed confusing: where to get the resumes, whom to invite to an interview, whether to conduct interview over the phone or in person, what to ask, how to evaluate the answers, etc.

Luckily, the structured interviewing approach Will wrote about recently can help. Below are some details of how we apply structured interviewing at Coral8 ( a Silicon Valley startup building a platform for real-time analysis of large volumes of data). While what follows applies mainly to engineers (software, QA, support, professional services), the same would apply to many other job functions and positions.

Like any complex process, the interviewing process is best structured and analyzed as a sequence of phases. At Coral8, we have four phases: email interview, phone interview, the first in-person interview (with 1-2 person), the second in-person interview (3-4 others). Whether you have the same stages or not is not important. What's important is having a clear understanding of a) why you are having each phase b) what you are trying to accomplish, and c) how you are going to evaluate the results. It helps if all the interviewers share this understanding, and keep the process as consistent across candidates as possible.

Let's consider the phases separately.

Email interview
---------------

Most startups complain that just finding a good candidates is the hardest part of the process. Post a job description at any jobs web site, and you'll receive hundreds of resumes, most of them from people who are not even remotely qualified. Hire a recruiter, and he'll be sending dozens of resumes your way, again often from poorly qualified candidates. Just opening and reading those resumes is often overwhelming.

The problem may seem impossible to solve, yet a simple solution exists, and here at
Coral8 we are still puzzled as to why so few companies use it. Here is what you do: You never, ever, publish a position description without an accompanying problem that someone MUST solve before you even open their resume. The problem
should: a) have a solution that can be easily reviewed b) take the right candidate about 10 minutes to complete c) test a skill that is core to the job
d) be somewhat interesting e) allow a super-star candidate to show off their
knowledge and skill. Since most of our positions require programming in some
language, our problems often require writing a simple program or function. The language and the complexity is adjusted for each position.

For example, when we interview C++ engineers, we want to make sure they understand polymorphism and virtual functions. It's disturbing how many people who call themselves C++ programmers do not. So here is a problem we sometimes
use: Illustrate the use of the keyword "virtual" by writing a short C++ program which contains this keyword and prints "Hello, world". If the keyword "virtual"
is removed, the modified program should print "Good bye, world".

Some may say that the problem is too simple, and it certainly is. But having a problem like this in your ad does wonders. First, it greatly raises the signal-to-noise ratio. Few Visual Basic programmers will bother to send their resumes if you require a solution to a C++ problem. If they do, or when the solution is wrong, you can quickly ignore the submission. The resumes you'll get are usually from people who are really motivated, and not just sending resumes to all the positions. If somebody gives you a correct solution, you immediately know that you have a strong candidate.

Some folks may think that having a problem discourages some good candidates from applying for a job. Who knows, maybe this is true for some candidates. In our experience, however, we find just the opposite. Some of the very best employees we've hired told us that they applied for a position with Coral8, back then a stealth-mode startup, precisely because they were intrigued and challenged by our posted problems.

What if you do not publish your job openings, but instead use recruiters? Easy!
Give the problem to the recruiter, and tell him that you will not accept any resumes without the solution to this problem. Some recruiters will tell you they don't want to do the extra work. Somehow these are often the recruiters who want to charge you 25-30% of the candidate's annual salary, and you just pass on them. A startup can and should negotiate a much better deal anyway, and, more important, a good recruiter loves the fact that you have this problem!
Why? They know that you get a lot of resumes, from many sources. They want to give you the candidates who are the most qualified, and they have to qualify them somehow anyway. And what better way is there to qualify the candidates than to use your own problem?

So this is what we call an "email interview". In most cases, unless we have a high-quality referral, we refuse to even look at the resume unless we see a solution to our problem. Life is too short to look at hundreds of resumes a week.

Phone interview
---------------

Phone interviewing is a much more traditional and better understood process, so I won't spend as much time on it. What we found important is to have a list of questions for each position, and to try to follow this list every time. It makes the process much better structured, and over time you learn which questions are harder than you thought, and which ones are easier. So it helps you calibrate your expectations better.

Since we only schedule phone interviews with the people who pass the email interview, we know they have at least some basic understanding of one area. So the phone interview is used to get a slightly better understanding of the breadth and the depth of the candidate's skills. We spend about 30 minutes with a candidate, and use a part of that time telling the candidate about the company and the position. After 30 minutes, it is usually clear if you want to invite the candidate to a face-to-face interview.

1st face-to-face interview
---------------------------

All right, the candidate comes in to see you, what do you do? As Will mentioned, too many interviewers go with "Tell me about yourself?" and "Tell me about your past projects?" kind of questions. These questions are ok, but spending more than 10-15 minutes on them is counter-productive. The last question is often useless, unless you happened to know a lot about specific areas the candidate worked in. If you do not, it's hard to evaluate how challenging the tasks really were, and whether the decisions he made were correct. Instead, what you should evaluate in these situations is the candidate's presentation skills. Whether you are an expert in his area or not, any good candidate should be able to clearly explain to you what he did, what the overall project was, what the trade-offs were, etc.

But most of the time should be spent with the candidates answering a carefully constructed set of questions. This may be a religious issue, but we at Coral8 strongly believe that the questions should be clearly related to the job, and not some puzzles that test nothing but the ability to solve puzzles.

Now, if you are interviewing programmers, then please, please, please, administer some programming exercises. There is an amazing number of programmers on the market with fancy resumes, fancy titles, and fancy degrees from fancy schools, who nevertheless cannot program well. You do not want to hire people like that. Any resistance to programming during the interview (e.g, "I don't program well during interviews") should be an immediate red flag.

Of course, when you ask somebody to code during the interview, be reasonable.
Many people do not remember the language syntax or the names of library functions. That's entirely ok. We either let people use some reference guide or Google, or just tell them that they should not worry syntax. To us, what matters most is algorithms, so this is what we pay attention to. You may worry more about something else, but whatever it is, make sure you carefully construct your questions to test for that, rather than just asking whatever you feel like at the moment.

2nd face-to-face interview
--------------------------

Now, the candidate passed your interview, and you invited him to come again to "meet the team." Sounds innocent, but this is one of the more challenging parts of the process. You've got to recognize that people have widely different interviewing skills, and sometimes even great engineers make poor interviewers.
Work with them. Agree, as a team, on which questions to ask, and who will ask them. Sit in on some interviews yourself, to make sure you are satisfied with the interview dynamics. It's often best if people interview in pairs: it takes less candidate's time and lets the interviewers to learn from each other.

The 2nd face-to-face interview is also a good time to further investigate some
personality issues. For example, the team may agree that during the interview,
one team member will try to push the candidate a bit, disagree with him strongly on some issues, and see how the candidate handles it. You don't want to go too far, but it's a useful test. Nothing kills productivity in a startup like engineers who do not know how to disagree with each other professionally.

All right, everybody has talked to the candidate, now what? This is the place where it's especially important to have a well-defined process. It's best to have a formal scale, on which everybody will grade the candidate. The scale we have at Coral8 was popularized by Siskel and Ebert: thumbs up, thumbs down, and we also have many gradations in between. It's not important what you have, the scale 1-10 works just as well.

It also helps to have a threshold. Let's say your threshold is 7. It means that you do not want to hire anybody with the average score below 7. If you have only one serious candidate for the position, you just need to decide whether he scored above your threshold or not. If you have multiple candidates, you need to compare them to each other, using their combined scores.

How you combine the scores given by different team members is an art, not a science. You may start with simple averaging. Or you may want to say that anybody with one or two votes below the threshold is automatically disqualified.
You may also keep in mind that one person's 9 may well be another person's 6.
The ultimate decision is yours, but what's important is that you as a team talk about the candidate, and discuss what people like and dislike about the person.
During this discussion, you'll learn a lot about the candidate, and you'll learn even more about your team!

Conclusion
----------

There are many parts of the hiring process that we have not covered yet: formal reference check, informal reference check, offer negotiation, etc. The important point, however, may have nothing to do with hiring per se. To many people, the words "startup" and "process" are mutually contradictory. Processes are for Fortune 500 companies, right? Not so fast. Like it or not, there are many processes going on in any startup. You may recognize, structure, and optimize them. Or you may hope that they just work by themselves. The interviewing process is a good example of a process that will produce some results either way: after all, no startups die because they cannot hire any people at all. But there are certainly ways to make this process much more effective, efficient, and enjoyable for all participants.