What are the pros and cons for the employer of code questions during an interview? [closed]

Question

The Joel Test question #11 is: "Do new candidates write code during their interview?". What are arguments for and against to ask new candidates to write code during the interview and to make a decision upon it?

Answer 1

I don't see the cons. An interview has many parts, and a candidate should be endorsed 'up the chain' a few times.

I interview ~10 people weekly. I really, really, REALLY appreciate the fact that HR has done all of the background work and presented me with plenty of notes. By the time they get to me, it's time for me to score their tests.

The tests depend entirely on the position. Generally, I try to probe:

• General programming skill. Can you use operators effectively? Can you conceive a numerical system that has a radix which is not 10?

• Do you know how to do what we're hiring you to do?

• Evaluation of your contributions to any open source projects that you listed

I try to keep it short, and fun. When I go into the office, I grab the answers, look them over and then conduct a secondary interview. To get hired, you typically have to get through three interviews.

I also gauge how well you will blend in with the team that will be receiving you. That team counts on me to do that effectively.

It is one thing to answer questions in meta form, it is another to actually produce code. If I'm going to hire you, I really need to see you produce code.

Answer 2

With apologies to Scott Whitlock:

Cons:

• none

Pros:

• Saves a LOT of time and heartache down the road if you prevent hiring someone who can't program
• Requires you to have a technical person in the interview

Answer 3

If you were going to hire a juggler, you'd be insane not to have him juggle for you. Or a musician you'd have audition. Otherwise you get stuff like the awesome yo-yo "master" K-strass.

Walking through something on a whiteboard is the programmers equivalent of a quick demo juggle.

Answer 4

I think it's super useful, and I always do it, but since the benefits have been covered so well I'm going to discuss only the (apparent) negatives.

I think code tests are fairly unlikely to give you false positives: the odds are low that someone who actually can't write code will manage to fake it in an interview, at least if you have a scale of questions of increasing difficulty. (Perhaps the most likely scenario is they're cheating by asking a friend, if it's not a face-to-face interview.)

The problems are more on the false-negative side: will code tests lead you to reject the person who's actually the best candidate?

Stage fright

You may have someone who's actually a really good developer, but who is very nervous about this interview, and they get essentially stage fright. Performing under pressure is important to some extent, but dealing with stage fright is not such a key part of being a programmer (compared to other professions), and it would be unfortunate to reject someone who suffers badly from it. This can compound: if the person can't answer a question they know they ought to answer, they may get more up tight. Or, as in this question, they feel they can't talk and code at the same time.

mitigation: Start off with some getting-to-know-you questions about their background, goals, etc, before you jump into technical questions. Perhaps start with some easier technical questions so they get some momentum. Don't be a dick during the interview (quibbling about semicolons etc).

It's a noisy measure

Interesting code questions might have more than one correct answer. If one person writes a correct answer, and another writes a correct and more efficient answer, how much weight should you put on that?

To some extent this is like the problem with some "puzzle" questions: the person either has the insight or not and you get a nearly binary result. Intelligence probably affects the probability of having that insight, but sampling only a few times gives you a crude measure.

This bothers me about code questions (though I still use them.) The best mitigation I can think of is to as much as possible have a ramp of possible solutions: the person can at least write a crude brute force answer, or an answer for a part of the problem. Realizing that's better than nothing is a good sign. Then if they discover more, they can make it more efficient or more elegant code. As much as possible, avoid asking questions that get binary responses.

It's not really representative

Programming isn't a job of solving ten-minute algorithmic problems one after the other: there's lots more work about understanding and designing larger systems and concentrating for extended periods of time, to say nothing of the interpersonal skills. Code questions don't really test this.

But, code questions aren't the only questions you're going to ask: you can look at their background, their references, their open source work (if any), to find evidence of sustained effort, creativity, interpersonal skills.

Knowing how to solve small algorithmic problems and how to reduce them to code is a necessary but not sufficient condition: if you can't solve small problems and you can't write nontrivial code then all the big-picture thinking in the world is not going to make you a productive developer.

Anybody could solve that

No, apparently not. As the famous FizzBuzz post points out, problems you might think are trivial trap not only new graduates but people with years of industry experience. I don't know why. Either the code test is a poor measure (which is possible, though I think it's unlikely), or they weren't contributing very much to the projects on their resumé, or they were getting a lot done by copying-and-pasting non-algorithmic code (which is possible.)

It's worth acknowledging that you really can get a lot done without writing any algorithmic code. People make a lot of money on apps whose value is in the graphics or business logic not in what you might call "programming", and that's fine. But, if you actually need programmers, it's not a good fit.

It may not be well calibrated

If you come up with a question, the answer may very well seem easy to you. However, if you're asked any otherwise comparable question out of the blue, or a question that's not skewed towards your own particular interests and background, it may be much harder.

mitigation: Run the tests over some developers you already know, and see how they do. Perhaps someone already on your team, who you know is very smart, will have trouble with one of them and you can consider adjusting it. Perhaps they will think of a better or different answer.

It's too much like trivia

I think code questions certainly can get in to trivia, if you insist people know obscure APIs by heart, or get the syntax perfect, or remember the exact definition of a nontrivial algorithm. Those are all reasonable to rely on docs, web searches, or compiler errors to pick up, and have little correlation with real expertise. Not even knowing where the API's likely to be is perhaps a clue the person hasn't used it recently, but that's not necessarily a problem as long as they're not lying on their resumé.

So the answer to this is pretty simple: don't ask trivial questions and don't get hung up on trivial mistakes. Remind the candidate what the API is called or let them look it up; fix up syntax errors; don't test for people memorizing data structure definitions.

How do you compare?

If you have two candidates and both answer the questions well, how do you pick between them? You could choose the one who finished fastest, but perhaps there you start to pick hares over tortoises. You could do another round and ask much harder questions but I'm not sure about that either. Perhaps you just give them both an A+ and try to choose between them on other criteria (or try to find the money to hire both.)

Answer 5

One con I can think of is that it is hard to "code out loud" in front of other people. I find it hard even to type with someone looking over my shoulder, and I'm not alone. I notice when someone calls me over to their workstation to help with something, they suddenly start making typos, selecting the wrong code-completion, even making downright mistakes — none of which they would have made had I not been sitting right there. Hell, I've seen people start using the menu for cut & paste operations, just because they were under observation. This is not normal behavior, and the coders I'm talking about are excellent programmers and quite smart besides.

I recently had an interview in which the interviewer asked me how I would code a particular operation and he said, "Just show me the math." Well, I had to think about the problem first before getting to the math of it, so that had me hemming and hawing. What I put down on the white board at first was embarrassing, and I felt I was losing at that point. I eventually got the A-ha moment and found the answer (actually when it finally occurred to me what he was really asking), but the "mess" I made before I got there made me feel very uncomfortable. Nevertheless, I got the job, but if the interviewer had been less patient I might not have.

I think if you give interviewees a coding task, give them some time alone to work it out on a computer, perhaps even in an IDE they're familiar with. Let them write the code for you and then talk about it. Ask them why they did things a certain way, and whether another way might not be better. You will find out more from that kind of process than by asking them to (figuratively speaking) pee into a cup right in front of you.

Answer 6

Cons: None. Any time you spend setting up a PC, designing a code test, and reviewing it will save untold headache in the future.

Pros: "Trust, but verify" - Ronald Regan. So many times I've seen and heard of people finally let go from a position, where in the interview you'd think you were getting a rock star. The proof is in the pudding; I want to see what they can do. It will represent what happens once you invest time and money hiring someone and stick a new project in front of them.

Answer 7

Cons:

• Requires you to have a technical person in the interview

Pros:

• Saves a LOT of time and heartache down the road if you prevent hiring someone who can't program

Answer 8

When I interviewed for my current job, I was given a list of questions to write code for by the recruiter. I was very impressed because the questions were obviously written by someone who had a deep knowledge of SQL, so it works both ways.

Answer 9

You really want to have the person write code in the interview - even better, get them to pair program with a member in your team for X amount of time (whatever you can comfortably afford in time/manpower).

It's pretty much one of the only ways you can tell if that person can code or not.

I slightly prefer the pair programming as it'll show their team work, gives them a real IDE to work with and lets them work on a 'real' problem (the other person in the pair can guide them past any environmental specifics that the interviewee could never reasonably know about).

We've started using this hiring policy and we're really happy with the results.

Answer 10

You judge a bird by its feathers and a programmer by his/her code.

When I started with the current company I'm working for they asked me to write some C code that generates or checks the parity bit of some binary input (depending on whether you are encoding or decoding). This was an interview question exactly because these kinds of problems are solved during work. Of course I'm thinking of not parity checking but rather working on a low-level.

Answer 11

All the answers so far (which I read) didn't address the fact that the Joel Test is NOT (just) a best practice list for entrepreneurs but a check list to ease your evaluation of a prospect employer.

The thing is ... if they thoroughly test their candiates then they probably hire people who know their stuff ... that means for you

1. less headache and also
2. the increased chance of learning something from your co-workors

instead of bug-fixing after them ...

Answer 12

I would say:

Pros

• Demonstrates the candidate has at least a passable knowledge of programming, since resumes can be fabricated/embellished
• If the interviewer discusses the code with the candidate, as opposed to it being more like a written test, could be a good indicator of how you "mesh" socially and if the candidate is a good fit for the company/company and team are good fits for the candidate

Cons

• Could be insulting to candidates if the questions are rote/trivial nonsense that would never come up during the job (e.g. "Write a bubble sort" when all modern frameworks that one would be using on the job have sorting built in, "Reverse a string" when there's a built-in Reverse method or similar, or for written tests things like "What are the arguments to the Foo method of the Bar class", when any idiot could Google that or use the documentation) as opposed to architectural/design questions that show the candidate can get things done and solve business problems.

Answer 13

One pro is that it shows that somebody does have basic knowledge of programming or whatever (the last time I encountered that, I was surprised how basic the SQL question was). It also can serve as a basis for a technical discussion, asking why the candidate did such and such, and how it could be improved.

It does take time in the interview, which could be used for other things. Moreover, writing code on a whiteboard is not a natural environment, and some people will have more and less serious problems with it. It could cause you to miss a developer who just gets nervous without normal tools or references.

Answer 14

Programming is a highly technical skill with a bunch of clear "deliverables." Either a candidate can or cannot deliver them. So there are no "cons" to asking technical questions. It's entirely in order to say, "show me some code for this application, or "show me the code for an application you have ALREADY written."

NOT doing so could lead to a result like the following: A rich man interviewed a tutor to teach his children to play chess (as a mind-expanding exercise). The tutor opened up a checkered board and started talking about the 64 squares but didn't touch a chess piece. Pressed for time, the father hired the tutor anyway. And the tutor taught the children to play CHECKERS.