At work, I have been responsible for conducting the coding tests during the interview process, and have been doing that for over a few years now. Over the time I have made some mistakes, learnt some things and this post is a summary of what one should consider when creating a coding test.

Coding tests are generally the first round of interview; it is either online or on campus, depends, but a coding round is considered as the first gate a candidate has to cross. How you conduct it has some implications, for example, if you asked the candidate to submit code online, you may want to verify that the code was written by the candidate, they understand it and that they did it in the stipulated time. You can do that with an additional pair programming round on campus. If you conduct the test on-campus there is little room for such doubt. But then you need to ensure things like a dedicated desk or meeting room for the test, a machine, and food and coffee may be. But there are some considerations integral to coding tests themselves, these are the ones we shall be looking at today.

Before we begin, let me call out that there are those who believe that interview is not the best process to understand the suitability of a candidate, and I would not disagree. But I believe that if done right, coding tests are the best way to understand a person’s approach to problem solving, their expertise and command on programming. How to go from there is your choice.

We shall see the points I have learnt to consider when creating a test problem, we shall also discuss the the thoughts and reasoning behind them.

First is to identify the skill set you need, the practices that matter to you for the role you are looking to fill. People consider knowledge of language C as a basic expectation in computer science, but if a developer is not expected to work on C it is pointless to ask a question with C in mind. Similarly, the most attractive questions for an interviewer: questions on shortest path algorithms, sorting and searching algorithms seem pointless. If there was ever a need to use them, I would not expect any employee to implement them anyway, why expect them to implement in interview. All we need is that they understand how those algorithms work. In an interview, we need to better target the skills that matter to us. Yet, it is better to skip any frameworks that you might use, frameworks can be taught and learnt, teaching problem solving approaches takes longer. I am also of the opinion that the programming language used to solve the problem should be of the interviewee’s choice, but that is not always feasible, since we should have enough skill in that language to review the code they submit!

Prioritize the skills in these categories: ‘must have’, ‘should have’, ‘good to have’ and ‘cool to have’. Model the problem in a way that targets the ‘must have’ skills and probably touches on some ‘should have’ skills. Skip anything under ‘cool’ category. This is not to say that we should hire a person who can do only the job we require them to do at the time of hiring. It is to say that the person should be able to do at-least that much. You have rest of the interview process to assess the person’s ability to learn or apply creativity, coding test is the gating criteria.

This matters a little in an online test, but is a huge consideration in an on-campus test. For on campus tests we need to have provisions for machines, food and meeting room or desk. If it is an interview drive, we need plan for as many number of machines as there are candidates or divide the candidates in batches, with delay in batches equivalent to time for test. If it is an online test, how much time to provide the candidate to revert and if the candidate can get a weekend to revert are considerations as well. Either way, how much time do we demand from the candidate is a question and we need to decide on a problem that needs an estimated amount of time to solve.

It is also important to accommodate for the pressure the candidate may be under during the test. This matters even more for on-campus tests; it can put your time calculations off considerably. I try to make the problem statement very clear and as definitive as possible. Being explicit about what is expected and what is not helps, because an unclear question leaves room for random implementations or shortcuts where I least expect them. That makes it difficult to evaluate the solutions on equal grounds. On the other hand, unclear questions leave room for creativity, but I have seen people missing the point with these.

Interview is a window, window for the company into the candidate’s capabilities and window for candidate into company’s culture. Just as the company needs to like the candidate, the candidate needs to like the company. And the problem statement is the first impression of the company! It is important to make the problem ‘fun’ to work on. I prefer to use casual way to describe the problem and try to make it enjoyable to read and to solve. It should have the feel of a fun place. You should choose a style that best describes your organization. As a side note, nerd jokes or references to Hitchhiker’s Guide to the Galaxy or Star Wars do not always work. 😉

Over the period I established a process for creating the problem statement, enabling a predictable estimation of time and solution. Over time you can choose to skip some of the steps, but to begin with here is the process: 

1. Once the problem is identified, create the write-up describing it. Instructions, guidelines and rules about time, how to submit the code, languages allowed etc should be mentioned at the top.
2. Once the problem statement is formulated, ask a peer to read and explain it to you. Ensure it means what you expect it to mean.
3. Solve the problem. Record the time.
4. Ask a peer to solve the problem, record the time taken. The peer here should be from the target experience range and skillset. Ask for their experience.
5. Anything you learn during these discussions, or while solving the problem, convert them into instructions and add them to the top.
6. Fix the scope of the problem to fit the time.
7. Solve again, when you are in a different mindset and record the time.

This process, although tedious, sets the expectations right. Ponder on how good is your solution in that time and how acceptable are your mistakes to yourself. Consider that you already knew the problem and had (unknowingly/knowingly) designed a solution in your mind. That should help set expectations from an interviewee ‘under pressure, under time limit, likely less experienced than yourself, groomed in different culture with different practices than yours who was just presented the question.’ Sounds difficult, right? It is not to mean that the evaluation should be lenient though, it is only to identify what is acceptable.

While evaluating, one should remember that you are not looking for candidates who solve the question the same way as you do. In other words, you are not looking for candidates who think just like you. The different the better, it is always surprising how many different ways you find the problem solved.

That’s all from me, all the best for finding the right candidates!