Deeper dive: technical exercises
Showing off the moves
What's the point?
Though direct conversation with your candidates is an essential part of the hiring process, it’s hard to tell just from conversation where their technical skills lie. You can talk about technical topics all you like in your interviews, and in fact it’s good to have those conversations when hiring for a support role since a good part of the job is explaining technical topics to perhaps less-technical customers. But without seeing examples of their work you’ll never be able to have confidence that they have the requisite skills for the job.
But without actually hiring the candidate, or taking them on as a short-term contract, how can you get a view into their work process? This is where the tech exercise comes in. A good technical exercise resembles the day-to-day work of the role you’re hiring for, but in a much compressed timeframe and level of effort. As I mentioned in the last post, and so I won’t belabor it again here, there’s no way to be 100% certain that any given candidate is a fit. But a well-designed technical exercise can give you a lot of insight into their process and skillset. Here are some things to keep in mind when putting together the technical exercise, or exercises, in your own interview process.
How long should they be?
There’s a growing consensus across the industry that the hiring process should be as quick and streamlined as possible. It takes less time for the hiring manager and other folks on the hiring side, it’s much less onerous for the candidate, and coming to a conclusion faster is pretty much always preferable, all other things being equal. So a fairly short exercise makes sense—less time to do, faster to evaluate—even before addressing the second, more compelling argument: don’t make your candidates do real work for you. Technical exercises that simply boil down to ‘create this work product for me’ are both time-consuming and ethically wrong. If your candidate is doing real work, they should be paid accordingly. Unless you’re willing to cut a check for the time spent on your exercise, keep it short (a couple hours at most).
What should they be like?
Just as important as keeping it short is keeping it synthetic. What do I mean by synthetic? Well, go back to what I said above: don’t make candidates do real work for you in a tech exercise unless you’re planning to pay them. Even if there were no ethical issues around this, and there certainly are, consider the business implications. Do you want to share proprietary internal or customer information with someone who hasn’t officially signed on? Absolutely not. Do you want to provide your candidate’s output as a finished work product to your customers? Not unless you’re desperate. Just avoid all those issues entirely and make something up that’s similar to actual work, but denatured. Conveniently, the exercise of anonymizing actual customer issues gives you a golden opportunity to also simplify them, helping you limit the exercise to a reasonable size.
An example
Here’s an example. If you want to evaluate your candidate’s proficiency with the Linux command-line, because your support engineers are often troubleshooting system capacity issues, you could share with them an actual customer report on that topic. But how complicated was that issue? Was there any proprietary information in the report? Instead, consider taking the bones of the issue (the underlying hardware ran out of space) and building a synthetic exercise around it. You can share a similar vague customer report, simplify away a few of the confounding details, and end up with a simple problem statement that uses the same skillset:
The customer has reported periodic issues writing new files to disk. Investigate to determine the cause and provide a suggestion for resolving the issue.
This example could be shaped into a simple live exercise by generating a virtual machine with next to no disk space and letting the candidate investigate on their own, or with a bit more work you could generate a series of paper exercises if you’d prefer to have a live discussion. For example:
What commands might you run first? Why?
The commands indicate there is no disk space available. How can you figure out what’s taking up the most space?
The culprit is a runaway /var/log/messages. What is this file and what would you suggest doing about this?
How do you evaluate?
Similarly to evaluating interviews, it is important to know ahead of time what your criteria for success are. Work together with your team to establish not only the parameters of the exercise but also an objective scoring rubric. For example, the above exercise might have the following evaluation criteria for the live hands-on version:
Did the candidate correctly determine that the system was running out of space?
How long did it take them to get there?
Did the candidate run superfluous or harmful commands?
Did the candidate provide a correct and useful suggestion for ameliorating this situation?
For each question, an example of a correct and an incorrect answer should be provided. This will not only help keep you honest as you go through and evaluate various candidates’ exercises, but will help consistency if you have more than one person grading exercises. If you have several strong candidates, the question of whose technical exercise is stronger may come down to subtle variations in their processes, and those can be hard to see if different evaluators are not working from the same scoring rubric.
Remember too that these exercises are not the only important parts of the hiring process: always keep in mind that a strong score on a technical exercise doesn’t necessarily mean that the candidate will do well in live situations, and likewise a weaker score in a technical exercise might be counterbalanced by the fact that they’re particularly good at building rapport and interacting directly with customers. Take what you learn from these technical exercises as a useful data point, not as the only criterion that matters in determining a candidate’s overall fit for the role you’re trying to fill.
Thanks for reading Andy's Support Notes 💻💥📝!