What a technical skills assessment should measure
A technical skills assessment measures whether a candidate can produce the work the role requires, under conditions close enough to the real job that the result predicts something. It is a work sample with a rubric attached — not a quiz, and not an interview about technology.
That definition rules more out than it looks. A test of algorithm recall is not a technical skills assessment for most roles, because most roles do not require algorithm recall. A test of your specific framework is not one either, because framework familiarity transfers in a fortnight and the assessment is supposed to predict the year.
The useful question before you design anything: what does this role actually fail on? Not what is easiest to test. Technical hires rarely fail because someone could not invert a binary tree. They fail on judgement, on the parts of the work nobody wrote down, and on the handover to the people who depend on the output.
Why coding-puzzle screens mispredict
The puzzle screen survives because it is cheap, consistent and easy to defend internally. Every candidate gets the same problem, the results sort cleanly, and nobody has to argue about a score. Those are real virtues, and they are all properties of the process rather than of the signal.
- It measures practice, not ability. Puzzle performance improves sharply with a few weeks of dedicated preparation. You are partly measuring who had those weeks free, which correlates with things you did not intend to select for.
- It has no failure modes in it. Real work is ambiguous, under-specified and full of decisions about what not to build. A puzzle has one correct answer and states its own constraints.
- It is the most automatable part of the job. A problem with a known answer and a clean specification is now the thing a model does best, which makes the unproctored version of this screen close to worthless.
- It filters out a specific kind of strong candidate — experienced engineers who have not interviewed in six years and never learned the genre.
There is a fifth problem that only shows up once you compare levels. A puzzle grades one thing — whether the candidate finished — and finishing is the dimension where seniority makes the least difference. A strong senior and a strong junior both complete it, so the screen returns nearly the same signal for both, and the decision it was meant to inform gets made somewhere else. What separates them is judgement about scope, and above that, whether they picked the right problem at all. A puzzle can see neither, because it hands over the problem already chosen and already scoped.
None of this means live coding is useless. Watching someone work through an unfamiliar problem is genuinely informative. The mistake is the puzzle, not the format: give them a small piece of realistic work instead and the same session tells you three times as much.
What to test instead, by seniority
The same task graded the same way for every level is the second most common design error. Seniority does not mean doing the junior job faster — it means a different job, so it wants a different measurement.
| Level | Test for | Weak | Strong |
|---|---|---|---|
| Junior | Whether they can finish a scoped task and ask when stuck. | Silence, then an incomplete submission. No questions asked at any point. | Asks one clarifying question early, ships something complete but small. |
| Mid | Judgement about scope. What they chose not to build. | Builds everything mentioned, misses that two requirements conflict. | Names the conflict, picks one, says why in a sentence. |
| Senior | Trade-offs made explicit, and the failure modes they designed against. | Describes the solution. The word "trade-off" appears with no trade named. | States what was given up, and what would change their mind. |
| Staff and above | Whether they improve the problem statement before solving it. | Accepts the brief as written and executes it well. | Pushes back on the brief, correctly, before writing anything. |
The rows are cumulative — a senior candidate should still finish the task and still ask when stuck. What changes is which column decides the hire. For a staff role, a flawless implementation of the wrong thing is a fail, and the sample has to be ambiguous enough to make that possible.
Designing a one-hour work sample
One hour is the ceiling. Past that you are no longer screening, you are asking for unpaid work, and the strongest candidates decline first because they have other options. Five steps:
- Take a real task from the last month. Not a hypothetical. Strip out the context that requires knowing your codebase, and keep the decision that made it interesting.
- Make it under-specified on purpose. Leave one genuine ambiguity in the brief. What the candidate does with it is the most informative thing you will observe.
- Cut it until it fits in an hour — including the time they spend reading it. Then cut it again, because you know the context and they do not.
- Decide what "done" means before you send it. If you cannot describe a complete submission in two sentences, the task is still too big.
- Ask for a short note with the submission. What they assumed, what they left out, what they would do next. This is where the seniority signal lives, and it costs the candidate five minutes.
That closing note is the highest-value part of the whole exercise. It is also what makes a follow-up conversation possible: you have something specific to ask about, so the interview stops being a general chat about experience and becomes a discussion of a decision they made an hour ago. On the question of whether they wrote it themselves, when candidates use AI on your assessment covers why detection is the wrong response and what still works.
Scoring it consistently
Write the rubric before you send the task. Three or four competencies, and for each of them a description of what weak and strong look like — the table above is the shape. Score each competency separately and never collapse them into one number.
The test for your test
Could a competent person do this badly and still look good on paper? If yes, you are measuring presentation. Could an incompetent person do it well with an hour of preparation? If yes, you are measuring preparation.
Score submissions in a batch, not as they arrive. Scoring the first one sets an anchor that the next four are unconsciously measured against, and the effect is larger than most people expect. If you can, review without the name attached — you are assessing the work, and the rest of the file is available later.
Role-specific guides
The framework above is general. What actually distinguishes a strong hire differs sharply by role, and in each of these the thing worth testing is not the thing most screens test:
- Assessing a DevOps or SRE engineer — judgement under pressure rather than tool familiarity.
- Assessing a data engineer — the stakeholder conversation, which is the half that decides whether the hire works out.
- Assessing an engineering manager — the role that resists assessment hardest, and what no test will tell you.
What a technical assessment misses
A work sample is one hour of solo work with the answer unknown to nobody but the candidate. It cannot see how they behave in a code review, whether they finish things when the interesting part is over, or how they handle being wrong in front of the team.
It also systematically underrates people who work slowly and think well, because you imposed a time limit for your convenience rather than the job's. Where a strong candidate flags that trade in their submission note, believe them. For the wider picture of which methods cover which blind spots, see what candidate assessment is and how to choose a method.