The job has two halves and screens only test one
The technical half is well covered by every assessment tool on the market: model a warehouse, write the query, design the pipeline. It is testable, it is gradeable, and candidates expect it.
The other half is translating a vague request from someone non-technical into a specification, then reporting progress in a way that person can act on. On a small team this half is most of the job, and it is the half that goes wrong.
The single most predictive question
"Tell me about a time a stakeholder asked for something they did not actually need. What did you build?" A weak answer builds what was asked. A strong answer describes the conversation that changed the request.
Pipeline design — test for failure, not for the happy path
Any competent candidate can design a pipeline that works when the data arrives on time and in the expected shape. Data never does that for long. So the screen has to be about what happens when it does not.
| Weak | Strong |
|---|---|
| Brittle, monolithic scripts. A schema change upstream means a person intervenes. | Modular and idempotent. A rerun is safe, and a type change fails loudly at the boundary rather than silently downstream. |
| Late-arriving data is handled by rerunning yesterday manually. | Late-arriving data is a designed-for case, with a stated correctness trade-off. |
| Quality is something the analytics team notices afterwards. | Quality is validated at ingestion, and the data documents itself. |
A cheap, high-signal exercise: hand them a small dataset with three deliberate defects — a duplicated key, a timezone inconsistency, and a column that changed type midway. Ask what they would do. You are not testing whether they spot all three. You are testing whether their instinct is to clean it once or to make the pipeline refuse it.
Ownership, and the phrase to listen for
"That was the analytics team's job" is worth paying attention to. Sometimes it is a fair description of a real boundary. Often it describes someone who shipped a pipeline and considered correctness to be somebody else's problem from that point on.
On a small team there is no one to hand that to. The person who builds the pipeline owns whether the numbers are right, and you want to know before you hire whether they see it that way.
What not to screen for
- Whiteboard SQL under time pressure. It tests recall and nerves. Give them an editor and a real question instead.
- Your specific warehouse or orchestration tool. The concepts transfer in days; the judgement does not.
- Volume bragging. Having worked with large data is a fact about a previous employer, not a skill.
- Tool count on a résumé. Ten listed tools usually means shallow exposure to ten tools.
The stakeholder half of this role is the hardest thing on the page to test, and it generalises — assessing communication skills when you hire covers what to listen for and how to score it without penalising accent or fluency.