Most engineering interviews begin after the problem has been chosen. The interviewer supplies the specification, the constraints, and the expected form of the answer. The candidate solves it under observation.
Much of the work I care about begins earlier. The request is incomplete. The important constraint is hidden. Several people use the same word to mean different outcomes. The engineer must decide which question deserves an answer before writing the answer.
An interview that removes this work collects weak evidence about it. A difficult puzzle may reveal recall and pattern fluency. It gives little evidence about whether the candidate can turn an uncertain production question into a result that another person can inspect.
I want the interview to preserve enough uncertainty for judgment to become visible.
Start with an incomplete request
A useful exercise starts with a request similar in shape to the job. The prompt has a real objective and deliberately leaves some decisions open. It gives the candidate room to ask questions, state assumptions, and choose a first test.
For an engineer working on an AI system, the request might be: make a tool-using agent dependable enough for a consequential workflow. The candidate has to ask what dependable means. Does success mean a correct response, a confirmed external effect, or a recoverable failure? Which actions require authority? What evidence exists today? What happens when the tool times out after changing state?
The value of the exercise comes from the sequence of decisions. I watch whether the candidate:
- defines the user-visible result before choosing technology;
- finds the assumption most likely to change the design;
- separates known facts from working assumptions;
- chooses a small test that can reject a weak plan;
- makes failure behavior explicit;
- leaves a decision another engineer can review.
The interviewer can introduce one new fact halfway through. A dependency has weaker guarantees than expected, the latency target changes, or the workflow now includes an irreversible action. The useful evidence is how the candidate updates the plan.
Require an artifact
Conversation alone favors people who can produce a convincing explanation quickly. Engineering eventually has to become inspectable work.
The artifact should match the role. It may be a small implementation, a test plan, a decision record, a failure model, or a migration sketch. The scope must fit the available time. Completion matters because unfinished reasoning hides its hardest edges: naming the interface, handling an error, choosing a threshold, or stating what remains uncertain.
I want to see the point where the candidate converts a broad claim into something falsifiable. “We should add retries” becomes a policy with an idempotency condition and a terminal state. “We need observability” becomes a named decision that a trace or metric must support. “This design scales” becomes a workload assumption and a measurement.
A modest artifact with explicit limits gives me more evidence than an ambitious design whose key claims remain verbal.
Score evidence from decisions
Unstructured interviews reward familiarity between interviewer and candidate. A shared vocabulary can feel like shared judgment. The scorecard has to force a more disciplined comparison.
| Capability | Evidence |
|---|---|
| Problem framing | Names the observable result, user, and operating conditions. |
| Assumption control | Finds the uncertain claim that could reverse the plan. |
| Technical judgment | Chooses complexity in response to evidence. |
| Execution | Produces a bounded artifact that works or fails clearly. |
| Revision | Updates the decision when new evidence arrives. |
| Communication | Separates facts, choices, risks, and next actions. |
Each interviewer should record evidence before the group discussion. The discussion should begin with observed behavior, then connect it to the role. “Strong systems thinker” is a conclusion. “Identified the irreversible action, added an authority boundary, and removed a queue after measuring the workload” is evidence.
Keep the process fair
The exercise can resemble the work without recreating a stressful production incident. The candidate needs a clear time box, accessible tools, a stated evaluation method, and enough context to make progress. Any surprise should come from the engineering problem. The evaluation rules should be visible.
The same exercise should be run often enough to calibrate it. Interviewers should compare scores, inspect disagreements, and track whether the signals predict performance after hiring. A question that produces confident ratings but weak prediction is a poor instrument.
Candidate experience is also evidence about the company. An interview that rewards guessing the interviewer’s preferred solution teaches candidates that hidden preferences govern technical decisions. A process that rewards explicit assumptions, useful tests, and clear revision demonstrates a different operating standard.
Hire for the decisions ahead
A hiring loop should begin with the work the team needs to do over the next year. List the decisions that will recur, the failures the engineer must diagnose, and the artifacts other people will depend on. Build the interview around those capabilities.
The final question is practical: what evidence from this process supports giving this person real authority in this role?
If the evidence comes mainly from solving a problem whose statement, boundaries, and success condition were already supplied, the interview has skipped much of the job. A better exercise preserves the ambiguity that matters, requires a finished artifact, and changes one assumption so the candidate must update the decision.