Writing · Hiring
How to Interview Someone Who Says They Direct AI Coding Agents
The interview questions that separate people who actually run AI coding agents in production from people who have used a code-completion tool and updated their CV — what to ask about the harness, review, proof and failure, with the answers each kind of candidate gives.
Somewhere in your current stack of CVs is a line that says the candidate "directs AI coding agents" or "operates an agentic delivery pipeline". Somewhere in that stack there may also be a candidate for whom the line is true. The uncomfortable part, if you are the hiring manager, is that your interview loop as currently constructed almost certainly cannot tell you which one you are talking to.
The vocabulary is free now. Anyone who has spent a fortnight with Copilot and a few evenings with podcasts can speak fluent agentics: orchestration, multi-agent, evals, context windows, human-in-the-loop. Code completion is genuinely useful — but it is a different activity from being accountable for software agents produced at volume, over months, with the failures that entails. The two histories produce different answers, but only to different questions, and the standard loop does not ask them.
Consider what the standard loop measures. Live coding measures typing under observation — precisely the activity this way of working de-emphasises. System design measures fluent abstraction, and agentic abstraction is currently the easiest kind to fake, because few interviewers can pressure-test it two levels down. The take-home measures whether the candidate can use an agent, which by now everyone can, rather than whether they can answer for one.
I have spent fifteen years around hiring conversations, mostly in regulated fintech, and for the last stretch I have run my own delivery this way daily — several agents in parallel, gates between them and anything that matters. What follows are the questions I would put to anyone claiming the same, with the answer a practitioner gives and the answer a tourist gives. You do not need to run agents yourself to use them. Every one of them tests for two things that cannot be manufactured in the room: specificity and scar tissue.
First, redirect the whole conversation: harness, not model
The cheapest correction to your loop is to stop asking about models and start asking about the harness — everything wrapped around the model. Which model is strongest this month is podcast knowledge; it changes quarterly, and holding an opinion about it is evidence of nothing. The harness is different: how work is isolated, how it is reviewed, how completion is proven, what happens when it all goes wrong. That is where practice lives, because nobody builds a harness until an agent has burned them. A tourist arrives with model opinions and no harness. A practitioner will usually volunteer, unprompted, that their model opinions are held loosely and their harness rules are not — because one of those is expected to be wrong by next quarter and the other was paid for.
"Walk me through what happens between assigning a task to an agent and that change reaching your main branch"
Open with this, deliberately broad, and then say nothing. You are listening for whether the answer contains stages with boundaries, or just a conversation.
A practitioner describes a pipeline. The details vary, but the shape does not: isolation first, so that in-flight work cannot touch anything shared; a review step owned by something other than the builder; evidence collected before anything closes; and an explicit merge performed by a person or a gate — never by the agent that did the work. In my own setup, every parallel task runs in its own git worktree: a separate checkout of the repository on its own branch, one task per lane, so a half-finished or failed lane cannot poison a trunk it has never touched. That specific mechanism is not the point. The presence of some named mechanism at each boundary is.
A tourist describes a chat. "I write a detailed prompt, the agent makes the changes, I review the diff, we iterate until it looks right, then I merge." That is pair-programming with a very fast partner — a perfectly reasonable way to work, but nothing in it is being directed, and nothing in the answer exists outside the chat window.
The follow-up that settles it: "what does that look like on disk?" A practitioner gets more concrete under that question — directory layouts, branch conventions, the actual command they run. A tourist goes abstract, because there is no disk; there is only the conversation they are remembering.
"How do you review code you did not type?"
This is the volume question, and the honest answer begins with a concession: at agent throughput, reading every line personally stops scaling. What you want to hear next is the structure they built in place of heroic attention.
The strongest single answer, and the one I hold as an absolute rule myself: the thing that builds a change is never the thing that clears it. In my pipeline, if one engine writes the diff, a different vendor's engine reviews it — handed the change cold, with the stated requirements, and asked what would break in production. The reasoning matters more than the mechanism: a model re-reading its own output reads it through the same assumptions that produced the bug, so the blind spot that wrote it is the blind spot checking it. A separate engine has no investment in the diff. Nothing in a serious pipeline self-certifies, and a candidate who has run one will say something of that shape whatever their tooling looks like.
The tourist answers split two ways. "I read every line carefully" — which either is not true or concedes that their throughput is capped at their own reading speed, in which case what exactly is being directed? Or "I have the agent write tests for its own code" — offered without noticing that the tests share an author, and therefore share the author's misunderstanding of the requirement, with the code they are supposed to check.
"Tell me about a time an agent was confidently wrong — and how you found out"
Everyone who has operated agents for real has these stories, and they are strange enough to be specific: the test suite reported green that was never actually run; the assertion written so it cannot fail; the mock that verifies another mock; the crisp, confident "done" summary for a feature that does not exist in the build. The anecdote itself is not what you are grading. You are grading what they built afterwards.
A practitioner's answer ends in a mechanism: completion claims treated as untrusted input, verbatim command output demanded rather than summaries of it, a screenshot of the actual rendered screen for anything a human will look at, evidence checked against the exact change being graded. The pattern to listen for is receipt over report — the agent's account of what happened is an input to verification, never a substitute for it.
A tourist's answer ends in a prompt. "You get better at prompting, so it happens less." Or it ends in the market: "the newer models mostly don't do that any more." Both answers tell you that nothing this person shipped ever depended on them catching the lie — because when it does, prompting stops being a satisfying answer within the first week.
"How do you know a thing is actually done?"
The sharpest discriminator of the lot. Ask it plainly, and do not accept the first answer.
A practitioner treats "done" as a property of evidence, not of anyone's report — including their own. In my pipeline a task is not done until a proof artefact exists on disk: the verbatim output of the test that had to pass, the independent reviewer's written verdict, the screenshot where the change is visual — tied to the exact commit being graded, and fail-closed, meaning missing or stale evidence reads as not done rather than probably fine. "Tests pass" written in a commit message is a claim, and a claim is not a receipt.
A tourist says: "the agent confirms it, and CI is green." Green CI sounds independent until you remember who wrote the tests. Agents produce plausible test suites generously — coverage that climbs without behaviour being pinned down — and a green run over tests nobody has read pins down nothing. The gap between "the agent reports it passed" and "this command, run now, produces that output" is where a remarkable share of reported progress quietly evaporates; a practitioner has watched it happen often enough to stop trusting the first form.
"What has this approach cost you?"
Ask it exactly that plainly, and treat the answer as the most reliable signal in the loop. Months of real operation guarantee tuition: a lane of work orphaned and lost because nothing accounted for it at the end of a session; an agent that burned through its usable budget and simply stopped, mid-file, without a word; a measurement overridden on the strength of a confident argument, shipping a regression; a gate skipped once, for urgency, letting through exactly the class of defect the gate existed to catch.
A practitioner names one, prices it in concrete terms — the days redone, the release reverted, the work re-derived from nothing — and, crucially, states the rule they added afterwards. That mechanism-per-scar pattern is the signature of real operation. Every rule in a working pipeline is a fossilised incident, and people who have the rules can always tell you about the incidents.
A tourist reports either no failures or costless ones. "Sometimes it hallucinates, so you have to check its work" is a review of the technology, not an account of operating it — no cost attached, no rule created, nothing learned that a podcast could not have supplied. Be most suspicious of the flawless account. Nobody runs agents against production work for months without paying tuition, so a candidate with none to report either has not done it or is not being straight with you about it. Either way, you have your answer.
"Which model do you use?"
Ask this one as a gentle trap. The tourist has a favourite and will defend it, because the favourite is the whole of their expertise. The practitioner answers with a policy, because for them it is a routing question, not a loyalty. The expensive, high-effort models are reserved for where being wrong is costly — novel design, security-sensitive changes, the adversarial review itself, a bug that has survived a couple of attempts — while the bulk of the work (mechanical refactors, tests to an existing pattern, configuration) goes to cheaper, faster models entirely capable of it. Putting everything on the strongest model available is not rigour; it is waste dressed up as diligence, and matching the model to the task is one of the few skills of this job you can actually hear in an answer. A practitioner also expects this month's rankings to be wrong by next month, and their routing logic to outlive them.
The cheat sheet
| Question | A practitioner sounds like | A tourist sounds like |
|---|---|---|
| Task to main branch — walk me through it | Named stages with boundaries: isolation, independent review, evidence, an explicit merge the agent never performs | A conversation: prompt, diff, iterate, merge |
| How do you review code you did not type? | "Reading every line stops scaling, so —" followed by structure; the builder never clears its own work | "I read every line carefully", or the agent checks itself |
| Confidently wrong — how do you find out? | A specific incident, ending in a mechanism: claims are untrusted until the receipt exists | A prompting anecdote, or "newer models don't really do that" |
| How do you know it is done? | An artefact on disk, tied to the exact change, fail-closed | "The agent confirms it and CI is green" |
| What has it cost you? | A named failure, a concrete price, and the rule added afterwards | No failures, or costless generic ones |
| Which model? | A routing policy matched to task difficulty, held loosely | A favourite, defended |
What you are actually hiring
The market has mislabelled this role, which is why the CVs are hard to read. The person you are looking for is not a model whisperer. They are a delivery engineer whose working material happens to be agents: isolation so failure cannot spread, review that nothing is allowed to grade itself past, evidence in place of assurances, and an accounting of everything in flight. That discipline is considerably older than the technology it is currently pointed at — assume nothing, verify everything, design for the failure case — which is exactly why the genuine article is recognisable across whatever tooling they happen to run.
Notice, finally, that none of these questions require you to be deep in the technology yourself. Each one tests whether detail exists one level further down than the vocabulary, and whether the experience left marks. Tourists run out of detail precisely one level down, and arrive unscarred. The vocabulary will keep churning for a few more years, the way it always does. The candidates worth hiring are the ones whose answers would still stand up if you deleted the word "agent" from every one of them — because what remains, in their case, is engineering.