How to Interview AI Engineers (LLM, RAG, and Prompt Engineering Roles)

"AI engineer" is three different roles in a trench coat

Before you can design the loop, decide which of the following you are actually hiring:

1. Applied AI engineer — builds product features on top of foundation models. Owns prompts, evals, retrieval, latency, cost. Lives in your application codebase.
2. ML engineer (model-side) — trains, fine-tunes, or distills models. Owns datasets, training infra, evaluation suites. Lives in your training stack.
3. AI infrastructure engineer — runs the inference, vector DB, and observability layer. Owns latency budgets, GPU utilization, deployment pipelines. Lives in your platform.

These three roles share zero of their day-to-day work. A loop designed for one will mis-evaluate the other two. The rest of this guide focuses on the applied AI engineer, because that is what 80% of teams hiring "AI engineers" in 2026 actually need.

The four signals that actually predict on-the-job performance

After auditing dozens of applied-AI hiring loops, four signals consistently separate engineers who ship working LLM features from engineers who ship plausible-looking demos:

1. Prompt iteration discipline. Can they turn a vague product requirement into a prompt, an eval, and a measurable improvement loop?
2. Failure-mode reasoning. Do they default-assume the model will hallucinate, leak tokens, or pick the wrong tool — and design accordingly?
3. Eval design. Can they design an evaluation that catches the regression they actually care about, not just BLEU score?
4. Cost and latency awareness. Do they reason about tokens, caching, model selection, and batch shape — or do they reach for GPT-5 for every call?

None of these are testable with a LeetCode question. All four require a different interview format.

The four-round loop that works

Round 1: Async take-home (90 minutes, paid if your culture supports it)

Give the candidate a small, broken RAG system and a real user complaint. Example:

Our customer support bot keeps confidently citing a product feature that does not exist. Here is the repo, the prompt, the retrieval pipeline, and three example failures. Find the root cause and propose a fix you can defend.

What you are looking for:

• Do they read the retrieval results before blaming the prompt?
• Do they identify that the embedding model is probably wrong for this domain?
• Do they propose an eval to catch the regression next time?
• Do they distinguish "fix the symptom" from "fix the system"?

A take-home is the only format that gives you a clean read on debugging discipline. Live coding rounds are too short and too high-pressure to see whether someone actually reads the data. Pair it with a live followup so AI-generated submissions don't escape.

Round 2: Live prompt engineering session (60 minutes)

Pair the candidate with an interviewer and a real task: take a vague product brief and turn it into a working prompt with measurable evals, in real time, in your actual tooling.

Useful sub-tasks:

• Write the first-draft prompt.
• Write three eval cases — including one that should fail.
• Iterate the prompt until evals pass.
• Discuss what they would measure in production beyond the eval set.

Signal you are looking for: do they think in evals first, prompts second? Engineers who have shipped LLM features know that the prompt is downstream of the eval. Engineers who have only prototyped do the opposite.

A collaborative coding room with Monaco + Yjs works for this — the candidate writes prompts and eval scaffolding in real time, and the interviewer can intervene exactly when it matters.

Round 3: System design — but make it AI-shaped (60 minutes)

Drop the candidate into a realistic applied-AI design prompt:

Design a customer-facing meeting summarizer that ingests a 60-minute transcript and produces an action-item list. It must run in under 30 seconds, cost under $0.05 per call, and refuse to summarize content outside our product domain.

This is a system design interview with AI-specific constraints. Score it with the same dimensions — clarification, tradeoffs, failure modes, cost — but watch for the AI-specific moves:

• Do they consider chunking and map-reduce vs single-shot?
• Do they reason about model selection (smaller model + better prompt vs larger model)?
• Do they propose a guardrail for off-domain content (classifier? prompt? rejection eval?)?
• Do they design observability for hallucination rate, not just latency?

Bad candidates default to "use GPT-5, prompt it, return the answer." Good candidates spend 10 minutes on the cost/latency budget before drawing a single box.

Round 4: Behavioral on a shipped AI feature (45 minutes)

A structured behavioral round anchored on a specific feature they shipped. The questions that work:

• "Walk me through the eval suite you built. How did it change over time?"
• "Tell me about a regression that shipped to production. How did you find it?"
• "Describe a time you used a smaller model where a larger one would have been easier. Why?"
• "Tell me about a feature where the LLM was the wrong tool. What did you ship instead?"

That last question is the highest-signal one in the loop. Engineers who have only used LLMs as a hammer cannot answer it. Engineers who have shipped real product can usually tell you about three.

Questions that look smart but are not

These come up constantly in AI engineer interview guides and are mostly noise:

• "Explain how transformers work." Trivia. Every candidate has read the same blog post. Tells you nothing about whether they can ship.
• "What's the difference between RAG and fine-tuning?" Already in every training set. Use a followup pattern instead: "tell me about a time you picked one over the other and why."
• "Write a transformer from scratch." Only relevant for the model-side ML engineer role. For applied AI, this is signal of academic background, not shipping ability.
• Whiteboard prompt-engineering ("write a prompt that does X" with no eval and no iteration). Untethered from how the work actually happens. A prompt in isolation, without an eval, is theater.

Calibrating for seniority

The same loop works across levels — what changes is what you weight.

A staff applied-AI engineer should be evaluated mostly on system design and behavioral — by that level, you are buying judgment, not implementation speed. A mid-level engineer should be evaluated mostly on take-home and prompt session — that is where their value is. See the senior engineer interview loop for the equivalent breakdown on general engineering roles.

Integrity, but smarter

For applied-AI roles specifically, banning AI assistance during the interview is incoherent. The job is using AI well. The interview should reflect that.

What we recommend instead:

• Allow AI assistance openly in the take-home. Score the use of AI, not its presence. The candidate who uses Claude to explore the codebase and then writes a thoughtful root-cause analysis is exactly the candidate you want.
• Disallow AI in the live prompt session. The point is to watch them iterate. Use keystroke and screen signals to catch silent AI use.
• Require a live followup on every take-home. This is the single highest-signal interview most teams skip. Pair an LLM-coherence pass with a 30-minute live walkthrough.

The candidate who can defend their take-home for 30 minutes wrote it. The one who cannot, did not — regardless of what the keystroke signals say.

What to do next

If you are setting up an applied AI engineering loop from scratch:

1. Pick one of the three role definitions and write a job description that matches.
2. Build (or reuse) one broken-RAG take-home. Iterate it on internal engineers first.
3. Set up the live prompt session in a collaborative editor with shared eval scaffolding.
4. Train interviewers on the four signals above before they run a single round.

The teams hiring AI engineers well in 2026 are not the teams with the most clever questions. They are the teams whose loops mirror the actual work — iterate, eval, debug, ship — and who score for the judgment that takes years to build.