How to Grade Coding Exercises Consistently Across Multiple Interviewers
The problem you're actually solving
A senior engineer reads a candidate's submission and gives it a hire. A different senior engineer reads the same submission and gives it a no hire. They are not both right. One of them is leaking personal taste into the score and calling it judgement.
Inter-rater unreliability is the single biggest source of randomness in a hiring funnel that already has plenty of randomness. It also disproportionately hurts candidates from non-traditional backgrounds, because "I just got a vibe" maps cleanly onto pattern-matching against the interviewer's own past.
Step 1 — rubrics with behavioural anchors
A rubric that says "Code quality: 1–5" is not a rubric. It is a vibe rendered as a number. A rubric is what you get when each level has a behavioural anchor — a description of what the candidate would have had to do for the score to be 3 rather than 4.
Example, for a "problem decomposition" axis:
- 5 — Names the right sub-problems before writing code. Re-asks the problem to the interviewer in their own words. Identifies an edge case unprompted.
- 3 — Decomposes correctly once they start typing. Misses one edge case but addresses it when asked.
- 1 — Starts coding before they've understood the problem. Has to be steered back twice.
A proper rubric template has 4–6 axes like this. The point is that two interviewers reading the same transcript should end up at the same level on each axis, because the anchors are observable.
Step 2 — calibration sessions, monthly
Pull six submissions from the last month. Strip names. Each interviewer grades them independently, then the group meets and compares scores axis by axis. The discussion is the product, not the average.
You're looking for:
- Axes where the variance is high. If "communication" scores swing wildly, your anchors aren't behavioural enough. Rewrite them.
- Interviewers who systematically grade high or low. Not a bad person — just a calibration shift. The fix is feedback, not removal.
- Cultural drift. What you considered a "3" eight months ago might be a "4" today because the bar moved. That's fine, but make it explicit.
Run this monthly with a small team, quarterly with a larger one. Skip it for a quarter and the variance comes back.
Step 3 — kill anchoring bias on the day
Two anti-anchors that cost nothing and save a lot:
- Score independently before the debrief. Each interviewer writes their rubric score before hearing what the other interviewer thought. Then compare. This is the single highest-leverage intervention in the entire system.
- Anonymise where possible. For asynchronous code submissions, strip name, photo, school, and prior-company fields from the reviewer's view. The grading view should default to anonymised mode for the same reason peer-reviewed journals do.
Step 4 — AI as a calibration check, not as the grader
A well-tuned LLM grader is consistent across submissions in a way humans aren't. Use it as a noise floor: if your human reviewer scored a submission a 4 on correctness and the AI scored it a 2, that's a useful signal — go look. The disagreement is often the most informative artefact in the loop.
Never let the AI score auto-reject. Never show it to a reviewer before they've scored independently. Use it post-hoc, as a sanity check.
What this looks like in ClarityHire
The grading view shows the candidate's submission, the rubric with behavioural anchors, and (optionally) an AI-generated first-pass score that's hidden until the reviewer commits their own. The interviewer-calibration view surfaces inter-rater variance per axis over time so you can see which axes are drifting before the candidates do.
Pair this with the structured scorecard downstream and the same calibration discipline carries from "did this submission pass" all the way through to "do we hire this person".
TL;DR
Consistency comes from behavioural rubric anchors, independent scoring before debrief, monthly calibration sessions, and AI as a sanity check (never as the grader). Skip any of these and your code interview becomes a coin flip with extra steps.