How to Calibrate Interviewers So Two Engineers Score the Same Candidate the Same Way

The disagreement that matters

Two of your engineers interview the same candidate. One says strong hire. The other says no hire. This happens constantly. The instinct is to blame the engineers — "they have different bars." The truth is that they almost always have different rubrics in their heads, not different bars.

Calibration is the process of making those internal rubrics the same. It is not training people to be more lenient or more strict.

The minimum-viable calibration loop

You do not need a quarter of training. You need a recurring 60-minute meeting:

1. Pick one recently-completed loop. All interviewers who participated attend.
2. Each interviewer re-scores the candidate's submission against the rubric, independently, before the meeting.
3. In the meeting, reveal scores simultaneously. Where there's disagreement on a rubric dimension, the two highest- and lowest-scoring interviewers walk through their reasoning.
4. Update the rubric anchors. Almost always, the disagreement traces to an ambiguous anchor — "demonstrates clear communication" means different things to different people. Rewrite the anchor with a behavioral description.

Run this every 2–3 weeks. By the third session the disagreements get smaller and the rubric tightens.

What the data says

When ClarityHire teams run this loop on structured interviews, inter-rater reliability (the correlation between two interviewers scoring the same candidate) typically moves from ~0.45 to ~0.75 within 6–8 weeks. That is the difference between "interviewing is a coin flip" and "interviewing is signal."

Common calibration mistakes

• Calibrating on hires only. You learn most from candidates near the bar. A clear strong-hire teaches you nothing about where the bar is.
• Letting the loudest voice anchor the room. Reveal scores simultaneously. Have the lowest-scorer speak first.
• Skipping the rubric update. If the meeting ends without changing a single anchor, you didn't calibrate — you talked.
• Calibrating the people instead of the rubric. "Sarah is too harsh" is the wrong frame. "The rubric anchor for level 3 is ambiguous" is the right one.

What ClarityHire does

The structured-interview reports preserve each interviewer's score and rationale per rubric dimension, locked at submission time. You can pull a calibration view that lines up scores across interviewers for the same candidate without leaking who-scored-what until everyone has submitted. That makes the calibration meeting honest.