Kestrel Book a demo →

What Is an Evidence Cited AI Interview Scorecard? How to Audit One

An AI score without evidence is an opinion with a number on it. If an AI interviewer tells you a candidate scored 4 out of 5 on problem solving, the useful question is not whether 4 is right. It is what the candidate said that earned it.

An evidence cited AI interview scorecard answers that on the page. Every finding, strength or gap, is linked to the candidate’s own words, quoted exactly, so a hiring manager can check the claim in seconds instead of taking it on trust. This guide explains what such a scorecard contains, why model written scorecards need this safeguard, and how to audit any AI scorecard in five minutes. It ends with a template your human interviewers can use too. Sources were checked in October 2026.

What an interview scorecard is

An interview scorecard is a structured form for recording what you learned about a candidate against criteria you chose in advance. Greenhouse’s glossary describes it as a framework for everyone who interviews or assesses candidates, and lists four components: qualifications, technical skills, interpersonal skills and non required attributes that are nice to have.

The reason to use one is that structure predicts better than instinct. In their 2022 meta analysis in the Journal of Applied Psychology, Sackett and colleagues estimated an operational validity of .42 for structured interviews, the highest of the methods they compared, as summarised by SIOP. That is an average across many jobs, not a guarantee, but it explains why most hiring guides start with the scorecard.

A human scorecard has a known weakness. It is filled in from memory, after the fact, by someone who has already formed an impression. Open access HR textbooks list the usual errors: the halo effect, where one strong trait lifts every rating; the horn effect, its opposite; similar to me bias; contrast with the previous candidate instead of the job requirements; recency; and plain central tendency or leniency. A scorecard reduces these errors but does not remove them.

What changes when AI writes the scorecard

An AI interviewer can ask every candidate the same questions, record everything and draft the scorecard in minutes. That fixes consistency and speed. It also creates a new problem: the scorecard is now text written by a model, and models sometimes write things the source does not support.

How often depends on the task and the model. One recent benchmark, VeriGray, found that even state of the art models such as GPT 5 hallucinate in about 6% of sentences in summarisation tasks (Ding et al., arXiv, December 2025). That is a summarisation benchmark, not an interview scoring study, so it says nothing about the error rate of any hiring product. What it does show is that a fluent summary is not evidence.

The practical conclusion: treat an AI scorecard as a claim that needs checking, and choose tools that make the check fast.

What “evidence cited” means

In an evidence cited scorecard, each finding has four parts you can see together: the criterion, the score, the claim, and the exact words from the candidate that support it. Compare the two versions below. The candidate and answer are invented for illustration.

Without evidenceWith evidence
CriterionDatabase performanceDatabase performance
Score4 / 54 / 5
Claim“Strong understanding of query optimisation.”Diagnoses before changing anything, and weighs the cost of a fix.
EvidenceNone“I’d run EXPLAIN first to see if it’s a sequential scan. If the filter is on customer_id and created_at I’d try a composite index, but I’d check the write volume first because every index slows inserts.”
Can a reviewer verify it?Only by re interviewingIn ten seconds, against the transcript

The left column might be right, but you have no way to know. The right column lets you disagree precisely. You might think the quote shows good instincts but not depth, and score it a 3. That is a conversation about a sentence, which is far easier than a standoff about a number.

A complete scorecard should also show what it could not assess. If the interview never covered failure handling, the scorecard should say “no evidence” for it, not guess.

A five minute audit for any AI scorecard

Run this on a real scorecard from any vendor, including ours, before you rely on one.

CheckWhat to doPass looks likeFail looks like
1. Every finding has a quoteScan each score and claimA quote beside every findingScores with summaries only
2. Quotes are verbatimSearch the transcript for three quotesExact matchesParaphrases presented as quotes
3. Quotes support the claimRead the quote and ask if you would draw the same conclusionThe quote shows the behaviour claimedAccurate quote, unrelated claim
4. Gaps are visibleLook for criteria the interview did not cover“No evidence” or low confidence markedA confident score on an uncovered topic
5. Recommendation follows the scoresCompare the overall recommendation with the dimensionsYou can see how it was reachedA recommendation that contradicts its own evidence
6. A human can overrideTry to change the outcomeOverride is easy and loggedOutcome applied automatically

A scorecard that fails check 1 or 6 should not influence a hiring decision. If it fails check 2 or 3, test the tool further before you buy it.

Why this matters more as regulation arrives

The direction of travel is towards explainability and human oversight. India’s MeitY AI Governance Guidelines, issued in November 2025, are non binding but set out principles that include accountability and systems that are “understandable by design”. In the EU, AI used in employment is classed as high risk, and the date those obligations apply moved to 2 December 2027. India’s DPDP Rules are phased, with the main duties around 13 May 2027, so check the current dates before you rely on them.

An evidence trail will not make a tool compliant by itself, and this is not legal advice. It does give you something to show a candidate, an auditor or a colleague who asks why a decision was made.

What evidence citation does not do

  • It does not prove the score is valid. An accurate quote can still be a poor predictor of job performance. Validity needs separate evidence.
  • It can hide selectivity. A tool could quote the favourable lines and skip the rest. Read the whole transcript for at least a sample of candidates.
  • It does not fix bad questions. If the interview asks the wrong things, the scorecard will faithfully cite the wrong answers.
  • It does not replace the decision. The scorecard should recommend, and a person should decide.

A template you can use today

The same discipline works for human interviews. Copy this table into your notes tool and complete it during or straight after each interview, before any discussion with colleagues.

CriterionAnchor for 1 and 4Evidence (exact words or observed action)Score (1–4)Not covered
Example: Debugging approach1 = guesses; 4 = forms a hypothesis and tests it“...”
Example: Communication1 = unclear; 4 = explains trade offs simply“...”
Example: Ownership1 = blames others; 4 = describes fixing it in production“...”

Two rules make it work. Write the anchors before the interview, and do not enter a score until you have written the evidence.

How Kestrel does it

Kestrel is an AI native hiring platform for engineering teams at startups and scale ups. Its AI interview produces a scorecard with dimension scores, a confidence level and a recommended outcome (Advance, Hold or Reject). Findings cite the candidate’s own words, and each quote links to its timestamp in the transcript, which you can download as a PDF. The scorecard also lists areas with thin evidence and questions the candidate skipped.

The AI recommends and a person decides. After an interview the candidate moves to a human review stage, where a recruiter or hiring manager makes the call, and that decision is recorded in the activity log. The AI interview never rejects or hires anyone on its own.

Kestrel’s AI interviews are question based and run in text, with an optional voice mode that recruiters can enable per job. A separate Smart Canvas format covers system design. They run in the same pipeline as resume screening, MCQ and coding rounds and the applicant tracking system, so the scorecard sits next to the rest of a candidate’s record.

The audit above is a good way to evaluate us. You can book a demo, bring the checklist and run it on a real scorecard.

Frequently asked questions

What is the difference between an AI scorecard and a human scorecard?

A human scorecard is a form an interviewer fills in after a human interview. An AI scorecard is generated from an AI led interview and should cite the candidate’s own words for each finding. Both need anchored criteria, and both should end in a human decision.

Can an AI scorecard be biased?

Yes. Bias can come from the interview questions, the criteria, the model and the data it was built on. Evidence quotes make some problems easier to see, because a reviewer can check whether the cited words justify the score, but they do not remove bias. Test for it across candidate groups where you can lawfully collect the information.

Should an AI scorecard make the hiring decision?

No. Use it as a structured recommendation and evidence pack for the person who decides. Where human oversight is expected, a recommendation applied automatically is the wrong design.

How do I check that a quote is real?

Search for it in the full transcript. A trustworthy product makes this one click, and the quote should match exactly, not approximately.

Keep reading

All articles →
01 How to Evaluate an AI Interview Platform in India: A Buyer’s Checklist Eight questions, a weighted scorecard and a two week pilot. AI interviews 11 min 05 AI Resume Screening Thresholds: How to Set a Cutoff You Can Defend Rank, reject rarely, and read a sample of what you reject. Resume screening 11 min 03 How to Screen Your First 10 Engineers in India: A Founder’s Playbook A seven step screening process that runs on a spreadsheet and a few calendar invites. Hiring playbook 10 min