Rubric for scoring AI output

What it is

A rubric turns a judgment such as “usable support reply” into distinct checks. For example: does the reply reflect the supplied policy, avoid unsupported promises and identify when a human needs to intervene? These are illustrative criteria, not a universal scoring standard.

How it works

Write one criterion per property and describe pass, fail and insufficient-evidence cases. Separate factual checks from tone so good writing cannot cancel an incorrect claim. Choose a scoring scale and state which failures block use regardless of the total.

Ask reviewers to score a sample independently, then inspect disagreement. It may reveal unclear wording, missing context, different expertise or a genuinely ambiguous case. Adjust the rubric where justified and retain the original scores.

promptfoo documents deterministic and model-graded assertions. Langfuse provides human annotation queues for scores and comments. Neither tool establishes whether the chosen criteria cover the business task.

Benefits

Reviewers can explain their decisions with concrete evidence. A written rubric also makes revisions visible when policy or risk tolerance changes.

Limits and risks

Consistency does not guarantee validity: everyone can apply an incomplete rubric consistently. Automated graders may miss errors or reward the wrong style. Compare their judgments with reviewed examples and recheck after grader changes.

Avoid letting the system under evaluation rewrite the pass criteria simply because it failed them.