Golden set: reviewed evaluation examples

What it is

A golden set records examples and the standard by which their answers should be judged. A reference may be an exact label, required facts or a rubric allowing several acceptable responses. Langfuse’s dataset model supports input and expected-output pairs; the choice and approval of those examples remains a human responsibility.

How it works

IonBin recommends recording a case ID, permitted input, reference criteria, source version and review decision. Include both ordinary work and important failure cases. An answer copied from a past conversation still needs checking.

Separate development cases from a held-out set reserved for final evaluation. Once a case influences prompt revisions, describe it as development evidence rather than claiming it remains an independent test. Version changes to both cases and expected answers so comparisons have a clear basis.

Benefits

A shared set makes disagreements concrete: reviewers can inspect the same case instead of debating impressions. It also helps distinguish a model regression from a changed business policy.

Limits and risks

A high score on a small or unrepresentative set can hide rare failures. Stale reference answers test yesterday’s policy. Reusing test cases during development weakens the independence of the final result; it does not make every earlier score meaningless.

An evaluation set, a training dataset and a trained model are different artifacts. Creating reviewed examples does not train a model or establish production accuracy.