Data labelling and annotation

What it is

Data labelling assigns categories, spans, ratings or other structured judgments to examples. The useful output is a versioned annotation dataset with a defined purpose. An evaluation set adds reviewed reference answers or scoring rules; a trained model is a separate deliverable and does not appear merely because examples were labelled.

How it works

Begin with a small pilot containing ordinary, difficult and ambiguous examples. Write the label definitions, allowed evidence and uncertainty rule before asking reviewers to score. Use stable IDs so independent judgments can be joined later. Label Studio supports importing tasks and exporting annotation data. Verify which workflow features belong to your installed edition before relying on assignments or review controls.

Benefits

A clear record of labels makes model comparisons and error analysis easier to inspect. Keep the original label, reviewer rationale, adjudicated decision and rubric version rather than replacing disagreement with a single unexplained number. A pilot can reveal missing categories before a large collection is processed.

Limits and risks

Disagreement may reflect unclear instructions, incomplete evidence, different expertise or a legitimate difference of judgment. Investigate rather than automatically blaming the rubric. Avoid showing one reviewer another reviewer’s answer before independent scoring. Permission to read a document does not automatically grant permission to reuse it for model training. Check rights, sensitive fields and retention before exporting the dataset.