Method | 2026-08-11
Why a rubric is not just a score sheet
A high-quality rubric is an operating judgment system: it defines what to observe, how to decide, how disputes are adjudicated and how regression is checked after model changes.
Four layers of a rubric
- Capability layer: the capability object and failure cost.
- Task layer: samples, environment, inputs, outputs and boundaries.
- Judgment layer: dimensions, levels, exceptions, evidence and adjudication.
- Operations layer: training, consistency, versions, regression and issue loops.
Common failures
- Dimensions are abstract but cannot map to observable behavior.
- Rules have positive examples but no boundaries, counterexamples or dispute handling.
- Different evaluators use different implicit criteria.
- Old samples and weights continue after model versions change.
How to tell whether a rubric works
- Evaluators can understand and review evidence consistently.
- Boundary cases have clear escalation and adjudication mechanisms.
- Scores explain model failures instead of only giving totals.
- Version changes can reveal regressions through holdout sets and regression tests.