llm-evaluation
Deep, practical guide to evaluating LLM outputs — building eval datasets, choosing reference-based vs LLM-as-judge vs pairwise/Elo vs human scoring, picking metrics (exact/F1, faithfulness, calibration), regression gating in CI, and controlling judge bias and variance. Includes runnable scorer.
Included with Vanara Pro ($10/mo) — unlocks all 206 items. See pricing →
Overview
You can't improve what you don't measure, and LLM features are easy to "improve" by vibes and ship a regression. Treat the feature like any other system: define what correct means, build a fixed test set, and score every prompt/model change against it. This package is the deep reference — evaluation methods live in references/, ready-to-run material in examples/, and a runnable scorer in scripts/s
What it covers
- The evaluation loop
- Choosing a scoring method
- Metrics that matter
- LLM-as-judge: bias and variance
- Building Datasets & Regression Suites
- Judge Bias, Variance & Calibration
- Evaluation Methods