VanaraVANARAagentsSubscribe — $10/mo
proagentv2.0.0✓ verified

llm-evaluator

Use when an LLM feature's quality must become a number you can defend. Designs eval criteria, builds datasets, picks scoring (deterministic vs LLM-as-judge vs pairwise), runs baseline-vs-change, and gates regressions. Use before shipping any prompt/model change.

$npx vanara install llm-evaluator

Included with Vanara Pro ($10/mo) — unlocks all 206 items. See pricing →

Overview

You make LLM quality measurable. Without an eval, every prompt change is a guess, every model swap is a coin flip, and regressions ship silently because "it looked right in the demo." Your job is to convert a fuzzy claim ("the new prompt is better") into a defensible number ("pass rate 0.78 → 0.91 on a fixed 120-case set, no category regressed"). You own the measurement layer — the dataset, the ru

What it covers

Details

Tier
Pro — $10/mo
Version
2.0.0
Model
claude-sonnet-4-6
Tools
Read, Write, Edit, Grep, Glob, Bash
Depth
3 references · 2 examples · 1 runnable scripts
Verification
1/1 runnable checks passing ✓
Runs on
Your own Claude Code — no API keys
accessibility-auditorUse when auditing an existing UI for WCAG 2.2 A/AA conformance. Investigates semantics, keyboard operability, focus, color contrast, ARIA, forms, and reduced-motion, then reports findings per success criterion with severity and concrete fixes. Read-only — audits, does not refactor.ai-engineerUse PROACTIVELY when building or hardening LLM-powered features (chatbots, RAG, agents, summarizers, extractors, classifiers). Picks the right pattern (prompt → RAG → tools → agent), the right model tier, and ships with evals, guardrails, and cost/latency budgets.api-designerUse when designing a new HTTP/GraphQL API or changing an existing one — modeling resources, defining endpoint contracts, choosing status codes, pagination, filtering, error envelopes, versioning, and idempotency. Produces a reviewable API contract plus an OpenAPI snippet, not production code.api-documenterUse when documenting an HTTP/REST or GraphQL API for external or internal consumers — turning routes, schemas, or an OpenAPI spec into accurate reference docs with request/response examples, error catalogues, auth, pagination, and a getting-started guide. Read-only; produces docs, not code.app-release-managerUse when preparing a mobile app release or store submission. Walks through versioning, build config, store metadata, and the review-readiness checklist to avoid rejections.budget-plannerUse when building, reviewing, or stress-testing a budget. Turns priorities into a realistic allocation of money, tracks budget vs. actuals, forecasts cash flow and runway, and flags variances and shortfall risks early with concrete corrective actions.
← All agents