Cost formula
Eval cost = (candidates × generation tokens priced per 1M) + (judgments × judge input/output tokens priced per 1M); Pairwise cost ≈ 2× pointwise judge input for same comparison count
Calculator
Eval & LLM-as-judge cost calculator
Plan CI smoke checks, nightly rubrics, and pairwise preference evals. Split candidate generation from judge scoring, and see cost per example and per full run.
Eval suite
Suite cost / run
$4.19$0.008375 / example
GPT-5 · 500 samples × 3 judge calls
Candidates$0.3438
Judge$3.84
Cost / run$4.19
Cost / judge call$0.002562
Judge calls / run1,500
Candidate calls / run500
Suite
Eval configuration
Multi-criterion rubric (3 judge calls / example)
Sensitivity
Sample-count cost table
How suite cost scales with N examples × judge calls.
N=50$0.4187
N=100$0.8375
N=250$2.09
N=500 (current)$4.19
N=1,000$8.38
N=2,000$16.75
Judge at $0.002562 / call; candidates from GPT-5 mini. One run costs $4.19.
Eval quality depends on rubrics and judges—this prices API calls only. Updated 2026-07-31. Approximate static list prices for planning only. Always verify against each provider’s official pricing page before production budgeting.
Guide
How to get value from this calculator
Price evaluation pipelines that generate candidate outputs and score them with LLM-as-judge pointwise or pairwise comparisons. Eval spend scales with dataset size, candidate count, and judge model tier—often overlooked in ML budgets until CI runs nightly. Plan eval cost before automating quality gates.
Why it matters
LLM-as-judge evals can exceed production inference cost during active development. Teams running large pairwise suites on flagship judges spend thousands monthly without a dedicated eval line item.
How to use it
- Enter eval dataset size (number of test cases).
- Set candidates generated per case and tokens per candidate.
- Choose judge model and pointwise vs pairwise scoring pattern.
- Enter judge tokens per scoring call including rubric and candidate text.
- Multiply by eval runs per month (CI frequency × manual reruns).
- Compare judge cost across smaller judge models if quality allows.
Planning tips
- Use a smaller judge model for screening and frontier judge for final eval subsets.
- Pairwise judging is more accurate but roughly doubles judge input tokens vs pointwise.
- Cache static rubric prefixes in judge prompts to cut input cost on large suites.
- Sample eval subsets in CI; run full suites weekly instead of on every commit.
- Use the suite-cost metrics at different sample counts before locking CI frequency.
- Track eval token usage separately in provider dashboards for visibility.
Frequently asked questions
Is LLM-as-judge necessary for eval?
Not always—rule-based metrics are cheaper. This calculator prices LLM judge pipelines when you choose that approach.
Are judge model prices official?
CentsPerToken uses approximate list prices for planning. Verify judge model rates on your provider's site.
How often should I run full evals?
Balance quality risk against cost. Nightly full evals on 1,000 cases add up—consider tiered eval strategy.
Does candidate generation use output pricing?
Yes. Generated candidates bill as output tokens on the generator model. Judge calls add separate input and output tokens.