Two meters: generate, then judge
Most suites first produce candidate answers, then score them. Re-judging stored outputs skips generation and is the cheapest way to iterate on rubrics.
Criteria multiply judge calls
A three-dimension rubric is often three judge calls per example (or one larger call). Either way, judge tokens usually dominate once the dataset grows past a smoke set.
Prefer batch for nightly runs
Offline evals tolerate hours of latency. Batch discounts cut candidate and judge spend when the provider supports them—use the batch toggle when planning nightly jobs.
FAQ
Is pairwise twice as expensive as pointwise?
Generation often is (two candidates). Judging may be a single comparison call with a larger context window—not always 2× the judge bill.