Eval sample size calculator

How many questions does your eval need?

Standard sample-size math assumes every answer is graded correctly. Enter how often your grader gets it wrong, and see what that costs you.

The improvement

Current accuracy

Target accuracy

True accuracy, as a perfect grader would measure it.

Your grader

Fails correct answers

Passes incorrect answers

Estimate both from a grading audit. A golden set, human raters and an LLM judge each have their own rates.

The comparison

Same questions both versions answer every question Separate questions each version gets its own set

Versions disagree on

Share of questions where one version is right and the other wrong, with perfect grading.

Grader mistakes on the same question

Independent

A wrong answer key fails both versions together. Independent mistakes are the worst case.

What the grader lets you see

Real improvement

What your grader reports

The grey part is signal your grader's mistakes erase. You make up for it with more questions.

Questions needed

—

shared questions

Perfect grader

Your grader

Statistical power 1−β 80%

How often a real improvement of this size is detected.

Significance level α 5%

How often a difference is reported when there is none.

With shared questions,

What grader error costs

Questions needed for your improvement and design as the grader's error rate rises (same rate in both directions). The dot marks your current settings.

How this is calculated

The grader's observed pass rate is f + J × true accuracy, where f is the share of incorrect answers it passes and J = (share of correct answers it passes) + (share of incorrect answers it fails) − 1. The true gap shrinks by a factor of J.
Separate questions: a two-proportion normal approximation, using the pass rates the grader will report.
Same questions: McNemar's test, using the observed disagreement after grader mistakes. Mistakes on the same question can be independent or shared; shared mistakes create no fake disagreement.
Mistakes are assumed to be random and to treat both versions the same. If your grader favors one version, more questions won't fix it: audit and correct the grader first.
Error rates are treated as known. If you estimated them from a small audit, the true requirement could be noticeably higher or lower.