Define the benchmark
Sample the task distribution, identify difficult cases, and write scoring rules with domain experts. Reserve a final test set that remains separate from repeated experiment selection.
Task benchmark and scoring rubricModel evaluation & research
Know which change actually improves the model.
A benchmark is useful when it predicts performance on the work you care about. We construct domain evaluations from real task requirements, calibrate scoring against expert judgment, and compare interventions under a fixed protocol. The result is a basis for choosing what to train, what to fix, and what to deploy.
Fixed task suite
Held-out tasksSame inputs. Same harness.
ReferenceDeployed modelPreserved configuration
InterventionCandidate modelOne controlled change
Independent decision
Task correctness
Critical failures
Serving budget
Promote only if criteria holdDoes the gain transfer to unseen tasks?
Separate the training objective from the final evaluation. Review error categories and uncertainty, not only an aggregate score.
The release decision is separate from the training reward.
Run the deployed model and its candidate through the same held-out task suite. Compare task quality, critical failures and serving constraints before promotion.
Which errors disappear inside an average?
Two models can receive the same overall score while failing in very different ways. We separate retrieval misses, reasoning errors, unsupported claims, and failed actions; then examine their frequency and consequence. This makes the next research question concrete: change the training data, improve the tools, revise the reward, or choose a different base model.
The research work.
Each intervention has a purpose, a record of what changed, and a way to assess its effect.
Compare interventions
Hold tools, prompts, and inference budgets fixed for model comparisons. Use targeted ablations to test whether gains come from the dataset, the training objective, or changes to the surrounding system.
Controlled comparisons and ablationsInspect the evidence
Report paired task outcomes and uncertainty, then examine disagreements with expert reviewers. Translate the important failure categories into regression tests and release criteria.
Error taxonomy and release decisionWhat determines
whether it works.
We set the evaluation around the application’s requirements, including the failures an aggregate score can hide.
- Measurement quality
- Measure grader agreement with expert labels; test sensitivity to response order, verbosity, and unsupported but persuasive answers.
- Causal clarity
- Use the same task instances and budgets for comparisons, with repeated sampling where stochastic outputs could change the conclusion.
- Operational usefulness
- Report completion and error severity alongside tail latency and cost per successful task, rather than token price alone.
What leaves
the lab.
Domain evaluation harness
Versioned tasks, scoring code, expert annotations, and the execution settings needed to repeat a comparison.
Comparative research report
Paired results, ablations, error analysis, and a recommendation grounded in the workload’s quality and operating requirements.
Release and regression checks
A protected test set, criteria for model promotion, and production signals that reveal whether the evaluation still represents current use.
Before we begin.
Can a public benchmark tell us which model to use?
Public benchmarks help narrow the candidates. Your task distribution, tools, and error costs determine the final choice. We test shortlisted models in the environment where they will operate, including the difficult cases a broad benchmark may barely represent.
Can we use an LLM as the judge?
Yes, for criteria that cannot be checked directly. We define the rubric, compare the judge with expert labels, and investigate disagreements. Executable tests and verified outcomes remain preferable wherever they capture the actual requirement.
Work with the lab
Bring us a research problem.
Tell us where the model falls short and what better performance would mean for your team.