Evaluation set & rubric
Representative tasks, edge cases, and explicit scoring criteria, with a separation between development examples and held-out checks.
AI Evaluation Sprint / 2–4 weeks
LLM Evaluation & Benchmarking
Know which model changes are worth shipping.
Compare models, prompts, and agents on your real workload. Build a repeatable quality, cost, and latency baseline before the next release.
Discuss this sprint ↗Scope, fee, and acceptance criteria agreed before work begins.
The decision
When this sprint is useful
- Which model performs best on our actual customer tasks?
- Did the new prompt improve quality or just change the failure modes?
- What quality, cost, and latency trade-off should we ship?
The handover
What you receive
Repeatable benchmark
A runnable evaluation workflow for the models or system variants agreed in scope, including configuration and run instructions.
Failure analysis
A breakdown of failure types and quality, cost, and latency results. Uncertainty and evaluator limitations stay visible.
Release recommendation
An evidence-backed recommendation and regression baseline, with thresholds your team can review and maintain.
Before we start
A clear scope on both sides
Useful inputs for scoping
- Representative tasks or approved examples
- Access to the system or model endpoints
- Quality criteria and latency or cost limits
Scope boundaries
Public leaderboard scores are context, not a substitute for your workload. The sprint does not guarantee a quality gain; it establishes evidence for the next decision.
Fees depend on the decision, system complexity, and access requirements. Share your budget range and timeline to help establish fit.
Our research
Explore the thinking behind the work
Rewards & Verifiers →
Rewards translate outcomes and preferences into the signal used to improve a policy. They may come from humans, learned reward models, AI judges, tests, or executable verifiers. This topic follows how reward signals scale, where they fail, and how evaluation can detect proxy optimization before it becomes a production problem.
Environments →An RL environment defines the tasks, observations, actions, transitions, and outcomes through which a policy gains experience. For frontier AI, environment engineering increasingly includes browsers, code repositories, simulators, synthetic users, and tool APIs. Environment diversity and fidelity often determine whether gains transfer beyond a benchmark.
Post-training →Post-training turns a pretrained model into a useful system through supervised fine-tuning, preference optimization, reinforcement learning, distillation, and targeted data generation. The choice of method should follow the desired behavior, available feedback, and evaluation setup rather than a preference for one algorithm.
Read our RL Scaling Bottleneck Map →Common questions
Before an engagement
Can you evaluate agents as well as individual LLMs?
Yes. Agent evaluations can include task completion, tool use, recovery, human escalation, and the cost and latency of the whole workflow. The unit of evaluation is agreed during scoping.
What if we do not have an evaluation dataset?
Dataset design can be part of the sprint. We first check whether representative tasks, approved examples, and domain reviewers are available. Synthetic cases need review before they can stand in for real workloads.
Can we keep using the benchmark after the sprint?
The handover includes the agreed evaluation workflow, configuration, and documentation so your team can rerun it. Ownership, access, and any third-party dependencies are specified in the proposal.
Next step
Bring a real problem.
Start with the scope.
Describe the goal, constraints, and timeline. We’ll review the fit and discuss a useful next step.