Product
Matching the model to the evaluation
Browser agents, repository review and bulk grading stress models differently. How we think about speed, depth and context when choosing one.
There is no single best model for evaluation. A model that is excellent at reading a large codebase can be a poor choice for grading two hundred short submissions, and the fastest model on the list may struggle to drive a browser for forty steps without losing the thread. The useful question is narrower: which model suits this evaluation?
Every module stresses a model differently
Evals.sh runs a few distinct kinds of work. They all end in the same place, a set of scored criteria with reasoning, but they get there in very different ways. That changes what you need from the model.
| Workload | What it leans on | What failure looks like |
|---|---|---|
| Browser agent | Reliable tool use over many steps | Malformed tool calls, repeating the same action, forgetting earlier results |
| Repository review | Context length and code reasoning | Missing cross-file issues, surface-level comments, invented line references |
| Bulk grading | Cost, latency and consistency | Similar work scored differently, slow cohorts, budgets spent early |
Browser agents
Our UI evaluation is a long-horizon agent. It reads a page snapshot, chooses a tool, acts, checks the result and repeats, often for dozens of steps. Each step is small, but the session as a whole is demanding.
The property that matters most is tool-use reliability. The model has to produce a valid call every step, target the right element from the snapshot and pass sensible arguments. A model that is brilliant in conversation but occasionally emits a malformed call will waste steps recovering.
The second property is staying on task over a long horizon. Halfway through a session the model needs to remember which test cases have passed, which failed and what is left. We help by compacting older steps into a running summary, but the model still has to use that summary well and not re-test what it has already covered.
Raw reasoning depth matters less here than you might expect. Most individual decisions (“click Save, then confirm the toast”) are simple. What hurts is inconsistency across many simple decisions.
Repository review
Code review has the opposite shape. There are fewer steps, but each one can involve a lot of material. The agent explores the repository with read-only tools (an overview, directory listings, file reads and search) and has to hold enough of the codebase in mind to make judgements that span files.
That puts weight on two things. Context length decides how much code the model can consider at once before older material has to be summarised or dropped. Code reasoning decides whether it notices that the validation in one module is bypassed by a route in another, or that a test suite passes because it never asserts anything.
For a single important review, such as a capstone project or a pull request you care about, a deeper reasoning model is usually worth the extra time. The review reads like it was written by someone who understood the code, not just someone who scanned it.
Bulk grading
Grading a whole cohort changes the economics. The same rubric is applied many times, and the total cost and wall-clock time scale with the number of submissions. A model that is a little slower per call becomes a lot slower across a group.
The most important property here is consistency: two submissions of similar quality should receive similar scores, and the reasoning should apply the rubric the same way each time. A model that is occasionally insightful and occasionally erratic is worse for grading than one that is reliably solid.
Faster, lower-cost models often do well on bulk work when the rubric is clear and the criteria are observable. If the rubric asks for subtle judgement, you may prefer a stronger model and accept the longer run.
Speed, depth and context
Across all three workloads the same trade-off appears. Models that reason more deeply and handle more context tend to be slower and more expensive per call. Faster models are cheaper and more responsive but give up some depth. Neither end is wrong; they suit different jobs.
Inside each tier, model families have different strengths. Some are especially steady at structured tool calls, some are stronger at reading large amounts of code, and some offer a very large context at a reasonable speed. These characteristics also change as providers release new versions, which is one reason we avoid tying the product to a single family.
Defaults, and when to change them
Evals.sh offers models from several families, including OpenAI GPT, Anthropic Claude and Google Gemini. Each module ships with a default we have chosen for that workload: a model that handles the tool use and context the module needs at a sensible cost. Most people never need to change it.
When you do want control, you can pick a different model per evaluation. Good reasons to switch:
- The work is unusually large or subtle. A big repository or a rubric that needs careful judgement benefits from a deeper model with more context.
- The work is high volume and well specified. A clear rubric across many submissions is a good fit for a faster tier.
- You want a second opinion. Running the same evaluation on a model from a different family is a cheap way to check whether a surprising score holds up.
- Consistency with past runs. If you are comparing cohorts across terms, keeping the model fixed removes one source of variation.
Whatever you choose, the evaluation records which model produced it, so results stay comparable and auditable. The current list of models, and what each is suited to, is on the models page.