evals.sh
ProductModelsSecurityPricingDocsBlog
Sign inGet started
Blog

Product

Matching the model to the evaluation

Browser agents, repository review and bulk grading stress models differently. How we think about speed, depth and context when choosing one.

Evals.sh Engineering·October 23, 2026·8 min read

There is no single best model for evaluation. A model that is excellent at reading a large codebase can be a poor choice for grading two hundred short submissions, and the fastest model on the list may struggle to drive a browser for forty steps without losing the thread. The useful question is narrower: which model suits this evaluation?

Every module stresses a model differently

Evals.sh runs a few distinct kinds of work. They all end in the same place, a set of scored criteria with reasoning, but they get there in very different ways. That changes what you need from the model.

WorkloadWhat it leans onWhat failure looks like
Browser agentReliable tool use over many stepsMalformed tool calls, repeating the same action, forgetting earlier results
Repository reviewContext length and code reasoningMissing cross-file issues, surface-level comments, invented line references
Bulk gradingCost, latency and consistencySimilar work scored differently, slow cohorts, budgets spent early

Browser agents

Our UI evaluation is a long-horizon agent. It reads a page snapshot, chooses a tool, acts, checks the result and repeats, often for dozens of steps. Each step is small, but the session as a whole is demanding.

The property that matters most is tool-use reliability. The model has to produce a valid call every step, target the right element from the snapshot and pass sensible arguments. A model that is brilliant in conversation but occasionally emits a malformed call will waste steps recovering.

The second property is staying on task over a long horizon. Halfway through a session the model needs to remember which test cases have passed, which failed and what is left. We help by compacting older steps into a running summary, but the model still has to use that summary well and not re-test what it has already covered.

Raw reasoning depth matters less here than you might expect. Most individual decisions (“click Save, then confirm the toast”) are simple. What hurts is inconsistency across many simple decisions.

Repository review

Code review has the opposite shape. There are fewer steps, but each one can involve a lot of material. The agent explores the repository with read-only tools (an overview, directory listings, file reads and search) and has to hold enough of the codebase in mind to make judgements that span files.

That puts weight on two things. Context length decides how much code the model can consider at once before older material has to be summarised or dropped. Code reasoning decides whether it notices that the validation in one module is bypassed by a route in another, or that a test suite passes because it never asserts anything.

For a single important review, such as a capstone project or a pull request you care about, a deeper reasoning model is usually worth the extra time. The review reads like it was written by someone who understood the code, not just someone who scanned it.

Bulk grading

Grading a whole cohort changes the economics. The same rubric is applied many times, and the total cost and wall-clock time scale with the number of submissions. A model that is a little slower per call becomes a lot slower across a group.

The most important property here is consistency: two submissions of similar quality should receive similar scores, and the reasoning should apply the rubric the same way each time. A model that is occasionally insightful and occasionally erratic is worse for grading than one that is reliably solid.

Faster, lower-cost models often do well on bulk work when the rubric is clear and the criteria are observable. If the rubric asks for subtle judgement, you may prefer a stronger model and accept the longer run.

Speed, depth and context

Across all three workloads the same trade-off appears. Models that reason more deeply and handle more context tend to be slower and more expensive per call. Faster models are cheaper and more responsive but give up some depth. Neither end is wrong; they suit different jobs.

faster, cheaper per calldeeper reasoningfrontier / reasoning tierdeep repo review, hard rubricsbalanced tierbrowser agents, single reviewsfast / small tierbulk grading, triagecircle size ≈ usable contextthe trade-off you are choosing along
A qualitative picture, not a benchmark. Positions show the general shape of the trade-off between tiers, not measured scores for any specific model.

Inside each tier, model families have different strengths. Some are especially steady at structured tool calls, some are stronger at reading large amounts of code, and some offer a very large context at a reasonable speed. These characteristics also change as providers release new versions, which is one reason we avoid tying the product to a single family.

A practical heuristic

Start with the default. If results feel shallow, move up a tier. If a large group is taking too long or costing too much and the scores look stable, try a faster tier on a sample and compare before switching the whole run.

Defaults, and when to change them

Evals.sh offers models from several families, including OpenAI GPT, Anthropic Claude and Google Gemini. Each module ships with a default we have chosen for that workload: a model that handles the tool use and context the module needs at a sensible cost. Most people never need to change it.

When you do want control, you can pick a different model per evaluation. Good reasons to switch:

  • The work is unusually large or subtle. A big repository or a rubric that needs careful judgement benefits from a deeper model with more context.
  • The work is high volume and well specified. A clear rubric across many submissions is a good fit for a faster tier.
  • You want a second opinion. Running the same evaluation on a model from a different family is a cheap way to check whether a surprising score holds up.
  • Consistency with past runs. If you are comparing cohorts across terms, keeping the model fixed removes one source of variation.

Whatever you choose, the evaluation records which model produced it, so results stay comparable and auditable. The current list of models, and what each is suited to, is on the models page.

On this page

Every module stresses a model differentlyBrowser agentsRepository reviewBulk gradingSpeed, depth and contextDefaults, and when to change them

Try it on your own project

Free credits every month. No card required.

Get started

Keep reading

Engineering

Inside our computer-use agent: evaluating apps through a real browser

October 23, 2026

Security

What happens to your code when you run an evaluation

October 23, 2026

Engineering

Agentic API testing: beyond the 200 OK

October 23, 2026

evals.sh

Autonomous evaluation for software teams and educators. Agents that review code, drive real browsers and grade work against your rubric.

Product

  • Project evaluation
  • Code review
  • Frontend audit
  • API testing
  • Assignment grading

Platform

  • Models
  • Security
  • Pricing
  • Documentation

Resources

  • Blog
  • Guides
  • Contact

Legal

  • Privacy policy
  • Terms of service
  • Data handling

© 2026 Evals.sh. All rights reserved.

Customer data is never used to train AI models.