Engineering
Inside our computer-use agent: evaluating apps through a real browser
How Evals.sh drives an isolated, headless browser to click, type and verify user flows, and why we capture evidence for every step.
A web app is only finished when someone can use it. Reading the source tells you a button has anonClick handler; it doesn't tell you whether clicking it saves the record, shows an error on bad input, or silently does nothing. So our UI evaluation doesn't read the code at all. It opens the app in a real browser and uses it.
This post walks through how that agent works: the loop it runs, the actions it can take, how each job is isolated, what evidence it keeps, and the places where it deliberately stops.
Why a real browser
Static analysis and unit tests answer narrow questions well. They are poor at answering the question an instructor or reviewer actually cares about: does this app do what the brief asked for? That depends on routing, client-side state, network calls, validation messages, layout that hides or reveals controls, and the order in which a person does things.
A headless browser gives the agent the same surface a user has. It loads the deployed URL, runs the JavaScript, follows redirects and renders whatever the app renders. If the signup form only works when the email field is filled before the password, the agent finds that out the same way a person would.
Perceive, plan, act, verify
The agent is a language model running in a tool loop. On each step it reads the current state of the page, decides on one action, takes it, and checks what changed. That cycle repeats until it has enough evidence to score the app, or until it reaches its step budget.
- 01
Perceive
The agent reads a structured snapshot of the page: the accessibility tree, with roles, names and states for each element. It can also take a screenshot when the visual layout matters. - 02
Plan
Before touching anything it turns the task instructions into concrete test cases (“create a todo”, “mark it done”, “reload and confirm it persisted”). If the brief is thin, it writes reasonable test cases itself and says so in the report. - 03
Act
It issues a single browser action, such as clicking a button or filling a field. - 04
Verify
It re-reads the page and compares what happened with what should have happened. A passing check needs observed evidence, not an assumption that the click worked.
Two rules shape the loop. First, the agent must call a tool on every step; it can't drift into writing prose halfway through. Second, it finishes by calling a dedicated report tool with a fixed schema: one integer score per criterion and a rationale for each. When the step budget runs out, the final step allows only that report call, so a long session always ends with a structured result.
Long sessions also produce a lot of context. Once the conversation grows past a threshold, older steps are compressed into a rolling summary that keeps the test cases, what has passed or failed, which screenshots were taken and what they showed, and what is left to check. The agent keeps working from that summary plus its most recent steps.
The action vocabulary
The browser is exposed to the model as a small, explicit set of tools. Keeping the vocabulary narrow makes each step easy to log and easy to reason about afterwards.
| Action | What it does | Typical use |
|---|---|---|
| navigate | Load a URL, go back or forward | Open the app, visit a deep link, test a route guard |
| click | Click an element identified in the snapshot | Submit forms, open menus, toggle state |
| type / fill | Enter text into one field or a whole form | Signup, search, validation edge cases |
| select | Choose an option in a dropdown | Filters, settings, multi-step forms |
| wait | Wait for text to appear or disappear, or for time to pass | Async loads, toasts, spinners |
| snapshot | Read the accessibility tree of the current page | Perceiving state before and after an action |
| screenshot | Capture the rendered page | Proof points, failures, layout problems |
The agent can also evaluate small pieces of JavaScript against the page and read console messages. We nudge it to prefer those for precise assertions, such as counting rows in a list or reading a value from local storage, because they are cheaper and less ambiguous than inferring the same fact from a screenshot.
One sandbox per job
Every UI evaluation gets its own browser in its own isolated container. The container is created when the job starts, labelled with that job's ID, and talks to our service only through the browser tool protocol. Nothing is shared between jobs: no cookies, no cache, no logged-in sessions left over from the previous submission.
When the job finishes, whether it succeeds, fails or times out, the container is stopped and removed, and the scratch directory it wrote into is deleted. If a worker stops halfway through a job, the recovery process removes any container still tagged with that job's ID before the job is picked up again, so a crash doesn't leave a browser running.
Evidence for every step
A score without evidence is just an opinion. We record every step of the session as it happens: the tool the agent called, its arguments, the result and the model usage for that step. If you want to know why a criterion scored the way it did, you can trace the decision back to the actions behind it.
Screenshots are captured at milestones the agent chooses: before and after key interactions, at failures, and at states worth showing in a report. Each one is saved to the evaluation as it is taken and attached to the final result. If the agent reaches the end of its budget without taking any, it is prompted to capture a baseline screenshot before reporting, so a reviewer always has at least one view of the app.
The final report for the functionality criterion lists each planned test case with the expected behaviour, the actual behaviour and the evidence that supports it. That structure is what lets a reviewer skim a report in a minute and still check any claim they doubt.
Handling flaky interfaces
Real apps are messy. Data loads late, buttons are disabled for a moment, toasts cover content, and modals appear when you least expect them. The agent handles most of this the way a careful tester would.
- Wait for the page to settle. After an action that triggers a request, the agent waits for the expected text or for a loading indicator to disappear before it reads the page again.
- Retry with a fresh snapshot. If a click lands on an element that has since been re-rendered, the agent takes a new snapshot and tries again against the current element rather than a stale reference.
- Recover from interruptions. Cookie banners, browser dialogs and unexpected modals are dismissed or answered so the test can continue. If the interruption is itself a bug, it goes in the report.
- Distinguish slow from broken. A page that eventually loads is treated differently from one that never does. The rationale says which it saw.
All of this sits inside the step budget and an overall time limit. A broken app can't keep the agent busy forever; when the budget is spent, it reports on what it observed.
Where it stops
Some flows are designed to stop automated browsers, and we don't try to defeat them. CAPTCHAs, SMS or authenticator-based two-factor prompts, and real payment forms are out of scope. If the flow you want evaluated sits behind one of these, give the agent a test account, a staging environment with the check disabled, or sandbox payment credentials, and describe them in the task instructions.
The agent also only knows what you tell it and what the app shows. Vague instructions produce reasonable but generic test cases. A short, specific brief (“a logged-in user can create a project, rename it and delete it; deleted projects should not reappear after reload”) produces a sharper, more useful evaluation.
That is the whole design: a real browser, a narrow set of actions, a clean sandbox for every job, and a record of every step. It's closer to how a person tests an app than to how a test runner does, and it leaves enough evidence behind that you never have to take its word for anything.