Engineering
Smoke-testing every release with a browser agent
Scripted end-to-end tests break on every redesign. How engineering teams use plain-language instructions and the Evals.sh API to check critical flows after each deploy.
Most teams agree that someone should click through sign-up and checkout after every deploy. Few teams manage to keep that habit going. Here’s how to hand the job to an agent that reads instructions in plain language, so the check survives your next redesign.
Why scripted smoke tests rot
A traditional end-to-end smoke test is a script: find this selector, type into that input, wait for this element, assert on that text. It works on the day it’s written. Then the design team renames a button, a component library update changes the DOM structure, or a modal appears on first visit. None of those changes break the product, but each one breaks the script.
After a few false alarms people stop trusting the suite. Someone marks the failing job as allowed to fail, and the safety net is gone without anyone deciding to remove it.
A browser agent works from a different contract. Instead of selectors, you describe the journey and what success looks like. The agent reads the page through its accessibility tree, finds the control a person would use, and carries on. A renamed button with the same meaning doesn’t stop it. A broken checkout does.
Choosing what to check
A smoke test isn’t a regression suite. Its job is to answer one question quickly: is the release safe to put in front of customers? Pick the three to five journeys that would cause an incident if they broke, and write one evaluation for each. Separate evaluations run in parallel and fail independently, so a red result tells you exactly which journey is affected.
| Journey | What the agent does | What counts as a pass |
|---|---|---|
| Sign-up | Creates an account with a unique test email | Lands on onboarding with no error message |
| Sign-in | Signs in with the seeded test account | Dashboard loads and shows the account’s name |
| Core action | Creates the main object your product is about (a project, an invoice, a post) | The new item appears in the list after a reload |
| Checkout | Adds an item to the cart and reaches the payment step | Order summary shows the right item and total |
| Settings | Opens billing or account settings | Page renders with current plan details |
Writing the instructions
Good instructions read like a note to a careful new colleague. Say what to do, in order, and make the pass condition something that’s visible on screen. Leave out anything you don’t care about, because the agent will treat every sentence as part of the job.
Sign in with the provided test account.
Open any product from the catalogue and add it to the cart.
Open the cart and continue to checkout.
Pass if:
- the cart badge shows 1 after adding the item
- the checkout page shows the same product name and price
- no error banner or blank page appears at any point
Do not submit payment.That last line matters. The agent follows instructions literally and stays inside the URL you give it, but it’s still a real browser on a real site. Say explicitly where it should stop.
Test accounts and environments
Run smoke tests against staging or a preview deployment where you can, and against production only with accounts that exist for testing. For any journey behind a login, set isProtected and pass a dedicated test user in credentials. Never use an employee’s real account.
- Seed the data the journey needs. If sign-in should show a dashboard with projects, make sure the test account has projects.
- Make sign-up repeatable. Use plus-addressing or a timestamp in the email so each run creates a fresh user, and clean them up on a schedule.
- Keep payments in test mode. Use your payment provider’s sandbox on staging, and stop before payment on production.
- Turn off friction for test users. CAPTCHAs and SMS codes stop any automated visitor. Allow-list the test accounts or disable those checks on staging.
Wiring it into a deploy
- 01
Create an API key
Generate a key from the API Keys page and store it as a secret in your CI provider. It’s sent as a Bearer token. - 02
Start one evaluation per journey
POST each journey to /api/project-evaluation with the deployed URL and its instructions. The response is a 202 with an evaluationId. - 03
Wait for the results
Poll GET /api/project-evaluation/:id until the status is completed or failed, or subscribe to the /stream endpoint to follow progress. - 04
Gate on the score
Read evalScore and the per-feature pass/fail list. Block promotion if any journey falls below your threshold.
#!/usr/bin/env bash
set -euo pipefail
API="https://evals.sh/api"
AUTH="Authorization: Bearer $EVALS_API_KEY"
ID=$(curl -sf -X POST "$API/project-evaluation" -H "$AUTH" \
-H "Content-Type: application/json" \
-d @- <<JSON | jq -r .evaluationId
{
"url": "$STAGING_URL",
"evaluationPrompt": $(jq -Rs . < journeys/checkout.txt),
"isProtected": true,
"credentials": { "username": "$TEST_USER", "password": "$TEST_PASS" }
}
JSON
)
while :; do
RESULT=$(curl -sf "$API/project-evaluation/$ID" -H "$AUTH")
STATUS=$(jq -r .status <<<"$RESULT")
[[ "$STATUS" == "completed" || "$STATUS" == "failed" ]] && break
sleep 15
done
SCORE=$(jq -r '.evalScore // 0' <<<"$RESULT")
echo "checkout: $STATUS, score $SCORE"
(( SCORE >= 80 )) || { echo "Smoke test below threshold"; exit 1; }Reading a failure
When a journey fails, the report gives you more than a red cross. Each feature in the test plan is marked passed or failed, with the step log of what the agent did and screenshots taken along the way. The score reasoning explains in plain language why the result came out where it did.
Work through it in this order:
- Read the reasoning. It usually names the failing step: “the cart total did not update after adding the item.”
- Look at the screenshot for that step. An error banner, an empty state or a spinner that never resolved will usually be visible.
- Check the step log for what the agent tried. If it retried a control several times, that’s often a click target that doesn’t respond, which is a real bug.
- Rerun once before escalating. If the second run passes cleanly, look for timing issues such as slow third-party scripts or cold caches before blaming the release.
Where it fits, and where it doesn’t
Agent smoke tests sit on top of your other tests, not instead of them. Unit and integration tests are still the fastest, cheapest way to check logic, and they should run on every commit. A browser agent takes minutes rather than seconds and costs more per run, so use it where it’s strongest: once per deploy, on the handful of journeys that matter most, against a real environment.
It also isn’t perfectly repeatable. Two runs of the same journey can take slightly different paths through the UI. Clear pass conditions shrink that variance a lot, which is why the instructions above spell out exactly what counts as a pass. In exchange you get a check that keeps working through a redesign, and a report anyone on the team can read.