evals.sh
ProductModelsSecurityPricingDocsBlog
Sign inGet started
Blog

Engineering

Agentic API testing: beyond the 200 OK

Status codes tell you an endpoint answered. An agent can tell you whether it answered correctly, consistently and safely.

Evals.sh Engineering·October 23, 2026·8 min read

Most API monitoring answers one question: did the endpoint respond? A 200 OK is reassuring, but it says nothing about whether the body has the right shape, whether bad input is rejected, or whether an endpoint that should require a token actually does. Those are the questions our API tester is built to answer.

What a status code tells you

A status code is the API's own opinion of how the request went. It is useful, and it is also easy to get wrong. Endpoints return 200 with an error message in the body. They return 500 for what should be a validation error. They return 201 for a record that should never have been created.

A conventional test suite catches these if someone wrote a test for each case. In practice, most suites cover the happy path well and the edges thinly, because writing edge-case tests is tedious. An agent is good at tedious, systematic exploration, and it can explain what it found in plain language.

How a test run works

You describe the API; the agent designs and runs the tests.

  1. 01

    Describe the endpoints

    Give a base URL and a list of endpoints, each with a method and path. For POST, PUT and PATCH you can include a JSON request schema; any endpoint can take query parameters. A single run covers up to 100 endpoints.
  2. 02

    Configure headers

    Global headers, such as an Authorization bearer token or an API key, are applied to the run. Use credentials for a test account or staging environment.
  3. 03

    The agent generates test cases

    For each endpoint it writes a set of named test cases with synthetic payloads: valid requests, invalid ones, missing fields, boundary values, and variations on authentication.
  4. 04

    Requests run against your API

    Each case is sent for real, and the request headers, body, response status and response body are recorded. Progress streams to the results page while the run continues in the background.
  5. 05

    Results are assessed and scored

    Every case is marked passed or failed against what a correct API should do. The run gets an overall score out of 100 with written reasoning.

What the agent checks

The point of using an agent is that it judges responses, not just collects them. These are the kinds of questions it asks of each endpoint.

ENDPOINTshapeerrorsauthedge casesconsistencystatusGET /users/{id}passedPOST /userspartialPATCH /users/{id}partialDELETE /users/{id}partialall cases passedsome cases failedfailedillustrative
An illustrative run against a hypothetical users API. Each endpoint is probed along several dimensions; a single failing case is enough to mark an endpoint as partially failed.

Response shape

Does the body match what the endpoint is supposed to return? If the request schema implies a user object with an id and an email, a response that wraps it in an unexpected envelope, drops a field or returns a string where a number belongs is worth flagging, even with a 200.

Error handling

When a request is wrong, does the API say so clearly? Good APIs return a 4xx status with a useful message for client mistakes and reserve 5xx for real server faults. The agent looks for errors reported as successes, stack traces or internal details leaking into error bodies, and generic responses that would leave a client guessing.

Auth behaviour

Endpoints that should be protected are tried without credentials and with credentials that are malformed. An endpoint that returns data to an anonymous request, or answers a bad token with 500 instead of 401, is a finding.

Edge-case inputs

Empty strings, missing required fields, wrong types, very long values, unexpected characters and out-of-range numbers. This is where most bugs live, because it is where most hand-written test suites stop.

Consistency

Do similar endpoints behave alike? If one endpoint returns errors as { error: string } and another as { message: string }, or field names switch between camelCase and snake_case, every client has to handle both. The agent notes these patterns across the run.

An example finding

Here is a hypothetical case of the kind the tester produces. The endpoint answers with a success status, so a simple uptime check would pass it. The agent does not.

Illustrative test case · POST /users
POST /users
Content-Type: application/json
Authorization: Bearer <test token>

{ "email": "not-an-email", "name": "" }

← 201 Created
{ "id": "u_1042", "email": "not-an-email", "name": "" }

FAILED · edge-case input
Expected a 4xx validation error for a malformed email and an
empty name. The endpoint accepted the payload and returned a
created record, so invalid data can reach storage. The valid-
payload case for this endpoint passed.

The value is in the last paragraph. It states what was expected, what happened, why it matters, and how this case relates to the others for the same endpoint. That is the difference between a red cell in a dashboard and something a developer can act on.

Reading the results

Results are grouped by endpoint. Each endpoint is marked passed, partially failed or failed, and expands into its individual test cases. Every case shows the exact request that was sent and the response that came back, so you can reproduce it with any HTTP client.

At the top of the report is an overall score out of 100 with the assessor's reasoning, based on reliability, response validation and security behaviour across the run. From there you can share the report with a link, download it as a PDF, or re-run the same test after a fix to confirm it is resolved.

Scores are a summary, not the result

The score is useful for tracking progress over time and comparing runs. The test cases are where the information is. Read the failing cases first, even when the score looks healthy.

Getting useful results

  • Point it at a test environment. The agent sends real requests, including writes and deletes. Use staging or a sandbox, not production data.
  • Include request schemas. A schema for each write endpoint lets the agent build realistic valid payloads and targeted invalid ones.
  • Use a dedicated test credential. Supply it as a global header so auth checks have a known-good baseline to compare against.
  • Test related endpoints together. Consistency findings only appear when the agent can see several endpoints side by side.

A 200 OK means an endpoint is reachable. Whether it's correct is a separate question, and that one needs someone, or something, to actually read the response.

On this page

What a status code tells youHow a test run worksWhat the agent checksAn example findingReading the resultsGetting useful results

Try it on your own project

Free credits every month. No card required.

Get started

Keep reading

Engineering

Inside our computer-use agent: evaluating apps through a real browser

October 23, 2026

Engineering

Code review at cohort scale

October 23, 2026

Engineering

Smoke-testing every release with a browser agent

October 23, 2026

evals.sh

Autonomous evaluation for software teams and educators. Agents that review code, drive real browsers and grade work against your rubric.

Product

  • Project evaluation
  • Code review
  • Frontend audit
  • API testing
  • Assignment grading

Platform

  • Models
  • Security
  • Pricing
  • Documentation

Resources

  • Blog
  • Guides
  • Contact

Legal

  • Privacy policy
  • Terms of service
  • Data handling

© 2026 Evals.sh. All rights reserved.

Customer data is never used to train AI models.