evals.sh
ProductModelsSecurityPricingDocsBlog
Sign inGet started
Blog

Guides

Writing rubrics an AI grader can apply consistently

Observable criteria, explicit weights and clear failure states: the habits that make automated grading repeatable and fair.

Evals.sh Engineering·October 23, 2026·9 min read

An AI grader is only as consistent as the rubric you hand it. Give it a vague one and you get thoughtful, well-written scores that drift from one submission to the next. Give it a precise one and the 40th submission is graded the same way as the first. This guide covers the habits that make the difference.

Why wording matters more than you expect

A human teaching assistant who grades 60 projects builds up a private interpretation of the rubric as they go. By the tenth submission they know what “clean code” means to them, and they apply it more or less evenly. A model doesn’t accumulate that context. It reads your rubric fresh for every submission, so every ambiguity in the text is an opportunity for the interpretation to shift.

That sounds like a weakness, but it is also useful. Because the grader has nothing to go on except the words you wrote, fixing the words fixes the grading. You don’t need to retrain anyone or hold a calibration meeting. You need criteria that a careful reader with no prior context would apply the same way twice.

In Evals.sh the rubric goes in two places, depending on what you are grading. For written work, you paste the assignment text (or upload a PDF, DOCX or TXT file) and put the rubric in the instructions field. For deployed apps, you give the browser agent a target URL and an agent task description that describes what to test. Both fields accept up to 5,000 characters, which is plenty for a focused rubric and a useful forcing function against a sprawling one.

Observable over subjective

The single most effective change you can make is to rewrite every criterion so it describes something the grader can observe: a page that loads, a value that changes, a section that exists, an error that appears. Subjective criteria (“good UX”, “well structured”, “shows understanding”) are not wrong, but they ask the grader to supply its own standard, and that standard is where variance comes from.

A useful test: could two people who have never met agree on whether a submission meets this criterion, just by looking at it? If not, the criterion needs a concrete check underneath it.

Vague criterionObservable rewrite
Good user experienceEvery form shows an inline error message when a required field is left empty, and the submit button is disabled while a request is in flight.
Cart works properlyAdding two items updates the cart badge to 2, and the cart total equals the sum of the item prices.
Clean, well-organised codeData access lives in its own module; no component file exceeds roughly 300 lines; no API keys are committed.
Strong analysisThe essay states a thesis in the first paragraph and supports it with at least two cited sources.
Handles errorsSubmitting the login form with a wrong password shows a message and does not navigate away.

Notice that the rewrites are longer. That is fine. A rubric is not a slogan, and the extra words are doing real work. You can still keep a short label for each criterion; just put the observable check directly under it.

Weights that sum to 100

Weights tell the grader, and your students, what matters most. Make them add up to 100 so the overall score reads as a percentage and nobody has to do arithmetic to understand it. When the evaluation agent scores a deployed app, each criterion is scored from 0 to 100 with a written rationale, and the overall score is the weighted combination. If your weights are sloppy, the total is too.

Give the largest weight to the thing the assignment is actually about. If the brief is “build a working checkout”, the checkout flow should dominate, and README formatting should be a small line item, not a third of the grade. A quick sanity check: if a submission nails the core task but misses every minor criterion, does it still pass? If not, rebalance.

RUBRIC · 100 POINTSweights sum to 100passpartialfail40Checkout flow completesConfirmation page shows an order number4020025Cart total updatesTotal changes when quantity changes2512020Form validationEmpty email shows an inline error2010015README run stepsFresh clone runs with documented commands1570
Anatomy of a weighted rubric. Each criterion has a weight, a concrete check, and a point value for every state, so a partial result is never left to interpretation.

Explicit pass, partial and fail

Most grading disagreements are not about the clear passes or the clear failures. They are about the middle: the checkout that works except for one payment method, the form that validates email but not phone number. If your rubric doesn’t say what partial credit looks like, the grader will decide, and it may decide differently next time.

For each criterion, write one line per state. Keep it mechanical:

  • Pass (25): the total updates immediately when quantity changes, for every item.
  • Partial (12): the total updates, but only after a page refresh, or not for every item.
  • Fail (0): the total never changes, or the cart cannot be opened.

The assignment grader reports each criterion on a five-level scale (Excellent, Good, Satisfactory, Needs Improvement, Poor) alongside remarks, strengths, weaknesses and suggested improvements. If you prefer three states, say so explicitly and describe what each one means; if you are happy with five, describe what separates Good from Satisfactory for each criterion. Either works. Leaving it unstated is the only option that doesn’t.

A rule of thumb

If you find yourself writing “use your judgement” in a rubric, that is the criterion most likely to produce inconsistent scores. Replace it with the two or three things your own judgement would actually look for.

Writing instructions for the browser agent

When you evaluate a deployed app, the rubric doubles as a test plan. The agent opens the URL in an isolated browser, follows your instructions, and records what it sees. It will do what you ask, in the order you ask, so write the task the way you would brief a careful new tester: step by step, with the expected result after each step.

Agent task description (example)
Log in with [email protected] / the password provided.
Add two different items to the cart from the products page.
Verify the cart badge shows 2 and the total equals the sum of both prices.
Change the quantity of the first item to 3. Verify the total updates
without a page reload.
Proceed to checkout and submit the form with the email field empty.
Verify an inline error appears and the order is not placed.
Fill in the form correctly and place the order.
Verify a confirmation page shows an order number.

A few things make instructions like these work well:

  • One action per line. It keeps the agent’s evidence easy to map back to your rubric.
  • A verification after each meaningful action. “Verify the total updates” is checkable; “check the cart” is not.
  • Names the student will recognise. Refer to buttons and pages by their visible labels, not by your mental model of the app.
  • No hidden steps. If a student must seed data or dismiss a cookie banner first, say so.

Test credentials and seeded data

If the app is behind a login, tick “App Requires Authentication?” and enter a test username and password. Ask students to create a dedicated test account for grading and to include it in their submission, rather than sharing a personal one. For cohort runs, a common pattern is to require the same test account on every project (for example [email protected] with a password you publish in the assignment brief), which keeps the instructions identical across submissions.

Seeded data matters just as much. A cart test fails for the wrong reason if the product list is empty. Put “the app must ship with at least three products” in the assignment requirements, not only in the rubric, so students know it is part of the deliverable.

Pilot before you run a cohort

No rubric survives first contact with real submissions unchanged. Before you grade a whole class, run it on a handful and read the results closely.

  1. 01

    Pick three to five varied submissions

    Include one you would score highly, one you would fail, and a few from the messy middle. The middle is where rubrics break.
  2. 02

    Grade them yourself first

    Write down your scores before you look at the grader’s. Otherwise it is too easy to rationalise whatever comes back.
  3. 03

    Run the rubric and compare per criterion

    Look at each criterion’s score and rationale, not just the total. A matching total can hide two criteria that disagree in opposite directions.
  4. 04

    Rewrite the criteria that disagreed

    Every disagreement is either a vague criterion or a missing partial state. Tighten the wording and run the same submissions again.
  5. 05

    Freeze the rubric, then run the cohort

    Once the pilot scores match your own, stop editing. Changing the rubric halfway through a cohort makes early and late scores incomparable.

The pilot usually takes less than an hour, and it is the best hour you will spend on the assignment. A rubric that has been tested on real work is one you can defend to a student who asks why they lost points, and that is ultimately what consistency is for.

On this page

Why wording mattersObservable over subjectiveWeights that sum to 100Explicit pass, partial and failInstructions for the browser agentTest credentials and seeded dataPilot before you run a cohort

Try it on your own project

Free credits every month. No card required.

Get started

Keep reading

Guides

What a frontend audit actually measures

October 23, 2026

Guides

Grading a full cohort in one session with Groups

April 2, 2026

Guides

Choosing a project evaluation tool in 2026: a buyer's framework

April 2, 2026

evals.sh

Autonomous evaluation for software teams and educators. Agents that review code, drive real browsers and grade work against your rubric.

Product

  • Project evaluation
  • Code review
  • Frontend audit
  • API testing
  • Assignment grading

Platform

  • Models
  • Security
  • Pricing
  • Documentation

Resources

  • Blog
  • Guides
  • Contact

Legal

  • Privacy policy
  • Terms of service
  • Data handling

© 2026 Evals.sh. All rights reserved.

Customer data is never used to train AI models.