Education
The evaluation bottleneck: when students ship faster than you can grade
AI coding assistants have multiplied how much students build. Grading capacity hasn't kept up. Here's what changes when evaluation is automated.
AI coding assistants changed how much a student can build in a week. They did not change how long it takes a person to open that project, run it, click through it and write useful feedback. That gap between production and review is the evaluation bottleneck, and it is now one of the main constraints on how technical courses are taught.
The asymmetry
A few years ago, a capstone web app was a multi-week effort, and a large part of what instructors assessed was whether the student could get it working at all. Today a student with an assistant can scaffold authentication, a database layer and a responsive interface in an afternoon. The interesting questions have moved: did they understand what was generated, did they handle the edge cases, does the thing actually behave the way the brief asked?
Answering those questions still takes an instructor’s time, and the time per project has not gone down. If anything it has gone up, because there is more surface area to check. Every additional feature a student can ship is another feature someone has to verify.
So the volume of work rises, the review capacity stays flat, and something has to give.
What the bottleneck costs
When grading capacity is the constraint, courses adapt around it, usually in ways nobody would choose on purpose:
- Feedback arrives late. By the time a student hears that their form validation was broken, they have moved on to the next assignment and the lesson does not land.
- Review gets shallow. With a stack of projects and a deadline, it is tempting to check that the homepage loads and move on.
- Assignments get smaller. Instructors sometimes simplify a brief so that grading it stays manageable, which caps what students are asked to build.
- Office hours fill with debugging. Time that could go to design discussion is spent working out why something does not run.
Verification vs. judgement
It helps to separate two kinds of work that usually get bundled together as “grading.”
Verification is checking whether the project does what the brief says. Can a user register? Does the cart total update? Is there an error message when the API fails? These questions have answers you can observe, and they take a lot of clicking.
Judgement is everything that needs context about the student and the course: whether an architectural choice was reasonable, whether a rough project shows more growth than a polished one, what the student should work on next.
Verification is where most of the hours go, and it is also the part that can be automated without losing anything that matters. Judgement is the part students actually need from a teacher.
Consistency and fatigue
Anyone who has graded a stack of projects in one sitting knows that the first one gets a more careful look than the fortieth. That is not a character flaw; it is what happens to attention over a long evening. But it means two students with similar work can get different levels of scrutiny depending on where they landed in the pile.
An automated first pass applies the same instructions to every submission. That does not make it infallible, and AI models can vary between runs. What it does provide is a uniform starting point, and, more usefully, a written record for each submission of what was checked and what happened. When a score looks wrong, you can see why it was given and correct it.
What changes in practice
With Evals.sh, the verification pass looks like this. You write the brief as plain-language instructions once: “A user can sign up, create a note, edit it, and the note is still there after a reload.” For each submitted URL, an agent opens the app in an isolated headless browser, derives test cases from your instructions, works through them, and records each step with screenshots. It then scores each criterion from 0 to 100 with reasoning tied to what it observed.
For a whole class, you put the links into a Group, either pasted as a list or uploaded as a CSV, and the evaluations run in the background. Uploaded project files and folders can go through code review the same way, and written work through rubric-based assignment grading.
What you sit down to is different from a folder of links:
- a table of scores for the whole class, exportable to CSV, Excel or PDF;
- for each student, a report with the test plan, step log, screenshots and reasoning;
- a shareable link per report, so students see the same evidence you do.
Your time then goes to the reports that need it: low scores, surprising scores, and the borderline cases where a human decision matters. For the rest, you add the comment only you can write.
Caveats worth stating
Automating verification is not the same as automating teaching, and it is worth being clear about what it does not do.
- It evaluates what it can reach. A deployment that is down or crashing will be scored in that state, so it pays to tell students to check their links before the deadline.
- It follows your instructions. If the brief leaves a requirement out, the agent may still test something reasonable, but it cannot read your mind about what you meant.
- It is a first pass. The final grade, and the conversation about it, should stay with the instructor.
Students producing more is a good problem to have. The fix is not to ask them to build less, or to grade them less carefully. It is to stop spending instructor hours on the parts of review that a machine can do and document, and spend them on the parts that need a teacher.