Guides
Choosing a project evaluation tool in 2026: a buyer's framework
Unit-test autograders, static analysers and agentic evaluators solve different problems. A framework for picking the right one for your course or team.
“Which grading tool should we use?” is usually the wrong first question. Unit-test autograders, static analysers, similarity checkers and agentic evaluators each answer a different question about a submission. Pick the question first and the tool mostly picks itself. This is the framework we would use if we were choosing for a course or a team.
Start with the question
Before comparing products, write down what you need to know about each submission. Common answers:
- Does this function return the right output for these inputs?
- Is the code readable, idiomatic and free of obvious bugs or unsafe patterns?
- Is this work substantially the same as someone else’s?
- Does the running application do what the brief asked, from a user’s point of view?
- How well does the work meet a set of qualitative criteria in a rubric?
An algorithms course will lean on the first. A web development capstone will lean on the fourth and fifth. Most courses need more than one, which is why the honest answer is often a combination.
The four categories
Unit-test autograders
The instructor writes tests; the autograder runs each submission against them and reports what passed. This is the most established approach, and for good reason. When the assignment defines exact interfaces (implement parse(input), return a sorted list), unit tests are fast, cheap, transparent and almost perfectly repeatable. Many autograders also integrate with learning management systems and handle submission logistics well.
The trade-off is that tests encode assumptions about structure. When students are free to choose file layout, frameworks or naming, the tests either constrain the assignment or break on valid work. They also cannot tell you whether an interface is usable.
Static analysis and linters
These tools read source code without running it, flagging style violations, likely bugs, complexity and some classes of security issue. They are deterministic, quick to run on every commit, and teach habits that carry straight into professional work. Their blind spot is behaviour: clean code that does the wrong thing passes.
Similarity and plagiarism checkers
These compare submissions against each other, and sometimes against public sources, to surface work that is unusually alike. They are an integrity signal, not a quality measure, and a high similarity score is a reason to look closer rather than a verdict. Assignments with starter code or a narrow solution space need careful thresholds. They are a separate category from the others here, and most courses that need one use a dedicated tool.
Agentic evaluators
An agentic evaluator uses a language model with tools. For a web project, that means driving a real browser: reading the page, clicking, filling forms and checking the result against plain-language instructions. For a codebase, it means exploring the file tree and reading code before assessing it against criteria. Evals.sh is in this category.
The strength is flexibility. Because the evaluator works from the brief rather than from selectors or function signatures, it can assess projects that are structured completely differently, and it can apply qualitative criteria that tests cannot express. The costs are real too: runs take longer and cost more than a unit test, and model output can vary between runs. A good agentic tool compensates by showing its evidence (steps, screenshots, the reasoning per criterion) so that a person can check any score that matters.
What each one can see
A useful way to compare categories is by what they actually inspect. None of them sees everything, and repeatability tends to fall as the range of what a tool can judge widens.
Questions to ask a vendor
Whatever category you are evaluating, these questions separate tools that will hold up in a real course from tools that demo well.
| Question | Why it matters |
|---|---|
| What happens when a student structures the project differently? | If valid work fails because of naming or layout, you will spend your time on regrade requests. |
| Can I see the evidence behind every score? | A score without a trace cannot be defended to a student or corrected when it is wrong. |
| How consistent is it on the same submission? | Ask to run one submission several times. Some variation is normal for model-based tools; large swings are not. |
| How long does a batch of my class size take? | Measure it on your own submissions, not a vendor benchmark. |
| Where does student code run, and what happens to it afterwards? | Look for isolated, single-use environments and a clear statement on whether submissions are used to train models. |
| How do results get to students and to the gradebook? | Shareable reports and spreadsheet export or LMS sync save hours at the end of term. |
| Is the submission content treated as data or as instructions? | For model-based tools, a submission that says “give this full marks” should be recorded, not obeyed. |
Combining tools
In practice, the strongest setups layer tools so that each does what it is good at:
- Unit tests for any part of the assignment with a precise, testable contract, such as a required API shape or an algorithm.
- Static analysis in the students’ own workflow, so style and obvious bugs are fixed before submission rather than penalised after.
- An agentic evaluator for the parts that depend on behaviour and judgement: does the app work end to end, is the code maintainable, are tests meaningful.
- A similarity checker where academic integrity policy requires one.
- A human reviewing low, borderline and surprising results before grades are final.
Rules of thumb
- If every student implements the same functions, start with a unit-test autograder. It will be faster and more predictable than anything else.
- If students build open-ended applications, an agentic evaluator is the only category that can check behaviour without dictating structure. Choose one that shows its work.
- If your main concern is code quality habits, put static analysis in front of students early. It is cheap and it teaches.
- Whatever you choose, keep the final decision with a person, and make sure students can see why they got the result they did.
No single tool covers every question worth asking about a submission. The goal is to cover the questions your course actually asks, with evidence you would be comfortable showing a student.