evals.sh
ProductModelsSecurityPricingDocsBlog
Sign inGet started
Blog

Engineering

Code review at cohort scale

How bulk code review clones, reads and scores dozens of repositories in parallel, and how to read the results without drowning in them.

Evals.sh Engineering·October 23, 2026·8 min read

Reviewing one repository well takes an experienced engineer most of an hour. Reviewing forty of them the same way, with the same attention on the fortieth as on the first, is where review quietly turns into skimming. Bulk code review is built for that second situation.

The problem with reviewing 40 repos

At cohort scale, code review has two failure modes. The first is depth: there isn’t time to open every file, so reviewers look at the entry point, glance at the folder structure and move on. The second is drift: the standard applied on Monday morning is not quite the standard applied on Thursday night. Neither is a character flaw. They are what happens when a task that needs sustained attention is repeated dozens of times.

The goal of bulk review is not to replace your judgement. It is to make sure every repository gets the same thorough first pass, against the same criteria, so your judgement goes where it is needed: the outliers, the borderline cases, and the patterns that only show up when you look at a whole cohort together.

Setting up a bulk review

Bulk review runs inside a Group. A Group is one batch of work with a name, a task type and a shared set of instructions, which is what lets you compare results side by side afterwards.

  1. 01

    Create a Group and choose Code Review

    Name it after the assignment or sprint so it is easy to find later. You can turn on an email notification for when the batch finishes.
  2. 02

    Add one entry per submission

    Each entry becomes its own review. A submission can be an uploaded project folder, a set of files, or a pasted snippet.
  3. 03

    Write shared instructions

    One set of instructions applies to every submission, for example “focus on input validation and how errors are handled in the API layer.” This is what keeps the review consistent across the cohort.
  4. 04

    Start the review

    Every submission is queued as a separate job. Progress for each one streams into the Group page as it runs.

Keep uploads lean

Dependency and build folders are filtered out automatically, but it still helps to ask students to submit source only. Smaller uploads start faster, and the review spends its attention on code they actually wrote.

What happens to each repository

Each submission gets its own single-use workspace. The submitted code is unpacked into a directory that belongs to that job alone and filtered before the review starts. Dependency folders, build output, caches and version-control metadata (node_modules, dist, build, .git and similar) are skipped, so the review spends its time on code the student actually wrote. There is a size cap on each submission, which keeps one oversized upload from holding up the batch.

The review itself is agentic rather than a single prompt over a concatenated file dump. The reviewer starts from an overview of the repository and its file tree, then lists directories, reads files in line ranges, and searches across the codebase for patterns, following leads the way a person would. If it sees a database query in a route handler, it can search for where the input comes from before deciding whether it is safe.

Jobs run in parallel, and because each one has its own workspace, nothing from one submission can leak into another’s review. When a job finishes, successfully or not, its workspace is deleted.

SUBMISSIONSSINGLE-USE WORKSPACESRESULTSteam-01/todo-appfetch → explore → scoreteam-02/todo-appfetch → explore → scoreteam-03/todo-appfetch → explore → scoreteam-04/todo-appfetch → explore → scorereposcoreteam-0184team-0262team-0391team-0447+ n moreeach deleted when its job endsoutliers stand out
Fan-out: every submission is fetched into its own isolated workspace, reviewed independently and in parallel, and lands as one row in a shared results table. Scores are illustrative.

What you get back

Every submission is scored out of 100. By default the score combines three equally weighted criteria: security, maintainability, and test coverage. Each criterion gets its own score and a written rationale, so you can see why a repository landed where it did, not just where it landed.

CriterionWhat the reviewer looks for
SecurityUnsafe data handling, exposed secrets, injection risks, authentication issues and insecure defaults.
MaintainabilityStructure, readability, naming, modularity, and how easy the code would be to debug or extend.
CoverageMeaningful tests and validation, and whether important behaviour and edge cases are exercised.

Beneath the scores are specific findings, each tied to a file and line range with an explanation and, where it makes sense, a suggested change shown as a diff: a line to replace, a block to remove, or code to insert. These are the part students learn the most from, because they point at their own code rather than at a general principle.

The Group page shows every submission with its status and score. Any job that failed can be restarted individually or all at once, and the whole table exports to CSV, Excel or PDF for your gradebook or a teaching-team review.

Reading results without drowning

Forty reviews with a dozen findings each is nearly five hundred findings. Reading them in order is the fastest way to lose an afternoon. A better approach is to work from the outside in.

  1. Sort by score. Export to a spreadsheet and sort. The top and bottom of the list are where your attention pays off most.
  2. Read the outliers first. A very low score is either a genuinely weak submission or a problem with the submission itself: an empty repository, the wrong branch, or a project that was mostly generated boilerplate. Both are worth knowing about early.
  3. Check the borderline band. Submissions just either side of your pass mark deserve a human read. Use the rationale to see which criterion pulled the score down, then open that part of the code yourself.
  4. Compare criteria, not just totals. Two repositories at 70 can be very different: one strong on tests and weak on security, the other the reverse. The per-criterion scores tell you which feedback each student needs.

For the middle of the distribution, the written rationale and the top few findings are usually enough to write a short, specific comment without re-reviewing the whole repository.

Spotting cohort-wide patterns

The most useful thing bulk review shows you often isn’t about any single student. When the same finding appears in a large share of repositories, it is rarely forty individual mistakes. It is usually one gap in how the material was taught, or one misleading example in the starter code.

Suppose, as a hypothetical, that most of a cohort’s repositories are flagged for building SQL queries with string concatenation, or for committing an .env file with real-looking keys. That is a lecture topic, not forty separate comments. Addressing it once with the whole group, then re-running the review after students fix it, is a better use of everyone’s time than marking it down repeatedly.

A short checklist for each batch:

  • Which finding appears most often across the cohort, and is it something the course taught?
  • Is coverage low across the board? If so, the assignment may not have made testing an explicit requirement.
  • Do the lowest-scoring repositories share a cause, such as a missing build step or an unexpected project layout?
  • Did any submissions fail outright? Check the error before assuming the student did something wrong.

Used this way, bulk review gives you a consistent baseline for every submission and a view across the whole cohort that is hard to get any other way. The scores tell you where to look. The decisions are still yours.

On this page

The problem with reviewing 40 reposSetting up a bulk reviewWhat happens to each repositoryWhat you get backReading results without drowningSpotting cohort-wide patterns

Try it on your own project

Free credits every month. No card required.

Get started

Keep reading

Engineering

Inside our computer-use agent: evaluating apps through a real browser

October 23, 2026

Engineering

Agentic API testing: beyond the 200 OK

October 23, 2026

Engineering

Smoke-testing every release with a browser agent

October 23, 2026

evals.sh

Autonomous evaluation for software teams and educators. Agents that review code, drive real browsers and grade work against your rubric.

Product

  • Project evaluation
  • Code review
  • Frontend audit
  • API testing
  • Assignment grading

Platform

  • Models
  • Security
  • Pricing
  • Documentation

Resources

  • Blog
  • Guides
  • Contact

Legal

  • Privacy policy
  • Terms of service
  • Data handling

© 2026 Evals.sh. All rights reserved.

Customer data is never used to train AI models.