Engineering
Code review at cohort scale
How bulk code review clones, reads and scores dozens of repositories in parallel, and how to read the results without drowning in them.
Reviewing one repository well takes an experienced engineer most of an hour. Reviewing forty of them the same way, with the same attention on the fortieth as on the first, is where review quietly turns into skimming. Bulk code review is built for that second situation.
The problem with reviewing 40 repos
At cohort scale, code review has two failure modes. The first is depth: there isn’t time to open every file, so reviewers look at the entry point, glance at the folder structure and move on. The second is drift: the standard applied on Monday morning is not quite the standard applied on Thursday night. Neither is a character flaw. They are what happens when a task that needs sustained attention is repeated dozens of times.
The goal of bulk review is not to replace your judgement. It is to make sure every repository gets the same thorough first pass, against the same criteria, so your judgement goes where it is needed: the outliers, the borderline cases, and the patterns that only show up when you look at a whole cohort together.
Setting up a bulk review
Bulk review runs inside a Group. A Group is one batch of work with a name, a task type and a shared set of instructions, which is what lets you compare results side by side afterwards.
- 01
Create a Group and choose Code Review
Name it after the assignment or sprint so it is easy to find later. You can turn on an email notification for when the batch finishes. - 02
Add one entry per submission
Each entry becomes its own review. A submission can be an uploaded project folder, a set of files, or a pasted snippet. - 03
Write shared instructions
One set of instructions applies to every submission, for example “focus on input validation and how errors are handled in the API layer.” This is what keeps the review consistent across the cohort. - 04
Start the review
Every submission is queued as a separate job. Progress for each one streams into the Group page as it runs.
What happens to each repository
Each submission gets its own single-use workspace. The submitted code is unpacked into a directory that belongs to that job alone and filtered before the review starts. Dependency folders, build output, caches and version-control metadata (node_modules, dist, build, .git and similar) are skipped, so the review spends its time on code the student actually wrote. There is a size cap on each submission, which keeps one oversized upload from holding up the batch.
The review itself is agentic rather than a single prompt over a concatenated file dump. The reviewer starts from an overview of the repository and its file tree, then lists directories, reads files in line ranges, and searches across the codebase for patterns, following leads the way a person would. If it sees a database query in a route handler, it can search for where the input comes from before deciding whether it is safe.
Jobs run in parallel, and because each one has its own workspace, nothing from one submission can leak into another’s review. When a job finishes, successfully or not, its workspace is deleted.
What you get back
Every submission is scored out of 100. By default the score combines three equally weighted criteria: security, maintainability, and test coverage. Each criterion gets its own score and a written rationale, so you can see why a repository landed where it did, not just where it landed.
| Criterion | What the reviewer looks for |
|---|---|
| Security | Unsafe data handling, exposed secrets, injection risks, authentication issues and insecure defaults. |
| Maintainability | Structure, readability, naming, modularity, and how easy the code would be to debug or extend. |
| Coverage | Meaningful tests and validation, and whether important behaviour and edge cases are exercised. |
Beneath the scores are specific findings, each tied to a file and line range with an explanation and, where it makes sense, a suggested change shown as a diff: a line to replace, a block to remove, or code to insert. These are the part students learn the most from, because they point at their own code rather than at a general principle.
The Group page shows every submission with its status and score. Any job that failed can be restarted individually or all at once, and the whole table exports to CSV, Excel or PDF for your gradebook or a teaching-team review.
Reading results without drowning
Forty reviews with a dozen findings each is nearly five hundred findings. Reading them in order is the fastest way to lose an afternoon. A better approach is to work from the outside in.
- Sort by score. Export to a spreadsheet and sort. The top and bottom of the list are where your attention pays off most.
- Read the outliers first. A very low score is either a genuinely weak submission or a problem with the submission itself: an empty repository, the wrong branch, or a project that was mostly generated boilerplate. Both are worth knowing about early.
- Check the borderline band. Submissions just either side of your pass mark deserve a human read. Use the rationale to see which criterion pulled the score down, then open that part of the code yourself.
- Compare criteria, not just totals. Two repositories at 70 can be very different: one strong on tests and weak on security, the other the reverse. The per-criterion scores tell you which feedback each student needs.
For the middle of the distribution, the written rationale and the top few findings are usually enough to write a short, specific comment without re-reviewing the whole repository.
Spotting cohort-wide patterns
The most useful thing bulk review shows you often isn’t about any single student. When the same finding appears in a large share of repositories, it is rarely forty individual mistakes. It is usually one gap in how the material was taught, or one misleading example in the starter code.
Suppose, as a hypothetical, that most of a cohort’s repositories are flagged for building SQL queries with string concatenation, or for committing an .env file with real-looking keys. That is a lecture topic, not forty separate comments. Addressing it once with the whole group, then re-running the review after students fix it, is a better use of everyone’s time than marking it down repeatedly.
A short checklist for each batch:
- Which finding appears most often across the cohort, and is it something the course taught?
- Is coverage low across the board? If so, the assignment may not have made testing an explicit requirement.
- Do the lowest-scoring repositories share a cause, such as a missing build step or an unexpected project layout?
- Did any submissions fail outright? Check the error before assuming the student did something wrong.
Used this way, bulk review gives you a consistent baseline for every submission and a view across the whole cohort that is hard to get any other way. The scores tell you where to look. The decisions are still yours.