Guides
Grading a full cohort in one session with Groups
A practical walkthrough of bulk evaluation: preparing a CSV of submissions, writing one rubric, and reviewing results side by side.
Evaluating one project is easy to set up. Evaluating sixty the night after a deadline is where most tooling falls over. Groups is the part of Evals.sh built for that second case: one configuration, many submissions, and a single place to review the results. This guide walks through a full run.
When to use Groups
A Group is a batch of evaluations of the same type that share one set of instructions. You can create one for any of the main evaluation types:
- Project evaluation: an agent opens each live URL in a headless browser and checks it against your instructions.
- Code review: upload each submission’s project files or folder; each is copied into its own single-use workspace and reviewed.
- Frontend audit: performance, accessibility, SEO and security headers for each URL.
- Assignment grading: written submissions graded against a rubric.
If you are grading a cohort, re-checking last week’s submissions after students pushed fixes, or auditing a set of sites you maintain, a Group saves you from configuring the same thing over and over.
Prepare the submissions
For project evaluations and frontend audits, you can paste URLs into the form one at a time, but for a class it is faster to upload a CSV. (For a code review group, you upload each submission’s files or folder instead.) The importer reads the first column of each row as the URL. A header row containing the word “url” is skipped, duplicate links are ignored, and anything that is not a valid URL is reported and left out, so a messy export from your LMS will not silently produce bad tasks.
url,student
https://notes-app-amara.example.app,Amara O.
https://ben-taskboard.example.app,Ben K.
https://cohort4-team7.example.app,Team 7Only the first column is used. Keeping a second column with the student’s name costs nothing and makes it easy to match results back to your roster later.
Before you import, it is worth sending one reminder to students: check that your link loads in a private browser window. The agent evaluates the deployment it is given. If the app has gone to sleep, requires a login you did not provide, or crashes on load, that is what the report will show.
Write one set of instructions
Every task in a Group uses the same instructions, so this is where the quality of the whole batch is decided. For a project evaluation, describe the behaviour you expect in plain language, as specifically as you can.
A user can sign up with an email and password, then log in.
After logging in, they can create a note with a title and body.
Notes can be edited and deleted.
Submitting a note with an empty title shows a validation message.
Notes are still listed after a full page reload.The agent turns this into a test plan for each submission, so it will find the right buttons and fields even though every student named and styled them differently. What it needs from you is the behaviour, not the selectors.
If every project sits behind a login, you can mark the batch as protected and supply a test username and password that the agent will use. This works best when you give students a fixed test account to seed in their app.
Run the batch
- 01
Create the group
Give it a name you will recognise later, such as the course and assignment, and pick the evaluation type. - 02
Add targets and instructions
Upload the CSV or paste the list, then add the shared instructions and any test credentials. - 03
Start and walk away
Tasks run in the background, several at a time, each in its own isolated environment. You can turn on an email notification so you hear when the batch is finished rather than watching the page.
The group view updates live as tasks move from queued to running to completed. If a task fails for an operational reason, such as a site that timed out, you can restart just that task, or restart all failed tasks in the group in one action, without re-running the ones that succeeded.
Review results
When the batch finishes, you get one table with a row per submission and its score. That table can be exported as CSV, Excel or PDF, which is usually the easiest way to get numbers into a gradebook.
The table is a starting point, not the grade. Each row links to the full report for that submission: the test plan the agent derived, the step log for each feature, the screenshots it took and the reasoning behind each criterion score. A sensible review order is:
- The lowest scores first. Check whether the project is genuinely incomplete or whether something like a broken deploy got in the way.
- Anything near your pass mark. These are the cases where your judgement changes an outcome.
- A sample of the rest. Reading a few high-scoring reports is a quick way to confirm the instructions were interpreted the way you meant.
Individual reports can be shared with a link, so students see the same evidence you saw. That tends to make feedback conversations shorter and more concrete: instead of “the login didn’t work,” you can both look at the step where it failed.
A short checklist
- Ask students to verify their links load in a private window before the deadline.
- Export submissions as a CSV with the URL in the first column.
- Write instructions as observable behaviours, one per line.
- Test the instructions on one or two submissions first.
- Provide a shared test account if apps require login.
- Turn on the completion email and come back later.
- Review low and borderline scores by hand, then export.
For background on what the agent is doing inside each task, see how Evals.sh works.