Skip to main content
A Grader is an LLM-based judge that reads a completed conversation and evaluates it against the expected behaviors in your tests. After each simulation ends, the grader claims the conversation, reads it turn by turn, and writes one verdict per expected behavior — each verdict carrying a rationale, the turns it cites, and the grader that produced it. The same grading engine runs on both simulation transcripts and production telemetry.

​
The Grader Library

Egma ships a library of built-in graders, accessible at /graders in your dashboard. Each library entry defines a type of judgment — behavioral evaluation, latency measurement, tool-call checking, and so on — along with the form of parameters it accepts. You activate a grader for your project by pressing Use on a library entry. This creates a project grader: a copy of that library entry with your chosen parameters, scope, and settings. Your project can have multiple copies of the same library entry, each configured differently.
A grader is always a copy of a library entry. There is no blank-form authoring surface. This keeps every grader anchored to a well-defined judgment type with documented behavior.

​
Project graders

Once you press Use, the grader is active for your project. You can list your running graders at GET /api/graders. Each project grader carries these configurable fields:

​
Required vs. diagnostic graders

The required flag is the most important grader setting. It controls whether a grader’s verdict can fail a test.

Required (blocking)

A required: true grader can fail a test. If this grader writes a failed verdict for any expected behavior, the simulation’s overall verdict is failed. Use this for the behaviors your agent must always get right.

Diagnostic (non-blocking)

A required: false grader reports its findings but never fails a test. Its verdicts appear in a separate diagnostic lane in the results. Use this to observe a behavior you are not yet ready to enforce.
Switching a grader from required to diagnostic (or back) takes effect immediately across all historical runs. The verdicts themselves do not change — only what they add up to changes, through the fold Egma computes at read time. A run that failed on a grader alone reads as passed the moment you set it to diagnostic.

​
Grader versioning

Graders are versioned, but what triggers a new version depends on which field changes: This distinction matters for auditability. The params fields are what a verdict was decided by — changing them should not retroactively alter the meaning of past results, so a new version is the right answer. The other fields say where and how loudly a grader applies, and those are live settings rather than historical facts. Every verdict row references the specific grader version_id that produced it, so you can always trace back to the exact configuration that judged a given conversation.

​
Configuring graders

​
Via the dashboard

Navigate to /graders, find the library entry you want, and click Use. Fill in the params form, set the scope, and choose whether the grader is required or diagnostic.

​
Via the API

Activate a grader with POST /api/graders:
Edit an existing grader with PATCH /api/graders/:graderId. Only the fields you include are updated:
Send params as a properly typed JSON object, not as a string. Send required as true or false, not as "true" or "false". Egma refuses wrong types by name rather than silently coercing them — a mis-typed field would otherwise leave your grader configured differently from what you intended, with nothing in the response to say so.

​
Switching a grader off

Deleting a grader is the only way to turn it off. There is no enable/disable flag. From the moment a grader is deleted, it judges nothing in its scope. Use DELETE /api/graders/:graderId. Deletion is a soft delete. Every verdict the grader already wrote remains readable and still references the version that produced it. An old run keeps its full meaning — you are not losing evidence, only stopping the grader from writing new verdicts going forward.

​
Mock tools and grading

The egma/mock-tools.md file in your repository controls what Egma returns for your agent’s tool calls during simulation. This is not a grader setting, but it is central to what graders are judging: the agent’s behavior in a controlled world where tool responses are predictable. Individual tests can override the project-level mock tools by including a ## Mock tools section in their own file. That override is versioned alongside the test — changing it mints a new test version — while the project’s mock tools are unversioned and a push always replaces what was there.
Use mock tools to test failure branches. Set a tool to return an error, and add an expected behavior that asserts your agent handles the failure gracefully. Then graders can evaluate whether the agent did the right thing in a situation that is hard to reproduce with a real backend.