> ## Documentation Index
> Fetch the complete documentation index at: https://docs.egma.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Graders: LLM-Based Judges for Voice Agent Behavior

> Learn how Egma's LLM-based graders evaluate conversations, how to configure params and scope, and the difference between required and diagnostic graders.

A **Grader** is an LLM-based judge that reads a completed conversation and evaluates it against the expected behaviors in your tests. After each simulation ends, the grader claims the conversation, reads it turn by turn, and writes one verdict per expected behavior — each verdict carrying a rationale, the turns it cites, and the grader that produced it. The same grading engine runs on both simulation transcripts and production telemetry.

## The Grader Library

Egma ships a library of built-in graders, accessible at **/graders** in your dashboard. Each library entry defines a type of judgment — behavioral evaluation, latency measurement, tool-call checking, and so on — along with the form of parameters it accepts.

You activate a grader for your project by pressing **Use** on a library entry. This creates a **project grader**: a copy of that library entry with your chosen parameters, scope, and settings. Your project can have multiple copies of the same library entry, each configured differently.

<Note>
  A grader is always a copy of a library entry. There is no blank-form authoring surface. This keeps every grader anchored to a well-defined judgment type with documented behavior.
</Note>

## Project graders

Once you press **Use**, the grader is active for your project. You can list your running graders at `GET /api/graders`.

Each project grader carries these configurable fields:

| Field                    | Description                                                                                                                                                                            |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | What your project calls this copy of the grader. Defaults to the library entry's name.                                                                                                 |
| `params`                 | The filled-in form values for this grader — for example, `{ "metric": "turn_response_latency", "bound": 2000 }`. Read the library entry to see what it asks for; some ask for nothing. |
| `scope`                  | Where this grader judges: `simulations`, `production`, or `both`.                                                                                                                      |
| `required`               | Whether this grader is blocking (`true`) or diagnostic (`false`).                                                                                                                      |
| `production_sample_rate` | What share of live production traffic this grader judges, as a whole percentage 0–100.                                                                                                 |

## Required vs. diagnostic graders

The `required` flag is the most important grader setting. It controls whether a grader's verdict can fail a test.

<CardGroup cols={2}>
  <Card title="Required (blocking)" icon="lock">
    A `required: true` grader can fail a test. If this grader writes a `failed` verdict for any expected behavior, the simulation's overall verdict is `failed`. Use this for the behaviors your agent must always get right.
  </Card>

  <Card title="Diagnostic (non-blocking)" icon="magnifying-glass">
    A `required: false` grader reports its findings but never fails a test. Its verdicts appear in a separate diagnostic lane in the results. Use this to observe a behavior you are not yet ready to enforce.
  </Card>
</CardGroup>

<Tip>
  Switching a grader from required to diagnostic (or back) takes effect immediately across all historical runs. The verdicts themselves do not change — only what they add up to changes, through the fold Egma computes at read time. A run that failed on a grader alone reads as passed the moment you set it to diagnostic.
</Tip>

## Grader versioning

Graders are versioned, but what triggers a new version depends on which field changes:

| Change                                                                          | Effect                                                              |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `params` updated                                                                | A new `version_id` is minted. The previous version is preserved.    |
| `scope`, `required`, `name`, `description`, or `production_sample_rate` updated | Written in place. No new version. Previous verdicts are unaffected. |

This distinction matters for auditability. The `params` fields are what a verdict was *decided by* — changing them should not retroactively alter the meaning of past results, so a new version is the right answer. The other fields say *where* and *how loudly* a grader applies, and those are live settings rather than historical facts.

Every verdict row references the specific grader `version_id` that produced it, so you can always trace back to the exact configuration that judged a given conversation.

## Configuring graders

### Via the dashboard

Navigate to **/graders**, find the library entry you want, and click **Use**. Fill in the params form, set the scope, and choose whether the grader is required or diagnostic.

### Via the API

Activate a grader with `POST /api/graders`:

```bash theme={null}
curl -X POST https://your-egma/api/graders \
  -H "Authorization: Bearer egma_..." \
  -H "Content-Type: application/json" \
  -d '{
    "library_id": "grl_01K…",
    "params": { "metric": "turn_response_latency", "bound": 2000 },
    "scope": "simulations",
    "required": true
  }'
```

Edit an existing grader with `PATCH /api/graders/:graderId`. Only the fields you include are updated:

```bash theme={null}
curl -X PATCH https://your-egma/api/graders/grd_01K… \
  -H "Authorization: Bearer egma_..." \
  -H "Content-Type: application/json" \
  -d '{ "required": false }'
```

<Warning>
  Send `params` as a properly typed JSON object, not as a string. Send `required` as `true` or `false`, not as `"true"` or `"false"`. Egma refuses wrong types by name rather than silently coercing them — a mis-typed field would otherwise leave your grader configured differently from what you intended, with nothing in the response to say so.
</Warning>

## Switching a grader off

Deleting a grader is the only way to turn it off. There is no enable/disable flag. From the moment a grader is deleted, it judges nothing in its scope. Use `DELETE /api/graders/:graderId`.

Deletion is a soft delete. Every verdict the grader already wrote remains readable and still references the version that produced it. An old run keeps its full meaning — you are not losing evidence, only stopping the grader from writing new verdicts going forward.

## Mock tools and grading

The `egma/mock-tools.md` file in your repository controls what Egma returns for your agent's tool calls during simulation. This is not a grader setting, but it is central to what graders are judging: the agent's behavior in a controlled world where tool responses are predictable.

Individual tests can override the project-level mock tools by including a `## Mock tools` section in their own file. That override is versioned alongside the test — changing it mints a new test version — while the project's mock tools are unversioned and a push always replaces what was there.

<Tip>
  Use mock tools to test failure branches. Set a tool to return an error, and add an expected behavior that asserts your agent handles the failure gracefully. Then graders can evaluate whether the agent did the right thing in a situation that is hard to reproduce with a real backend.
</Tip>