Create a custom grader

A custom grader uses an LLM to check conversation evidence against your instructions. It belongs to one project. You choose its model, scope, and pass threshold. The grader receives the transcript, tool names and arguments, observed metrics, and any expected behaviors from the test. Tool return payloads are not sent to the LLM. Use rules you can check with this evidence. The transcript and tool calls prefer the agent’s evidence when available. See Check which evidence arrived.

​

Create a grader

Open Graders in your project and select Create custom grader. Define a single rule with a clear pass and fail boundary. For an appointment agent, use: Choose Language model from the supported options. Under Scope, select Grades simulations and All simulations, or choose the test suites and tests that book appointments. Select Create grader. Egma saves the instructions as the first definition version and adds the grader to this project’s active graders with the chosen settings.

​

Write a check the evidence can settle

State what to inspect and what each result means. Include the case where the conversation never reaches the behavior being checked. For example, a rule about an appointment recap should say what happens when the caller asks only about opening hours. Without that detail, the grader can interpret the rule in different ways and return different results. Avoid instructions such as “be helpful” without a concrete test. Split independent concerns into separate graders so the result tells you what failed.

​

Check the grader

1. Run a test where the agent follows the rule and one where it breaks the rule.
2. Read the grader’s rationale and cited evidence in each simulation’s Results.
3. Check cases where the relevant action does not occur, the caller declines, or evidence is missing.
4. Set the scope and pass threshold for future conversations once the grader gives useful results.

​

Read the score

A check of one instruction returns 1 when met. It returns 0 when not met or when the model cannot determine whether it was met. A prompt that checks each supplied expected behavior can return a fractional score: the number of met behaviors divided by the total, including undetermined behaviors. Egma computes this score from the decisions; the model does not choose an arbitrary numeric score. An undetermined decision earns no credit toward the pass threshold. Its original decision, rationale, and citations remain in the grade details. A failed model request or an incomplete or invalid response produces a grading error with no score. Read the rationale and citations before changing your threshold. See Scores and thresholds.

​

Edit or clone a grader

Open the current custom definition in Grader library and select Edit core to change its grading instructions. A changed prompt creates a new immutable definition version. The name and description can change without creating a version. History and Egma-provided definitions are read-only. Select Clone grader on a current LLM definition to copy its instructions and this project’s effective settings into an independent custom grader. A clone receives no updates from its source. Code graders and historical versions cannot be cloned. Use Edit in Active graders to change the model, settings, scope, or threshold. These project settings do not create a definition version. Start a new run to use changed instructions or settings; queued runs and retries keep their saved selection.