Create and use a custom LLM grader
curl --request POST \
--url https://app.egma.ai/v1/grader-library/custom \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"name": "Appointment recap",
"gradingInstructions": "Decide whether the agent repeats the chosen appointment date and time and asks the caller to confirm them. Use the transcript.",
"passesWhen": "The agent repeats the chosen date and time and asks for confirmation. If no appointment is chosen, the rule is met.",
"failsWhen": "An appointment is chosen and the agent ends the conversation without repeating both its date and time and asking for confirmation.",
"scope": {
"simulations": [
{
"kind": "all"
}
],
"production": null
},
"passThreshold": 1
}
'{
"definition": {
"id": "<string>",
"name": "<string>",
"description": "<string>",
"owner": "egma",
"type": "llm_as_judge",
"scopeEditable": true,
"currentDefinitionVersion": 2,
"definitionVersion": 2,
"modalities": [
"chat"
],
"gradingInstructions": "<string>",
"requiredEvidence": [
"transcript"
],
"settingDefinitions": [
{
"key": "<string>",
"label": "<string>",
"valueType": "integer",
"defaultValue": 123,
"unit": "<string>",
"minimum": 123,
"maximum": 123
}
],
"activeProjectGraderId": "<string>",
"createdAt": "2023-11-07T05:31:56Z",
"updatedAt": "2023-11-07T05:31:56Z"
},
"grader": {
"id": "<string>",
"projectId": "<string>",
"graderDefinitionId": "<string>",
"name": "<string>",
"description": "<string>",
"owner": "egma",
"type": "llm_as_judge",
"modalities": [
"chat"
],
"scopeEditable": true,
"removable": true,
"scope": {
"simulations": [
{
"kind": "all"
}
],
"production": {
"samplePercent": 50.5
}
},
"settings": {},
"passThreshold": 0.5,
"createdAt": "2023-11-07T05:31:56Z",
"updatedAt": "2023-11-07T05:31:56Z"
}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}Graders
Create and use a custom LLM grader
Create a custom LLM grader owned by this project. Set its instructions, model, scope, and pass threshold. Omit settings to save the declared model defaults at creation.
POST
/
v1
/
grader-library
/
custom
Create and use a custom LLM grader
curl --request POST \
--url https://app.egma.ai/v1/grader-library/custom \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"name": "Appointment recap",
"gradingInstructions": "Decide whether the agent repeats the chosen appointment date and time and asks the caller to confirm them. Use the transcript.",
"passesWhen": "The agent repeats the chosen date and time and asks for confirmation. If no appointment is chosen, the rule is met.",
"failsWhen": "An appointment is chosen and the agent ends the conversation without repeating both its date and time and asking for confirmation.",
"scope": {
"simulations": [
{
"kind": "all"
}
],
"production": null
},
"passThreshold": 1
}
'{
"definition": {
"id": "<string>",
"name": "<string>",
"description": "<string>",
"owner": "egma",
"type": "llm_as_judge",
"scopeEditable": true,
"currentDefinitionVersion": 2,
"definitionVersion": 2,
"modalities": [
"chat"
],
"gradingInstructions": "<string>",
"requiredEvidence": [
"transcript"
],
"settingDefinitions": [
{
"key": "<string>",
"label": "<string>",
"valueType": "integer",
"defaultValue": 123,
"unit": "<string>",
"minimum": 123,
"maximum": 123
}
],
"activeProjectGraderId": "<string>",
"createdAt": "2023-11-07T05:31:56Z",
"updatedAt": "2023-11-07T05:31:56Z"
},
"grader": {
"id": "<string>",
"projectId": "<string>",
"graderDefinitionId": "<string>",
"name": "<string>",
"description": "<string>",
"owner": "egma",
"type": "llm_as_judge",
"modalities": [
"chat"
],
"scopeEditable": true,
"removable": true,
"scope": {
"simulations": [
{
"kind": "all"
}
],
"production": {
"samplePercent": 50.5
}
},
"settings": {},
"passThreshold": 0.5,
"createdAt": "2023-11-07T05:31:56Z",
"updatedAt": "2023-11-07T05:31:56Z"
}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}{
"error": "<string>",
"message": "<string>",
"details": {}
}Authorizations
bearerAuthsessionCookie
An Egma API key.
Query Parameters
Minimum string length:
1Body
application/json
One rule to decide and the conversation evidence to inspect.
The evidence that makes the rule pass. Include how to handle a conversation where the checked action never occurs.
The evidence that makes the rule fail.
The future simulations and/or production sample this project should grade.
Show child attributes
Show child attributes
The minimum score for this grader's individual result to pass.
Required range:
0 <= x <= 1A complete llm_provider and llm_model pair from Get grader model choices. Omit settings to use the declared defaults.