Evaluations

List Eval Jobs

get

List evaluation jobs

Authorizations

HTTPBearer

AuthorizationstringRequired

Bearer authentication header of the form Bearer <token>.

Query parameters

pageinteger · min: 1OptionalDefault: 1

limitinteger · min: 1 · max: 100OptionalDefault: 10

Responses

200

Successful Response

application/json

total_countintegerRequired

422

Validation Error

application/json

get

/evaluations

HTTPcURLJavaScriptPython

GET /api/public/v1/evaluations HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "jobs": [
    {
      "id": "text",
      "organization_id": "text",
      "config": {
        "organization_id": "text",
        "model_id": "text",
        "dataset": {
          "id": "text",
          "sample_ids": [
            "text"
          ],
          "prompt_template": "text"
        },
        "benchmark_model_id": "text",
        "temperature": 0,
        "max_tokens": 8192,
        "max_concurrent": 50
      },
      "cost_dollars": 1,
      "current_step": 1,
      "total_steps": 1,
      "status": "STARTING",
      "metrics": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "error_message": "text",
      "created_at": "2026-05-14T16:04:59.931Z",
      "updated_at": "2026-05-14T16:04:59.931Z"
    }
  ],
  "total_count": 1
}

Create Eval Job

post

Create a new evaluation job

Authorizations

HTTPBearer

AuthorizationstringRequired

Bearer authentication header of the form Bearer <token>.

Body

application/json

model_idstringRequired

benchmark_model_idany ofOptional

stringOptional

or

nullOptional

temperaturenumberOptionalDefault: 0

Responses

201

Successful Response

application/json

422

Validation Error

application/json

post

/evaluations

HTTPcURLJavaScriptPython

POST /api/public/v1/evaluations HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 134

{
  "dataset": {
    "id": "text",
    "sample_ids": [
      "text"
    ],
    "prompt_template": "text"
  },
  "model_id": "text",
  "benchmark_model_id": "text",
  "temperature": 0
}

{
  "id": "text",
  "organization_id": "text",
  "config": {
    "organization_id": "text",
    "model_id": "text",
    "dataset": {
      "id": "text",
      "sample_ids": [
        "text"
      ],
      "prompt_template": "text"
    },
    "benchmark_model_id": "text",
    "temperature": 0,
    "max_tokens": 8192,
    "max_concurrent": 50
  },
  "cost_dollars": 1,
  "current_step": 1,
  "total_steps": 1,
  "status": "STARTING",
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "error_message": "text",
  "created_at": "2026-05-14T16:04:59.931Z",
  "updated_at": "2026-05-14T16:04:59.931Z"
}

Get Eval Job

get

Get an evaluation job by ID

Authorizations

HTTPBearer

AuthorizationstringRequired

Bearer authentication header of the form Bearer <token>.

Path parameters

eval_idstringRequired

Responses

200

Successful Response

application/json

idstringRequired

organization_idstringRequired

cost_dollarsany ofOptional

numberOptional

or

nullOptional

current_stepany ofOptional

integerOptional

or

nullOptional

total_stepsany ofOptional

integerOptional

or

nullOptional

statusstring · enumRequiredPossible values:

metricsany ofOptional

or

nullOptional

error_messageany ofOptional

stringOptional

or

nullOptional

created_atstring · date-timeRequired

updated_atstring · date-timeRequired

422

Validation Error

application/json

get

/evaluations/{eval_id}

HTTPcURLJavaScriptPython

GET /api/public/v1/evaluations/{eval_id} HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "id": "text",
  "organization_id": "text",
  "config": {
    "organization_id": "text",
    "model_id": "text",
    "dataset": {
      "id": "text",
      "sample_ids": [
        "text"
      ],
      "prompt_template": "text"
    },
    "benchmark_model_id": "text",
    "temperature": 0,
    "max_tokens": 8192,
    "max_concurrent": 50
  },
  "cost_dollars": 1,
  "current_step": 1,
  "total_steps": 1,
  "status": "STARTING",
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "error_message": "text",
  "created_at": "2026-05-14T16:04:59.931Z",
  "updated_at": "2026-05-14T16:04:59.931Z"
}