OpenAPI 3.1.02026-08-226994571.1 MB

1e929292ddd5

OpenAI Evals API

Create Eval

Create a new evaluation.

Model-based routing (for multi-account support):

  • Pass model via header: x-litellm-model: gpt-4-account-1
  • Pass model via query: ?model=gpt-4-account-1
  • Pass model via body: {"model": "gpt-4-account-1"}

Example usage:

curl -X POST "http://localhost:4000/v1/evals"       -H "Authorization: Bearer your-key"       -H "Content-Type: application/json"       -d '{
    "name": "Test Eval",
    "data_source_config": {"type": "file", "file_id": "file-abc123"},
    "testing_criteria": {"graders": [{"type": "llm_as_judge"}]}
  }'

Returns: Eval object with id, status, timestamps, etc.

post/v1/evals

Query parameters

custom_llm_providerstring nullable

Response

Successful Response

idstring required
objectstring
created_atinteger required
updated_atinteger nullable
namestring nullable
data_source_configobject required
testing_criteriaobject[] required
metadataobject nullable