--- title: "Create eval" method: POST path: "/evals" tags: ["Evals"] --- # Create eval `POST /evals` Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals). ## Request body - CreateEvalRequest - `name` string — The name of the evaluation. - `metadata` Metadata, nullable — Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. - `data_source_config` union, required — The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation. - object — A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs. This schema is used to define the shape of the data that will be: - Used to define your testing criteria and - What data is required when creating a run - `type` 'custom', required — The type of data source. Always `custom`. - `item_schema` object, required — The json schema for each row in the data source. - `include_sample_schema` boolean — Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source) - object — A data source config which specifies the metadata property of your logs query. This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc. - `type` 'logs', required — The type of data source. Always `logs`. - `metadata` object — Metadata filters for the logs data source. - object — Deprecated in favor of LogsDataSourceConfig. - `type` 'stored_completions', required — The type of data source. Always `stored_completions`. - `metadata` object — Metadata filters for the stored completions data source. - `testing_criteria` union[], required — A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`). - union - CreateEvalLabelModelGrader — A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `type` 'label_model', required — The object type, which is always `label_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. Must support structured outputs. - `input` CreateEvalItem[], required — A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - union — A chat message that makes up the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - object - … - object — A message input to the model with a role indicating instruction following hierarchy. Instructions given with the `developer` or `system` role take precedence over instructions given with the `user` role. Messages with the `assistant` role are presumed to have been generated by the model in previous interactions. - … - `labels` string[], required — The labels to classify to each item in the evaluation. - `passing_labels` string[], required — The labels that indicate a passing result. Must be a subset of labels. - EvalGraderStringCheck — A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `type` 'string_check', required — The object type, which is always `string_check`. - `name` string, required — The name of the grader. - `input` string, required — The input text. This may include template strings. - `reference` string, required — The reference text. This may include template strings. - `operation` 'eq' | 'ne' | 'like' | 'ilike', required — The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - EvalGraderTextSimilarity — A TextSimilarityGrader object which grades text based on similarity metrics. - `type` 'text_similarity', required — The type of grader. - `name` string, required — The name of the grader. - `input` string, required — The text being graded. - `reference` string, required — The text being graded against. - `evaluation_metric` 'cosine' | 'fuzzy_match' | 'bleu' | 'gleu' | 'meteor' | 'rouge_1' | 'rouge_2' | 'rouge_3' | 'rouge_4' | 'rouge_5' | 'rouge_l', required — The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `pass_threshold` number, required — The threshold for the score. - EvalGraderPython — A PythonGrader object that runs a python script on the input. - `type` 'python', required — The object type, which is always `python`. - `name` string, required — The name of the grader. - `source` string, required — The source code of the python script. - `image_tag` string — The image tag to use for the python script. - `pass_threshold` number — The threshold for the score. - EvalGraderScoreModel — A ScoreModelGrader object that uses a model to assign a score to the input. - `type` 'score_model', required — The object type, which is always `score_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. - `sampling_params` object — The sampling parameters for the model. - `seed` integer, nullable — A seed value to initialize the randomness, during sampling. - `top_p` number, nullable — An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - `temperature` number, nullable — A higher temperature increases randomness in the outputs. - `max_completions_tokens` integer, nullable — The maximum number of tokens the grader model may generate in its response. - `reasoning_effort` 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max', nullable — Constrains effort on reasoning for reasoning models. Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the [reasoning guide](https://platform.openai.com/docs/guides/reasoning) for model-specific support. - `input` EvalItem[], required — The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `role` 'user' | 'assistant' | 'system' | 'developer', required — The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `content` union, required — Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - union — A single content item: input text, output text, input image, or input audio. - … - EvalItemContentItem[] — A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - … - `type` 'message' — The type of the message input. Always `message`. - `range` number[] — The range of the score. Defaults to `[0, 1]`. - `pass_threshold` number — The threshold for the score. ## Response `201` OK - Eval — An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like: - Improve the quality of my chatbot - See how well my chatbot handles customer support - Check if o4-mini is better at my usecase than gpt-4o - `object` 'eval', required — The object type. - `id` string, required — Unique identifier for the evaluation. - `name` string, required — The name of the evaluation. - `data_source_config` union, required — Configuration of data sources used in runs of the evaluation. - object — A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces. The response schema defines the shape of the data that will be: - Used to define your testing criteria and - What data is required when creating a run - `type` 'custom', required — The type of data source. Always `custom`. - `schema` object, required — The json schema for the run data source items. Learn how to build JSON schemas [here](https://json-schema.org/). - object — A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc. The schema returned by this data source config is used to defined what variables are available in your evals. `item` and `sample` are both defined when using this data source config. - `type` 'logs', required — The type of data source. Always `logs`. - `metadata` Metadata, nullable — Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. - `schema` object, required — The json schema for the run data source items. Learn how to build JSON schemas [here](https://json-schema.org/). - object — Deprecated in favor of LogsDataSourceConfig. - `type` 'stored_completions', required — The type of data source. Always `stored_completions`. - `metadata` Metadata, nullable — Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. - `schema` object, required — The json schema for the run data source items. Learn how to build JSON schemas [here](https://json-schema.org/). - `testing_criteria` union[], required — A list of testing criteria. - union - EvalGraderLabelModel — A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. - `type` 'label_model', required — The object type, which is always `label_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. Must support structured outputs. - `input` EvalItem[], required - `role` 'user' | 'assistant' | 'system' | 'developer', required — The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `content` union, required — Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - union — A single content item: input text, output text, input image, or input audio. - … - EvalItemContentItem[] — A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - … - `type` 'message' — The type of the message input. Always `message`. - `labels` string[], required — The labels to assign to each item in the evaluation. - `passing_labels` string[], required — The labels that indicate a passing result. Must be a subset of labels. - EvalGraderStringCheck — A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. - `type` 'string_check', required — The object type, which is always `string_check`. - `name` string, required — The name of the grader. - `input` string, required — The input text. This may include template strings. - `reference` string, required — The reference text. This may include template strings. - `operation` 'eq' | 'ne' | 'like' | 'ilike', required — The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. - EvalGraderTextSimilarity — A TextSimilarityGrader object which grades text based on similarity metrics. - `type` 'text_similarity', required — The type of grader. - `name` string, required — The name of the grader. - `input` string, required — The text being graded. - `reference` string, required — The text being graded against. - `evaluation_metric` 'cosine' | 'fuzzy_match' | 'bleu' | 'gleu' | 'meteor' | 'rouge_1' | 'rouge_2' | 'rouge_3' | 'rouge_4' | 'rouge_5' | 'rouge_l', required — The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. - `pass_threshold` number, required — The threshold for the score. - EvalGraderPython — A PythonGrader object that runs a python script on the input. - `type` 'python', required — The object type, which is always `python`. - `name` string, required — The name of the grader. - `source` string, required — The source code of the python script. - `image_tag` string — The image tag to use for the python script. - `pass_threshold` number — The threshold for the score. - EvalGraderScoreModel — A ScoreModelGrader object that uses a model to assign a score to the input. - `type` 'score_model', required — The object type, which is always `score_model`. - `name` string, required — The name of the grader. - `model` string, required — The model to use for the evaluation. - `sampling_params` object — The sampling parameters for the model. - `seed` integer, nullable — A seed value to initialize the randomness, during sampling. - `top_p` number, nullable — An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - `temperature` number, nullable — A higher temperature increases randomness in the outputs. - `max_completions_tokens` integer, nullable — The maximum number of tokens the grader model may generate in its response. - `reasoning_effort` 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max', nullable — Constrains effort on reasoning for reasoning models. Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the [reasoning guide](https://platform.openai.com/docs/guides/reasoning) for model-specific support. - `input` EvalItem[], required — The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. - `role` 'user' | 'assistant' | 'system' | 'developer', required — The role of the message input. One of `user`, `assistant`, `system`, or `developer`. - `content` union, required — Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - union — A single content item: input text, output text, input image, or input audio. - … - EvalItemContentItem[] — A list of inputs, each of which may be either an input text, output text, input image, or input audio object. - … - `type` 'message' — The type of the message input. Always `message`. - `range` number[] — The range of the score. Defaults to `[0, 1]`. - `pass_threshold` number — The threshold for the score. - `created_at` integer, required — The Unix timestamp (in seconds) for when the eval was created. - `metadata` Metadata, nullable, required — Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. --- [API](https://skmtc.net/openai/apis/openai-api.md) · [All operations](https://skmtc.net/openai/apis/openai-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/openai/openai-api/revisions/dcd35773a322/schema)