---
title: "POST /evaluation-jobs"
method: POST
path: "/evaluation-jobs"
---

# POST /evaluation-jobs

`POST /evaluation-jobs`

Creates an evaluation job.

## Request body

- object
  - `jobName` string, required — A name for the evaluation job. Names must unique with your Amazon Web Services account, and your account's Amazon Web Services region.
  - `jobDescription` string, password — A description of the evaluation job.
  - `clientRequestToken` string — A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If this token matches a previous request, Amazon Bedrock ignores the request, but does not return an error. For more information, see <a href="https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Run_Instance_Idempotency.html">Ensuring idempotency</a>.
  - `roleArn` string, required — The Amazon Resource Name (ARN) of an IAM service role that Amazon Bedrock can assume to perform tasks on your behalf. To learn more about the required permissions, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-security.html">Required permissions for model evaluations</a>.
  - `customerEncryptionKeyId` string — Specify your customer managed encryption key Amazon Resource Name (ARN) that will be used to encrypt your evaluation job.
  - `jobTags` Tag[] — Tags to attach to the model evaluation job.
    - `key` string, required — Key for the tag.
    - `value` string, required — Value for the tag.
  - `applicationType` 'ModelEvaluation' | 'RagEvaluation' — Specifies whether the evaluation job is for evaluating a model or evaluating a knowledge base (retrieval and response generation).
  - `evaluationConfig` object, required — The configuration details of either an automated or human-based evaluation job.
    - `automated` object — Contains the configuration details of an automated evaluation job that computes metrics.
      - `datasetMetricConfigs` EvaluationDatasetMetricConfig[], required — Configuration details of the prompt datasets and metrics you want to use for your evaluation job.
        - `taskType` 'Summarization' | 'Classification' | 'QuestionAndAnswer' | 'Generation' | 'Custom', required — The the type of task you want to evaluate for your evaluation job. This applies only to model evaluation jobs and is ignored for knowledge base evaluation jobs.
        - `dataset` object, required — Specifies the prompt dataset.
          - `name` string, password, required — Used to specify supported built-in prompt datasets. Valid values are <code>Builtin.Bold</code>, <code>Builtin.BoolQ</code>, <code>Builtin.NaturalQuestions</code>, <code>Builtin.Gigaword</code>, <code>Builtin.RealToxicityPrompts</code>, <code>Builtin.TriviaQA</code>, <code>Builtin.T-Rex</code>, <code>Builtin.WomensEcommerceClothingReviews</code> and <code>Builtin.Wikitext2</code>.
          - `datasetLocation` object — For custom prompt datasets, you must specify the location in Amazon S3 where the prompt dataset is saved.
            - `s3Uri` string — The S3 URI of the S3 bucket specified in the job.
        - `metricNames` EvaluationMetricName[], required — <p>The names of the metrics you want to use for your evaluation job.</p> <p>For knowledge base evaluation jobs that evaluate retrieval only, valid values are "<code>Builtin.ContextRelevance</code>", "<code>Builtin.ContextCoverage</code>".</p> <p>For knowledge base evaluation jobs that evaluate retrieval with response generation, valid values are "<code>Builtin.Correctness</code>", "<code>Builtin.Completeness</code>", "<code>Builtin.Helpfulness</code>", "<code>Builtin.LogicalCoherence</code>", "<code>Builtin.Faithfulness</code>", "<code>Builtin.Harmfulness</code>", "<code>Builtin.Stereotyping</code>", "<code>Builtin.Refusal</code>".</p> <p>For automated model evaluation jobs, valid values are "<code>Builtin.Accuracy</code>", "<code>Builtin.Robustness</code>", and "<code>Builtin.Toxicity</code>". In model evaluation jobs that use a LLM as judge you can specify "<code>Builtin.Correctness</code>", "<code>Builtin.Completeness"</code>, "<code>Builtin.Faithfulness"</code>, "<code>Builtin.Helpfulness</code>", "<code>Builtin.Coherence</code>", "<code>Builtin.Relevance</code>", "<code>Builtin.FollowingInstructions</code>", "<code>Builtin.ProfessionalStyleAndTone</code>", You can also specify the following responsible AI related metrics only for model evaluation job that use a LLM as judge "<code>Builtin.Harmfulness</code>", "<code>Builtin.Stereotyping</code>", and "<code>Builtin.Refusal</code>".</p> <p>For human-based model evaluation jobs, the list of strings must match the <code>name</code> parameter specified in <code>HumanEvaluationCustomMetric</code>.</p>
      - `evaluatorModelConfig` object — Contains the evaluator model configuration details. <code>EvaluatorModelConfig</code> is required for evaluation jobs that use a knowledge base or in model evaluation job that use a model as judge. This model computes all evaluation related metrics.
        - `bedrockEvaluatorModels` BedrockEvaluatorModel[] — The evaluator model used in knowledge base evaluation job or in model evaluation job that use a model as judge. This model computes all evaluation related metrics.
          - `modelIdentifier` string, required — The Amazon Resource Name (ARN) of the evaluator model used used in knowledge base evaluation job or in model evaluation job that use a model as judge.
      - `customMetricConfig` object — Defines the configuration of custom metrics to be used in an evaluation job.
        - `customMetrics` AutomatedEvaluationCustomMetricSource[], required — Defines a list of custom metrics to be used in an Amazon Bedrock evaluation job.
          - `customMetricDefinition` object — The definition of a custom metric for use in an Amazon Bedrock evaluation job.
            - `name` string, password, required — The name for a custom metric. Names must be unique in your Amazon Web Services region.
            - `instructions` string, required — The prompt for a custom metric that instructs the evaluator model how to rate the model or RAG source under evaluation.
            - `ratingScale` RatingScaleItem[] — Defines the rating scale to be used for a custom metric. We recommend that you always define a ratings scale when creating a custom metric. If you don't define a scale, Amazon Bedrock won't be able to visually display the results of the evaluation in the console or calculate average values of numerical scores. For more information on specifying a rating scale, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html#model-evaluation-custom-metrics-prompt-formats-schema">Specifying an output schema (rating scale)</a>.
              - …
        - `evaluatorModelConfig` object, required — Configuration of the evaluator model you want to use to evaluate custom metrics in an Amazon Bedrock evaluation job.
          - `bedrockEvaluatorModels` CustomMetricBedrockEvaluatorModel[], required — Defines the model you want to evaluate custom metrics in an Amazon Bedrock evaluation job.
            - `modelIdentifier` string, required — The Amazon Resource Name (ARN) of the evaluator model for custom metrics. For a list of supported evaluator models, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html">Evaluate model performance using another LLM as a judge</a> and <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-kb.html">Evaluate the performance of RAG sources using Amazon Bedrock evaluations</a>.
    - `human` object — Contains the configuration details of an evaluation job that uses human workers.
      - `humanWorkflowConfig` object — The parameters of the human workflow.
        - `flowDefinitionArn` string, required — The Amazon Resource Number (ARN) for the flow definition
        - `instructions` string, password — Instructions for the flow definition
      - `customMetrics` HumanEvaluationCustomMetric[] — A <code>HumanEvaluationCustomMetric</code> object. It contains the names the metrics, how the metrics are to be evaluated, an optional description.
        - `name` string, password, required — The name of the metric. Your human evaluators will see this name in the evaluation UI.
        - `description` string, password — An optional description of the metric. Use this parameter to provide more details about the metric.
        - `ratingMethod` string, required — Choose how you want your human workers to evaluation your model. Valid values for rating methods are <code>ThumbsUpDown</code>, <code>IndividualLikertScale</code>,<code>ComparisonLikertScale</code>, <code>ComparisonChoice</code>, and <code>ComparisonRank</code>
      - `datasetMetricConfigs` EvaluationDatasetMetricConfig[], required — Use to specify the metrics, task, and prompt dataset to be used in your model evaluation job.
        - `taskType` 'Summarization' | 'Classification' | 'QuestionAndAnswer' | 'Generation' | 'Custom', required — The the type of task you want to evaluate for your evaluation job. This applies only to model evaluation jobs and is ignored for knowledge base evaluation jobs.
        - `dataset` object, required — Specifies the prompt dataset.
          - `name` string, password, required — Used to specify supported built-in prompt datasets. Valid values are <code>Builtin.Bold</code>, <code>Builtin.BoolQ</code>, <code>Builtin.NaturalQuestions</code>, <code>Builtin.Gigaword</code>, <code>Builtin.RealToxicityPrompts</code>, <code>Builtin.TriviaQA</code>, <code>Builtin.T-Rex</code>, <code>Builtin.WomensEcommerceClothingReviews</code> and <code>Builtin.Wikitext2</code>.
          - `datasetLocation` object — For custom prompt datasets, you must specify the location in Amazon S3 where the prompt dataset is saved.
            - `s3Uri` string — The S3 URI of the S3 bucket specified in the job.
        - `metricNames` EvaluationMetricName[], required — <p>The names of the metrics you want to use for your evaluation job.</p> <p>For knowledge base evaluation jobs that evaluate retrieval only, valid values are "<code>Builtin.ContextRelevance</code>", "<code>Builtin.ContextCoverage</code>".</p> <p>For knowledge base evaluation jobs that evaluate retrieval with response generation, valid values are "<code>Builtin.Correctness</code>", "<code>Builtin.Completeness</code>", "<code>Builtin.Helpfulness</code>", "<code>Builtin.LogicalCoherence</code>", "<code>Builtin.Faithfulness</code>", "<code>Builtin.Harmfulness</code>", "<code>Builtin.Stereotyping</code>", "<code>Builtin.Refusal</code>".</p> <p>For automated model evaluation jobs, valid values are "<code>Builtin.Accuracy</code>", "<code>Builtin.Robustness</code>", and "<code>Builtin.Toxicity</code>". In model evaluation jobs that use a LLM as judge you can specify "<code>Builtin.Correctness</code>", "<code>Builtin.Completeness"</code>, "<code>Builtin.Faithfulness"</code>, "<code>Builtin.Helpfulness</code>", "<code>Builtin.Coherence</code>", "<code>Builtin.Relevance</code>", "<code>Builtin.FollowingInstructions</code>", "<code>Builtin.ProfessionalStyleAndTone</code>", You can also specify the following responsible AI related metrics only for model evaluation job that use a LLM as judge "<code>Builtin.Harmfulness</code>", "<code>Builtin.Stereotyping</code>", and "<code>Builtin.Refusal</code>".</p> <p>For human-based model evaluation jobs, the list of strings must match the <code>name</code> parameter specified in <code>HumanEvaluationCustomMetric</code>.</p>
  - `inferenceConfig` object, required — <p>The configuration details of the inference model for an evaluation job.</p> <p>For automated model evaluation jobs, only a single model is supported.</p> <p>For human-based model evaluation jobs, your annotator can compare the responses for up to two different models.</p>
    - `models` EvaluationModelConfig[] — Specifies the inference models.
      - `bedrockModel` object — Defines the Amazon Bedrock model or inference profile and inference parameters you want used.
        - `modelIdentifier` string, required — The ARN of the Amazon Bedrock model or inference profile specified.
        - `inferenceParams` string, password — Each Amazon Bedrock support different inference parameters that change how the model behaves during inference.
        - `performanceConfig` object — Specifies performance settings for the model or inference profile.
          - `latency` 'standard' | 'optimized' — Specifies whether to use the latency-optimized or standard version of a model or inference profile.
      - `precomputedInferenceSource` object — Defines the model used to generate inference response data for a model evaluation job where you provide your own inference response data.
        - `inferenceSourceIdentifier` string, required — A label that identifies a model used in a model evaluation job where you provide your own inference response data.
    - `ragConfigs` RAGConfig[] — Contains the configuration details of the inference for a knowledge base evaluation job, including either the retrieval only configuration or the retrieval with response generation configuration.
      - `knowledgeBaseConfig` object — Contains configuration details for knowledge base retrieval and response generation.
        - `retrieveConfig` object — Contains configuration details for retrieving information from a knowledge base.
          - `knowledgeBaseId` string, required — The unique identifier of the knowledge base.
          - `knowledgeBaseRetrievalConfiguration` object, required — Contains configuration details for knowledge base retrieval.
            - `vectorSearchConfiguration` object, required — Contains configuration details for returning the results from the vector search.
              - …
        - `retrieveAndGenerateConfig` object — Contains configuration details for retrieving information from a knowledge base and generating responses.
          - `type` 'KNOWLEDGE_BASE' | 'EXTERNAL_SOURCES', required — <p>The type of resource that contains your data for retrieving information and generating responses.</p> <p>If you choose to use <code>EXTERNAL_SOURCES</code>, then currently only Claude 3 Sonnet models for knowledge bases are supported.</p>
          - `knowledgeBaseConfiguration` object — Contains configuration details for the knowledge base retrieval and response generation.
            - `knowledgeBaseId` string, required — The unique identifier of the knowledge base.
            - `modelArn` string, required — The Amazon Resource Name (ARN) of the foundation model or <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html">inference profile</a> used to generate responses.
            - `retrievalConfiguration` object — Contains configuration details for retrieving text chunks.
              - …
            - `generationConfiguration` object — Contains configurations details for response generation based on retrieved text chunks.
              - …
            - `orchestrationConfiguration` object — Contains configuration details for the model to process the prompt prior to retrieval and response generation.
              - …
          - `externalSourcesConfiguration` object — The configuration for the external source wrapper object in the <code>retrieveAndGenerate</code> function.
            - `modelArn` string, required — The Amazon Resource Name (ARN) of the foundation model or <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html">inference profile</a> used to generate responses.
            - `sources` ExternalSource[], required — The document for the external source wrapper object in the <code>retrieveAndGenerate</code> function.
              - …
            - `generationConfiguration` object — Contains configurations details for response generation based on retrieved text chunks.
              - …
      - `precomputedRagSourceConfig` object — Contains configuration details about the RAG source used to generate inference response data for a Knowledge Base evaluation job.
        - `retrieveSourceConfig` object — A summary of a RAG source used for a retrieve-only Knowledge Base evaluation job where you provide your own inference response data.
          - `ragSourceIdentifier` string, required — A label that identifies the RAG source used for a retrieve-only Knowledge Base evaluation job where you provide your own inference response data.
        - `retrieveAndGenerateSourceConfig` object — A summary of a RAG source used for a retrieve-and-generate Knowledge Base evaluation job where you provide your own inference response data.
          - `ragSourceIdentifier` string, required — A label that identifies the RAG source used for a retrieve-and-generate Knowledge Base evaluation job where you provide your own inference response data.
  - `outputDataConfig` object, required — The Amazon S3 location where the results of your evaluation job are saved.
    - `s3Uri` string — The Amazon S3 URI where the results of the evaluation job are saved.

## Response `202`

Success

- CreateEvaluationJobResponse
  - `jobArn` string, required — The Amazon Resource Name (ARN) of the evaluation job.

## Other responses

- `480` — ResourceNotFoundException
- `481` — AccessDeniedException
- `482` — ValidationException
- `483` — ConflictException
- `484` — InternalServerException
- `485` — ServiceQuotaExceededException
- `486` — ThrottlingException

---

[API](https://skmtc.net/aws/apis/bedrock.md) · [All operations](https://skmtc.net/aws/apis/bedrock/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/aws/bedrock/versions/3bc874e43ecd/schema)