Input for safety metric.

Input for summarization quality metric.

Input for summarization helpfulness metric.

Input for tool call valid metric.

Input for bleu metric.

Input for question answering helpfulness metric.

Input for Comet metric.

Input for question answering quality metric.

Input for pairwise metric.

Input for tool name match metric.

Input for MetricX metric.

Input for summarization verbosity metric.

Input for tool parameter key value match metric.

Input for rouge metric.

Input for groundedness metric.

Instances and metric spec for TrajectorySingleToolUse metric.

Input for exact match metric.

Input for pairwise question answering quality metric.

Input for fluency metric.

Input for pointwise metric.

Input for question answering correctness metric.

Input for question answering relevance metric.

The metrics used for evaluation. Currently, we only support evaluating a single metric. If multiple metrics are provided, only the first one will be evaluated.

Instances and metric spec for TrajectoryAnyOrderMatch metric.

Input for fulfillment metric.

Input for coherence metric.

Instances and metric spec for TrajectoryExactMatch metric.

Instances and metric spec for TrajectoryInOrderMatch metric.

Instances and metric spec for TrajectoryPrecision metric.

Optional. The metrics (either inline or registered) used for evaluation. Currently, we only support evaluating a single metric. If multiple metrics are provided, only the first one will be evaluated.

Instances and metric spec for TrajectoryRecall metric.

Instance and metric spec for RubricBasedInstructionFollowing metric.

The configs for autorater. This is applicable to both EvaluateInstances and EvaluateDataset.

Input for pairwise summarization quality metric.

Input for tool parameter key match metric.

A single instance to be evaluated. Instances are used to specify the input data for evaluation, from simple string comparisons to complex, multi-turn model evaluations