v48

latestOpenAPI 3.1.0raw.githubusercontent.com2026-05-275867561.6 MB
run

Validate Rule

Validate a rule by executing it with test data without creating a saved rule.

This endpoint allows testing LLM-as-judge evaluators before saving them. It accepts a rule configuration (same as rule creation) and test data, executes the evaluator, and returns the evaluation results in the same format as batch_invoke_evaluator.

Only LLM-as-judge rules (evaluators) are supported. Code evaluators are not allowed.

The evaluator execution traces are written to the database (in the "evaluators" project), which allows users to see the evaluator execution history.

post/api/v1/runs/rules/validate

Request body

display_namestring required
session_idstring uuid nullable
is_enabledboolean
dataset_idstring uuid nullable
sampling_ratenumber required
filterstring nullable
trace_filterstring nullable
tree_filterstring nullable
backfill_fromstring date-time nullable
use_corrections_datasetboolean
num_few_shot_examplesinteger nullable
extend_onlyboolean
transientboolean
add_to_annotation_queue_idstring uuid nullable
add_to_dataset_idstring uuid nullable
add_to_dataset_prefer_correctionboolean
evaluator_idstring uuid nullable
evaluator_versioninteger nullable
create_alignment_queueboolean
include_extended_statsboolean
group_by'thread_id' nullable
tracer_session_issue_idstring uuid nullable
test_inputsobject nullable
test_outputsobject nullable
test_reference_outputsobject nullable
test_attachmentsobject nullable

Response

Successful Response

object[] required