Api V 2 Experiments List

GET: List experiments POST: Create and run an experiment workflow execution.

Design Philosophy: Dataset Log → [Arbitrary Workflow Tasks] → Run Eval Steps → Scores for Comparison

• Workflow tasks can be arbitrary (any combination, end with whatever)
• evaluator_ids field specifies which evaluators produce scores (preferred)
• evaluator_slugs is a deprecated alias for evaluator_ids (backward compat)
• Workflow step type eval is canonical; legacy evaluator is still accepted on input
• Multiple evaluators can run for comprehensive comparison (e.g., quality, safety, cost)

This endpoint processes a dataset through a sequence of workflow tasks and generates traces for experiment tracking.

Request Body:

{
    "dataset_id": "dataset-123",
    "workflow": [
        {
            "type": "prompt",
            "config": {
                "prompt_id": "my-prompt-slug"
            }
        },
        {
            "type": "completion",
            "config": {
                "model": "gpt-4",
                "temperature": 0.7,
                "max_tokens": 1000,
                "top_p": 1.0,
                "frequency_penalty": 0,
                "presence_penalty": 0
            }
        }
    ],
    "evaluator_ids": ["abc123-evaluator-uuid", "def456-evaluator-uuid"],
    "experiment_id": "exp-run-456",
    "name": "My Experiment",
    "description": "Testing GPT-4 with my prompt",
    "span_workflow_name": "My Experiment",
    "enable_tracing": true
}

Response:

{
    "task_id": "celery-task-id",
    "task_tracker_id": "tracker-id",
    "status": "pending",
    "message": "Workflow execution task started",
    "experiment_id": "exp-run-456"
}

Workflow Types & Configurations:

Key Fields:

• workflow: Array of workflow tasks (can be any combination)
• evaluator_ids: Array of evaluator IDs (optional) - runs after workflow tasks complete; omit or pass [] to produce outputs with no scores
• evaluator_slugs: Deprecated alias for evaluator_ids (still accepted)

Example Configurations:

• workflow: [prompt, completion] + evaluator_ids: ["abc123"] ✓
• workflow: [completion] + evaluator_ids: ["abc123", "def456"] ✓
• workflow: [] + evaluator_ids: ["abc123"] ✗ (workflow is required)
• workflow: [prompt, completion] + evaluator_ids: [] ✓ (runs the workflow, produces outputs with no scores)

1. Prompt Workflow (type: "prompt"):

   • Variables are dynamically filled from dataset entries (don't pass in config)
   • Config fields:
     • prompt_id: Prompt identifier (required)
   • Input: Dataset entry fields (mapped to prompt variables)
   • Output: Rendered messages array

2. Completion Workflow (type: "completion"):

   • Uses LLM configuration fields (temperature, max_tokens, etc.)
   • Messages come from previous workflow step or dataset input field
   • Config fields (all optional):
     • model: Model identifier (e.g., "gpt-4", "claude-3-opus")
     • temperature: Sampling temperature (0-2)
     • max_tokens: Maximum completion tokens
     • top_p: Nucleus sampling parameter
     • frequency_penalty: Frequency penalty (-2 to 2)
     • presence_penalty: Presence penalty (-2 to 2)
     • stop: Stop sequences (string or array)
     • n: Number of completions to generate
     • stream: Enable streaming (not recommended for experiments)
     • response_format: Response format (e.g., {"type": "json_object"})
     • tools: Function calling tools array
     • tool_choice: Tool choice strategy
     • reasoning_effort: Reasoning effort for o1 models
   • Input: Messages array (from "input" field in unified log format)
   • Output: Response message object (stored in "output" field)

3. Eval Workflow (type: "eval"):

   • Runs an evaluator on the unified log format (can be in workflow or via evaluator_slugs)
   • Produces scores for experiment comparison
   • Config fields:
     • evaluator_slug: Evaluator identifier (required)
   • Input: Unified log format with input/output/metrics/metadata
   • Output: Evaluation result with score

Note: Eval steps can be in workflow OR specified via evaluator_slugs field (recommended). The evaluator_slugs field at experiment level ensures all specified evaluators run after workflow tasks complete.

4. Condition Workflow (type: "condition"):

   • Evaluates condition policies (future implementation)
   • Config fields: TBD

5. Duplicate Workflow (type: "duplicate"):

   • Passes through dataset entry's input/output as-is (no LLM inference)
   • Useful for batch evaluation on existing production data
   • Config fields (all optional):
     • name: Display name for the workflow span
   • Input: Dataset entry's input field
   • Output: Dataset entry's output field (unchanged)
   • Evaluators then run on this duplicated data

Authentication:

• Supports both JWT (internal) and API Key (public) authentication