---
title: "Test Run Agent"
method: POST
path: "/agents/test-run"
tags: ["Agents"]
---

# Test Run Agent

`POST /agents/test-run`

Run an agent against a single sample record without deploying to Flink.

Synchronous, hard-capped at 15s wall-clock. Calls the LLM provider
directly + dispatches MCP and HTTP tools live; transform tools and
memory are intentionally not exercised (the Flink runtime owns
them). Per-tenant rate limit: shares the validate-llm bucket
(30 req/min/tenant) -> 429.

Provider failures (timeouts, schema errors, unsupported provider) land
as a structured ``error`` object on the envelope (HTTP still 200) so
the FE can render the partial timeline alongside the error context.
Only the input-token cap raises (422) - at that point we can't even
start the run.

## Request body

- TestRunRequest — Request body for ``POST /agents/test-run``. Carries a full agent config + a sample record. The BE runs the record through the agent (LLM + tools) without touching Flink and returns the timeline so the FE can show what happened.
  - `config` CreateAgentConfigRequest, required — Request body for POST /agents/config - create a config-based agent.
    - `name` string, required — Agent display name
    - `description` string, nullable — Human-readable context for the agent
    - `enabled` boolean — Kill switch. When false, the Flink runtime accepts the deploy but stops emitting output — used by support to disable a runaway tenant agent without a full cancel/redeploy.
    - `agentType` 'workflow' | 'react'
    - `input` AgentInputConfig, required — Input configuration for the agent.
      - `topicPattern` string, required — Regex pattern for input topics, e.g. '^(orders)$'
      - `inputSerialization` 'JsonConfluent' | 'AvroConfluent'
      - `fields` string[], nullable — Column-level filter - only these fields sent to LLM
      - `createTableSQL` string, nullable — Flink SQL CREATE TABLE (required if filterSQL is set)
      - `filterSQL` string, nullable — WHERE clause predicate for row-level filtering
    - `output` AgentOutputConfig, required — Output configuration for the agent.
      - `topic` string, required — Fixed output topic name
      - `deadLetterTopic` string, nullable — DLQ topic name
      - `outputSerialization` 'JsonConfluent' | 'AvroConfluent'
      - `schema` object — Output schema: {field_name: TYPE_STRING}
    - `llm` AgentLlmConfig, required — Inline LLM configuration embedded on the agent. Carries provider, credentials, model, and tuning controls as a single blob. The FE prefills the form from a saved :class:`AgentLlmConnection` (which now carries default model + tuning + reasoning controls) and the user can override per-agent before save; the resulting blob is stored verbatim and shipped to the Flink runtime at deploy. Two credential shapes (XOR, mirrors :class:`ValidateLlmRequest`): 1. **Inline** - ``apiKey`` is filled directly. The BE KMS-encrypts it at save and decrypts it at deploy / test-run. 2. **Linked** - ``llmConnectionId`` references a row in ``agent_connections.llmConnections`` and ``apiKey`` is left empty. The BE resolves the real key server-side at deploy and at test-run via :func:`agents_service.resolve_saved_llm_credentials`, so the browser never needs to hold (or re-paste) the stored plaintext when editing an existing agent. ``provider == ollama`` accepts an empty key without a link (the runtime is local and unauthenticated).
      - `provider` 'anthropic' | 'openai' | 'openai-responses' | 'ollama' | 'azure' | 'azure-openai' | 'bedrock' | 'qwen' | 'openai-compatible', required — Unified LLM provider enum. A single ``AgentLlmConnection`` row carries one provider and a set of capabilities (chat / embedding). ``PROVIDER_CAPABILITIES`` below pins which capabilities each provider can serve — picked by the FE Connections drawer and re-validated server-side on every write.
      - `model` string, required — Model name, e.g. claude-sonnet-4-20250514
      - `apiKey` string — API key - use '${SECRET:ENV_VAR_NAME}' for env var resolution. Empty when llmConnectionId is set.
      - `llmConnectionId` string, nullable — Tenant-scoped reference to ``agent_connections.llmConnections[].id``. When set with an empty ``apiKey``, the BE resolves the real key server-side at deploy and at test-run.
      - `baseUrl` string, nullable — Base URL for OpenAI-compatible proxies
      - `temperature` number, nullable — Sampling temperature (null = matrix-skipped)
      - `maxTokens` integer, nullable — Max output tokens (null = matrix-skipped)
      - `timeout` integer, nullable — Request timeout in seconds (null = matrix-skipped)
      - `reasoningEffort` 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh', nullable — OpenAI / GPT-5 reasoning effort. The accepted set is per-model: the matrix in app/utils/llm_capabilities.py rejects values not in caps.reasoning_effort_values for the chosen provider+model.
      - `thinkingBudgetTokens` integer, nullable — Anthropic Claude 4 extended-thinking budget (tokens). Non-null enables thinking.
      - `ollamaThink` boolean, nullable — Ollama 'think' toggle for reasoning models
      - `maxRetries` integer, nullable — Universal — max retry attempts on LLM call. Accepted by every provider.
      - `region` string, nullable — Bedrock: AWS region (e.g. us-east-1). Ignored for non-bedrock providers.
      - `strict` boolean, nullable — openai-responses: enable JSON-schema strict mode.
      - `store` boolean, nullable — openai-responses: server-side response storage flag.
      - `instructions` string, nullable — openai-responses: system-level instructions passed as a top-level Responses param.
      - `additionalKwargs` object, nullable — openai-responses / azure-openai: free-form extra request params forwarded verbatim by the runtime.
      - `apiVersion` string, nullable — azure-openai: Azure OpenAI API version (e.g. '2024-02-01'). Required for azure-openai.
      - `azureEndpoint` string, nullable — azure-openai: Azure resource endpoint (e.g. https://<resource>.openai.azure.com). Required.
      - `azureUrlPathMode` 'AUTO' | 'LEGACY' | 'UNIFIED', nullable — azure-openai: URL path resolution — AUTO / LEGACY / UNIFIED. Optional.
    - `prompts` AgentPromptConfig, required — Prompt configuration for the agent. The final system prompt is auto-assembled by ``build_system_prompt()`` from the agent type template + output schema + tool descriptions + custom_instructions. The ``system`` field stores the assembled result (set by the backend, not the user).
      - `system` string — Assembled system prompt (auto-generated by backend)
      - `customInstructions` string — User-provided instructions appended to the auto-generated prompt
      - `user` string — User prompt template - {input_json} is replaced with the input record
    - `mcpServer` AgentMcpServerConfig — Top-level MCP server config shared across all MCP tools. Two shapes (hard cutover): - **Streamkap MCP** (internal toolbelt): ``projectKeyId`` set, ``serverUrl`` and ``headers`` resolved server-side at deploy time from the agentic-enabled Project Key. The legacy "saved Streamkap MCP card" path (match-by-serverUrl against ``agent_connections.mcpConnections[source==streamkap]``) is gone. - **External MCP**: ``serverUrl`` + ``headers`` set inline; ``projectKeyId`` omitted. Headers are still composed at deploy time from the matching saved external connection (bearer / custom-header).
      - `projectKeyId` string, nullable — ID of an agentic-enabled Project Key. When set, deploy-time resolves the Streamkap MCP serverUrl + auth header from the PK row. Mutually exclusive with serverUrl - one or the other, never both.
      - `serverUrl` string, nullable — External MCP server URL. Required when projectKeyId is unset. Server-stamped from the PK row when projectKeyId is set.
      - `headers` object — Auth headers for MCP server
    - `tools` AgentToolConfig[] — Agent tools (MCP, HTTP, transform)
      - `name` string, required — Tool name (must be unique)
      - `type` 'http' | 'transform' | 'mcp', required
      - `description` string — Tool description for the LLM
      - `config` object — Type-specific configuration
      - `parameters` AgentToolParameterConfig[] — Tool parameters the LLM can provide
        - `name` string, required — Parameter name
        - `type` string — Parameter type (string, number, boolean)
        - `description` string — Parameter description for the LLM
    - `memory` AgentMemoryConfig — Memory configuration for the agent.
      - `keyField` string, nullable — Field to partition memory by (e.g. 'customer_id'). Default: topic key
      - `shortTerm` AgentShortTermMemoryConfig — Short-term (conversation) memory config.
        - `enabled` boolean — Enable short-term memory
        - `ttlMs` integer — Time-to-live in milliseconds (default: 1 hour)
        - `maxEntries` integer — Max conversation entries per key
      - `longTerm` AgentLongTermMemoryConfig — Long-term (vector store) memory config. References a saved ``AgentVectorStoreConnection`` by id. At deploy time the resolver reads the connection's ``apiKey`` + ``endpoint`` and composes the inline ``vectorStore`` block the Java runtime expects. Legacy agents may still carry ``destinationId`` (a Pinecone destination connector from the old flow). The deploy resolver handles both: if ``vectorStoreConnectionId`` is set it wins; otherwise it falls back to the legacy ``destinationId`` path. New agents created through the FE always use ``vectorStoreConnectionId``.
        - `enabled` boolean — Enable long-term memory
        - `vectorStoreConnectionId` string, nullable — Reference to a saved AgentVectorStoreConnection. Resolved at deploy time.
        - `namespace` string, nullable — Per-agent namespace override (falls back to connection's defaultNamespace)
        - `destinationId` string, nullable — (Legacy) ObjectId of a Pinecone destination connector. Prefer vectorStoreConnectionId.
    - `knowledgeBases` AgentKnowledgeBaseRef[] — Knowledge bases this agent can query at runtime for RAG
      - `id` string, required — Knowledge base entity ID
      - `name` string — Display name (denormalized for UI)
    - `processing` AgentProcessingConfig — Processing configuration for the agent.
      - `parallelism` integer — Parallelism
      - `checkpointIntervalMin` integer — Checkpoint interval in minutes
      - `maxIterations` integer — Max tool-call iterations per record (react type only)
      - `maxTokensPerHour` integer — Per-TaskManager rolling-hour token budget. 0 = unlimited. When exceeded, records pass through with a `_budget_exceeded: true` marker.
  - `sampleRecord` object, required — One Kafka-shaped record to feed the agent
  - `agentId` string, nullable — Resolves an empty / masked ``apiKey`` against the stored agent's encrypted blob. Post-cutover the wizard sends ``apiKey: ''`` for every existing-agent test-run; this lets the BE swap in the stored key for legacy inline-key agents that were saved before the Connection cutover (no ``llmConnectionId`` to resolve via).
  - `savedConnectionId` string, nullable — Mirror of the ValidateLlmRequest.savedConnectionId path.
  - `skipTools` boolean — When true, every MCP / HTTP tool dispatch returns a canned {skipped: true} payload to the LLM. Use to iterate on prompt shape without burning real tool calls.

## Response `200`

Successful Response

- TestRunResponse — Response body for ``POST /agents/test-run``.
  - `steps` TestRunStep[], required
    - `kind` 'llm' | 'tool', required
    - `model` string, nullable
    - `tokensIn` integer, nullable
    - `tokensOut` integer, nullable
    - `stopReason` string, nullable
    - `text` string, nullable
    - `toolCalls` object[], nullable
    - `toolName` string, nullable
    - `toolType` string, nullable
    - `request` object, nullable
    - `result` object, nullable
    - `skipped` boolean, nullable
    - `skippedReason` string, nullable
    - `durationMs` integer, nullable
    - `error` string, nullable
  - `finalOutput` unknown
  - `stoppedReason` 'completed' | 'max_iterations' | 'error' | 'timeout', required
  - `iterationsUsed` integer, required
  - `tokensIn` integer, required
  - `tokensOut` integer, required
  - `durationMs` integer, required
  - `approximations` string[] — Free-form list of caveats, e.g. 'memory_excluded', 'transforms_skipped'.
  - `suggestions` TestRunSuggestion[] — Deterministic hints (max 4) computed from the agent config + run result. Empty when the run produced a useful structured output. Each suggestion has a stable ``kind`` the FE can map to icon / copy.
    - `kind` 'missing_output_schema' | 'tools_not_invoked' | 'near_empty_response' | 'skip_instruction_may_apply' | 'max_tokens_truncated', required
    - `message` string, required
  - `warnings` TestRunWarning[] — Static config-vs-record checks (e.g. memory keyField missing on the sample) that the Flink runtime would silently swallow at deploy. Empty when nothing is amiss.
    - `kind` 'memory_key_field_missing' | 'input_fields_not_in_record' | 'max_tokens_clamped', required
    - `message` string, required
  - `outputSchemaCheck` TestRunSchemaCheck — Diff between the declared ``output.schema`` and the LLM's ``finalOutput``. Computed only when (a) the agent declared a schema and (b) the run produced a structured (non-text-envelope) final output. The Flink runtime DLQs records that fail this check at deploy time; surfacing the same comparison in test mode closes the audit's "silent output-schema bypass" gap.
    - `ok` boolean, required
    - `missing` string[] — Fields declared in schema but absent from finalOutput
    - `extra` string[] — Fields present in finalOutput but not declared
    - `typeMismatches` TestRunSchemaTypeMismatch[] — Fields whose value types diverge from the declared schema
      - `field` string, required
      - `expected` string, required — Declared schema type, e.g. STRING / DOUBLE
      - `actual` string, required — Inferred type of the value the LLM returned
  - `systemPrompt` string, nullable — Composed system prompt the LLM actually received - base template + auto-injected schema fields + tool descriptions + customInstructions. Identical to what Flink sends at deploy time (built by the same ``build_system_prompt``). Null when the run errored before reaching the LLM (e.g. ``llm_resolution_error``).
  - `userMessage` string, nullable — First user message dispatched to the LLM, after the ``prompts.user`` template wraps the (input-fields-filtered) sample record. Lets the test pane show the exact pair of strings the LLM saw on iteration 1. Null when the run errored before message construction.
  - `error` TestRunError — Structured error envelope on a failed test-run.
    - `kind` string, required — One of: llm_resolution_error, llm_provider_error, timeout, unsupported_provider, http_connection_resolution_error, …
    - `message` string, required

## Other responses

- `422` — Validation Error

---

[API](https://skmtc.net/streamkap/apis/streamkap-rest-api.md) · [All operations](https://skmtc.net/streamkap/apis/streamkap-rest-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/streamkap/streamkap-rest-api/versions/8aea6143d003/schema)