Create an LLM

Creates a connection to an external Large Language Model (LLM) for Retrieval Augmented Generation (RAG) and chat. You can connect OpenAI API-compatible models from providers like Anthropic, Azure, Google, or custom-hosted endpoints. After creation, reference your custom LLM by name in query generation parameters.

Connect external LLMs that use the OpenAI-compatible API format
Configure multiple LLM providers for different use cases
Override the platform's built-in LLMs with your own models
Use custom models for RAG, chat, and document summarization

Example providers:

OpenAI

Type: openai-compatible Models: GPT-4o, GPT-5 Auth: Bearer token

{
  "type": "openai-compatible",
  "name": "my-gpt5",
  "model": "gpt-5",
  "uri": "https://api.openai.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "sk-..."
  }
}

OpenAI Responses API

Type: openai-responses Models: o1-preview, o1-mini, o3-mini (reasoning models) Auth: Bearer token Note: For reasoning models that don't support streaming

{
  "type": "openai-responses",
  "name": "my-o1",
  "model": "o1-preview",
  "uri": "https://api.openai.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "sk-..."
  }
}

Anthropic Claude

Type: openai-compatible Models: claude-4-opus, claude-4-5-haiku, claude-4-5-sonnet Auth: Bearer token with header

{
  "type": "openai-compatible",
  "name": "my-claude",
  "model": "claude-sonnet-4-5-20250929",
  "uri": "https://api.anthropic.com/v1/messages",
  "auth": {
    "type": "bearer",
    "token": "sk-ant-..."
  },
  "headers": {
    "anthropic-version": "2023-06-01"
  }
}

Azure OpenAI

Type: openai-compatible Models: GPT-3.5, GPT-4 (Azure-deployed versions) Auth: Custom header (api-key)

{
  "type": "openai-compatible",
  "name": "my-azure-gpt4",
  "model": "gpt-4",
  "uri": "https://YOUR-RESOURCE.openai.azure.com/openai/deployments/YOUR-DEPLOYMENT/chat/completions?api-version=2024-02-15-preview",
  "auth": {
    "type": "header",
    "header": "api-key",
    "value": "your-azure-key"
  }
}

Google Vertex AI (Gemini) — Service Account

Type: vertex-ai Models: gemini-2.5-pro, gemini-2.5-flash Auth: Service account

{
  "type": "vertex-ai",
  "name": "my-gemini",
  "model": "gemini-2.5-flash",
  "uri": "https://us-central1-aiplatform.googleapis.com/v1/projects/YOUR-PROJECT/locations/us-central1",
  "auth": {
    "type": "service_account",
    "key_json": "{...service account JSON...}"
  }
}

Google AI Studio (Gemini) — API Key

Type: vertex-ai Models: gemini-2.5-pro, gemini-2.5-flash Auth: API key

{
  "type": "vertex-ai",
  "name": "my-gemini",
  "model": "gemini-2.5-flash",
  "uri": "https://generativelanguage.googleapis.com/v1beta",
  "auth": {
    "type": "api_key",
    "api_key": "your-google-api-key"
  }
}

The uri field is flexible — you can provide a base URI or a full URL copied from Google docs (including model path and :generateContent suffix). The platform normalizes it automatically.

Custom OpenAI-Compatible

Type: openai-compatible Models: Any self-hosted or custom LLM, such as OpenRouter. Auth: Bearer or custom header

{
  "type": "openai-compatible",
  "name": "my-custom-llm",
  "model": "llama-3-70b",
  "uri": "https://my-llm-endpoint.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "custom-token"
  }
}

Request-Timeoutinteger

The platform makes a best effort to complete the request in the specified seconds, or it times out.

Request-Timeout-Millisinteger

The platform makes a best effort to complete the request in the specified milliseconds, or it times out.

Request body

OR

Example request

{
  "name": "Claude 3.7 Sonnet",
  "description": "The Anthropic Claude 3.7 Sonnet model",
  "model": "claude-3-7-sonnet-20250219",
  "uri": "https://api.anthropic.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "abcdef......"
  },
  "idle_timeout_seconds": 300,
  "test_model_parameters": {
    "max_tokens": 512
  }
}

Response

The created LLM.

idstring required

The ID of the LLM.

namestring required

Name of the LLM.

descriptionstring

The description of the LLM.

enabledboolean

Indicates whether the LLM is enabled.

defaultboolean

Whether this is the default LLM. Queries use the default LLM when they do not specify a generator.

ownership'platform' | 'customer'

Indicates whether the LLM is provided by the platform or created by the customer. Platform LLMs are pre-configured and cannot be modified or deleted. Customer LLMs are created and managed by the customer.

type'openai-compatible' | 'openai-responses' | 'vertex-ai' | 'anthropic' required

The provider type. It determines which authentication and configuration apply. Responses always include this field. If a stored LLM cannot be mapped to one of these values, the request fails with HTTP 500.

modelstring

Provider-specific model identifier (e.g. gpt-4o, claude-3-5-sonnet-20241022, gemini-2.5-flash).

uristring uri

The API endpoint URI configured for this LLM.

headersobject

Additional HTTP headers configured for requests to the LLM API. Not applicable to vertex-ai.

idle_timeout_secondsinteger nullable

The maximum time in seconds that the platform waits for the model to send data before it closes the stale connection. During streaming, this is the SSE idle timeout. If no new server-sent events arrive within this window, the stream closes with an error. For non-streaming requests, where the model sends the entire response at once, this is the maximum time to wait for that response. If unset, the platform uses its default read timeout for the provider. On update, omit the field to keep the configured value, or send an explicit null to clear it.

requests_per_secondinteger

The maximum number of requests per second for this LLM. The platform omits this field when the LLM has no limit.

Example response

{
  "id": "llm_1021844",
  "name": "Claude 3.7 Sonnet",
  "description": "The Anthropic Claude 3.7 Sonnet model.",
  "ownership": "platform",
  "idle_timeout_seconds": 300,
  "auth": {
    "type": "bearer",
    "token": "abcdef......"
  }
}

v3