---
title: "Create an LLM"
method: POST
path: "/v2/llms"
tags: ["Large Language Models"]
---

# Create an LLM

`POST /v2/llms`

Creates a connection to an external Large Language Model (LLM) for Retrieval Augmented Generation (RAG) and chat. You can connect OpenAI API-compatible models from providers like Anthropic, Azure, Google, or custom-hosted endpoints. After creation, reference your custom LLM by name in query generation parameters.
- Connect external LLMs that use the OpenAI-compatible API format
- Configure multiple LLM providers for different use cases
- Override the platform's built-in LLMs with your own models
- Use custom models for RAG, chat, and document summarization

**Example providers:**

### OpenAI

**Type:** `openai-compatible`
**Models:** GPT-4o, GPT-5
**Auth:** Bearer token

```json
{
  "type": "openai-compatible",
  "name": "my-gpt5",
  "model": "gpt-5",
  "uri": "https://api.openai.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "sk-..."
  }
}
```

### OpenAI Responses API

**Type**: openai-responses
**Models**: o1-preview, o1-mini, o3-mini (reasoning models)
**Auth**: Bearer token
**Note**: For reasoning models that don't support streaming

```json
{
  "type": "openai-responses",
  "name": "my-o1",
  "model": "o1-preview",
  "uri": "https://api.openai.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "sk-..."
  }
}
```

### Anthropic Claude

**Type:** `openai-compatible`
**Models:** claude-4-opus, claude-4-5-haiku, claude-4-5-sonnet
**Auth:** Bearer token with header

```json
{
  "type": "openai-compatible",
  "name": "my-claude",
  "model": "claude-sonnet-4-5-20250929",
  "uri": "https://api.anthropic.com/v1/messages",
  "auth": {
    "type": "bearer",
    "token": "sk-ant-..."
  },
  "headers": {
    "anthropic-version": "2023-06-01"
  }
}
```

### Azure OpenAI

**Type:** `openai-compatible`
**Models:** GPT-3.5, GPT-4 (Azure-deployed versions)
**Auth:** Custom header (api-key)

```json
{
  "type": "openai-compatible",
  "name": "my-azure-gpt4",
  "model": "gpt-4",
  "uri": "https://YOUR-RESOURCE.openai.azure.com/openai/deployments/YOUR-DEPLOYMENT/chat/completions?api-version=2024-02-15-preview",
  "auth": {
    "type": "header",
    "header": "api-key",
    "value": "your-azure-key"
  }
}
```

### Google Vertex AI (Gemini) — Service Account

**Type:** `vertex-ai`
**Models:** gemini-2.5-pro, gemini-2.5-flash
**Auth:** Service account

```json
{
  "type": "vertex-ai",
  "name": "my-gemini",
  "model": "gemini-2.5-flash",
  "uri": "https://us-central1-aiplatform.googleapis.com/v1/projects/YOUR-PROJECT/locations/us-central1",
  "auth": {
    "type": "service_account",
    "key_json": "{...service account JSON...}"
  }
}
```

### Google AI Studio (Gemini) — API Key

**Type:** `vertex-ai`
**Models:** gemini-2.5-pro, gemini-2.5-flash
**Auth:** API key

```json
{
  "type": "vertex-ai",
  "name": "my-gemini",
  "model": "gemini-2.5-flash",
  "uri": "https://generativelanguage.googleapis.com/v1beta",
  "auth": {
    "type": "api_key",
    "api_key": "your-google-api-key"
  }
}
```

The `uri` field is flexible — you can provide a base URI or a full URL copied from Google docs
(including model path and `:generateContent` suffix). The platform normalizes it automatically.

### Custom OpenAI-Compatible

**Type:** `openai-compatible`
**Models:** Any self-hosted or custom LLM, such as OpenRouter.
**Auth:** Bearer or custom header

```json
{
  "type": "openai-compatible",
  "name": "my-custom-llm",
  "model": "llama-3-70b",
  "uri": "https://my-llm-endpoint.com/v1/chat/completions",
  "auth": {
    "type": "bearer",
    "token": "custom-token"
  }
}
```

## Headers

- `Request-Timeout` integer
- `Request-Timeout-Millis` integer

## Request body

- union — Request to create a Large Language Model connection for generative capabilities.
  - object — Common fields for OpenAI-compatible and Responses API LLM requests.
    - `name` string, required — The name that references the LLM. Other endpoints (like query) use this name to select the LLM. If the name conflicts with a global LLM (an LLM that is preconfigured with the platform), this LLM overrides the global LLM for all usages.
    - `description` string — Description of the LLM.
    - `model` string, required — The model name to use with the API (e.g. gpt-4, claude-2, etc). The platform sends this name to the remote LLM provider.
    - `uri` string, uri, required — The URI endpoint for the API (can be OpenAI or any compatible API endpoint)
    - `auth` union — Authentication configuration for connecting to a remote service.
      - object — Bearer token authentication
        - `type` string, required — Must be "bearer" for bearer token auth
        - `token` string, required — The bearer token to use for authentication
      - object — Custom header-based authentication
        - `type` string, required — Must be "header" for header-based auth
        - `header` string, required — The header name to use (e.g. x-api-key)
        - `value` string, required — The header value to use
      - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
        - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
        - `client_id` string, required — The OAuth2 client ID.
        - `client_secret` string, required — The OAuth2 client secret.
        - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
        - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
    - `headers` object — Additional HTTP headers to include with requests to the LLM API.
    - `idle_timeout_seconds` integer, nullable — The maximum time in seconds that the platform waits for the model to send data before it closes the stale connection. During streaming, this is the SSE idle timeout. If no new server-sent events arrive within this window, the stream closes with an error. For non-streaming requests, where the model sends the entire response at once, this is the maximum time to wait for that response. If unset, the platform uses its default read timeout for the provider. On update, omit the field to keep the configured value, or send an explicit null to clear it.
    - `test_model_parameters` object — Any additional parameters that are required for the LLM during the test call.
    - `capabilities` LLMCapabilities — The capabilities of a Large Language Model. If you do not provide capabilities when you create an LLM, the platform infers them from the model name and provider type. Fields you provide explicitly override the inferred defaults.
      - `image_support` boolean — Whether the model supports image inputs.
      - `context_limit` integer — Maximum context window size in tokens.
      - `tool_calling` boolean — Whether the model supports tool/function calling.
      - `structured_outputs` boolean — Whether the model supports structured output generation.
      - `requires_role_alternation` boolean — Whether the model requires strict role alternation in conversations. When true, the platform groups consecutive messages of the same role together.
    - `requests_per_second` integer, nullable — The maximum number of requests per second for this LLM. Omit the field or set it to null to apply no limit. The platform rejects requests above the limit with HTTP 429.
    - `type` string, required — Must be "openai-compatible" for OpenAI and OpenAI-compatible APIs (like Anthropic Claude, Azure OpenAI, etc)
  - object — Common fields for OpenAI-compatible and Responses API LLM requests.
    - `name` string, required — The name that references the LLM. Other endpoints (like query) use this name to select the LLM. If the name conflicts with a global LLM (an LLM that is preconfigured with the platform), this LLM overrides the global LLM for all usages.
    - `description` string — Description of the LLM.
    - `model` string, required — The model name to use with the Responses API (e.g. o1-preview, o1-mini). The platform sends this name to the remote LLM provider.
    - `uri` string, uri, required — The URI endpoint for the API (can be OpenAI or any compatible API endpoint)
    - `auth` union — Authentication configuration for connecting to a remote service.
      - object — Bearer token authentication
        - `type` string, required — Must be "bearer" for bearer token auth
        - `token` string, required — The bearer token to use for authentication
      - object — Custom header-based authentication
        - `type` string, required — Must be "header" for header-based auth
        - `header` string, required — The header name to use (e.g. x-api-key)
        - `value` string, required — The header value to use
      - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
        - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
        - `client_id` string, required — The OAuth2 client ID.
        - `client_secret` string, required — The OAuth2 client secret.
        - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
        - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
    - `headers` object — Additional HTTP headers to include with requests to the LLM API.
    - `idle_timeout_seconds` integer, nullable — The maximum time in seconds that the platform waits for the model to send data before it closes the stale connection. During streaming, this is the SSE idle timeout. If no new server-sent events arrive within this window, the stream closes with an error. For non-streaming requests, where the model sends the entire response at once, this is the maximum time to wait for that response. If unset, the platform uses its default read timeout for the provider. On update, omit the field to keep the configured value, or send an explicit null to clear it.
    - `test_model_parameters` object — Any additional parameters that are required for the LLM during the test call.
    - `capabilities` LLMCapabilities — The capabilities of a Large Language Model. If you do not provide capabilities when you create an LLM, the platform infers them from the model name and provider type. Fields you provide explicitly override the inferred defaults.
      - `image_support` boolean — Whether the model supports image inputs.
      - `context_limit` integer — Maximum context window size in tokens.
      - `tool_calling` boolean — Whether the model supports tool/function calling.
      - `structured_outputs` boolean — Whether the model supports structured output generation.
      - `requires_role_alternation` boolean — Whether the model requires strict role alternation in conversations. When true, the platform groups consecutive messages of the same role together.
    - `requests_per_second` integer, nullable — The maximum number of requests per second for this LLM. Omit the field or set it to null to apply no limit. The platform rejects requests above the limit with HTTP 429.
    - `type` string, required — Must be "openai-responses" for OpenAI Responses API (reasoning models like o1, o3)
  - object — Request to create a Vertex AI Large Language Model connection for Gemini models.
    - `type` string, required — Must be "vertex-ai" for Google Cloud Vertex AI Gemini models
    - `name` string, required — The name that references the LLM. Other endpoints (like query) use this name to select the LLM. If the name conflicts with a global LLM (an LLM that is preconfigured with the platform), this LLM overrides the global LLM for all usages.
    - `description` string — Description of the LLM.
    - `model` string, required — The model name to use (e.g. gemini-2.5-flash, gemini-2.5-pro, gemini-2.0-experimental-1219, etc).
    - `uri` string, uri, required — The base URI for the Gemini API. You can provide URIs in various formats. The platform normalizes them automatically and strips any model path, method suffix, or query parameters. **Vertex AI** (for service account auth): Provide the project/location base URI. Example: `https://us-central1-aiplatform.googleapis.com/v1/projects/YOUR-PROJECT/locations/us-central1` **Google AI Studio** (for API key auth): Provide the Generative Language API base URI. Example: `https://generativelanguage.googleapis.com/v1beta` Full URIs copied from Google docs also work. The platform strips the model path and `:generateContent` suffix and rebuilds them from the `model` field.
    - `auth` union, required — Authentication configuration for Vertex AI
      - object — API key authentication for Vertex AI
        - `type` string, required — Must be "api_key" for API key authentication
        - `api_key` string, required — The Google Cloud API key for authentication
      - object — Service account authentication for Vertex AI
        - `type` string, required — Must be "service_account" for service account authentication
        - `key_json` string, required — The service account JSON key file contents as a string
    - `headers` object — Additional HTTP headers to include with requests to the Gemini API.
    - `idle_timeout_seconds` integer, nullable — The maximum time in seconds that the platform waits for the model to send data before it closes the stale connection. During streaming, this is the SSE idle timeout. If no new server-sent events arrive within this window, the stream closes with an error. For non-streaming requests, where the model sends the entire response at once, this is the maximum time to wait for that response. If unset, the platform uses its default read timeout for the provider. On update, omit the field to keep the configured value, or send an explicit null to clear it.
    - `test_model_parameters` object — Any additional parameters that are required for the LLM during the test call.
    - `capabilities` LLMCapabilities — The capabilities of a Large Language Model. If you do not provide capabilities when you create an LLM, the platform infers them from the model name and provider type. Fields you provide explicitly override the inferred defaults.
      - `image_support` boolean — Whether the model supports image inputs.
      - `context_limit` integer — Maximum context window size in tokens.
      - `tool_calling` boolean — Whether the model supports tool/function calling.
      - `structured_outputs` boolean — Whether the model supports structured output generation.
      - `requires_role_alternation` boolean — Whether the model requires strict role alternation in conversations. When true, the platform groups consecutive messages of the same role together.
    - `requests_per_second` integer, nullable — The maximum number of requests per second for this LLM. Omit the field or set it to null to apply no limit. The platform rejects requests above the limit with HTTP 429.
  - object — Request to create an Anthropic Large Language Model connection for Claude models (direct API, Bedrock, or Vertex).
    - `type` string, required — Must be "anthropic" for Anthropic Claude models
    - `name` string, required — The name that references the LLM. Other endpoints (like query) use this name to select the LLM. If the name conflicts with a global LLM (an LLM that is preconfigured with the platform), this LLM overrides the global LLM for all usages.
    - `description` string — Description of the LLM.
    - `model` string, required — The Claude model name to use (e.g. claude-3-5-sonnet-20241022, claude-3-opus-20240229, etc).
    - `uri` string, uri — An optional custom base URL for the Anthropic API. Defaults to https://api.anthropic.com when you use the direct Anthropic API (header auth with x-api-key). Not required with Bedrock or Vertex authentication because the platform determines the endpoint from the region or project configuration.
    - `auth` union, required — Authentication configuration for Anthropic LLM via direct Anthropic API (use RemoteAuth with header "x-api-key"), AWS Bedrock, or GCP Vertex AI Model Garden
      - object — Bearer token authentication
        - `type` string, required — Must be "bearer" for bearer token auth
        - `token` string, required — The bearer token to use for authentication
      - object — Custom header-based authentication
        - `type` string, required — Must be "header" for header-based auth
        - `header` string, required — The header name to use (e.g. x-api-key)
        - `value` string, required — The header value to use
      - object — AWS Bedrock authentication with explicit IAM credentials
        - `type` string, required — Must be "bedrock_static_iam" for static AWS IAM credentials
        - `aws_access_key_id` string, required — AWS access key ID
        - `aws_secret_access_key` string, required — AWS secret access key
        - `region` string, required — AWS region for Bedrock
      - object — AWS Bedrock API key authentication
        - `type` string, required — Must be "bedrock_api_key" for Bedrock API key auth
        - `api_key` string, required — The Bedrock API key
        - `region` string, required — AWS region for Bedrock
      - object — Google Cloud Vertex AI service account authentication
        - `type` string, required — Must be "vertex_service_account" for service account auth
        - `key_json` string, required — The service account key JSON
        - `project` string, required — GCP project ID
        - `region` string, required — GCP region for Vertex AI
      - object — Google Cloud Vertex AI access token authentication
        - `type` string, required — Must be "vertex_access_token" for access token auth
        - `access_token` string, required — The GCP access token
        - `project` string, required — GCP project ID
        - `region` string, required — GCP region for Vertex AI
    - `headers` object — Optional additional headers to send with the request
    - `idle_timeout_seconds` integer, nullable — The maximum time in seconds that the platform waits for the model to send data before it closes the stale connection. During streaming, this is the SSE idle timeout. If no new server-sent events arrive within this window, the stream closes with an error. For non-streaming requests, where the model sends the entire response at once, this is the maximum time to wait for that response. If unset, the platform uses its default read timeout for the provider. On update, omit the field to keep the configured value, or send an explicit null to clear it.
    - `test_model_parameters` object — Any additional parameters that are required for the LLM during the test call.
    - `capabilities` LLMCapabilities — The capabilities of a Large Language Model. If you do not provide capabilities when you create an LLM, the platform infers them from the model name and provider type. Fields you provide explicitly override the inferred defaults.
      - `image_support` boolean — Whether the model supports image inputs.
      - `context_limit` integer — Maximum context window size in tokens.
      - `tool_calling` boolean — Whether the model supports tool/function calling.
      - `structured_outputs` boolean — Whether the model supports structured output generation.
      - `requires_role_alternation` boolean — Whether the model requires strict role alternation in conversations. When true, the platform groups consecutive messages of the same role together.
    - `requests_per_second` integer, nullable — The maximum number of requests per second for this LLM. Omit the field or set it to null to apply no limit. The platform rejects requests above the limit with HTTP 429.

## Response `201`

The created LLM.

- LLM — A Large Language Model. An LLM enhances query results with a generated response and acts as the responder during a chat.
  - `id` string, required — The ID of the LLM.
  - `name` string, required — Name of the LLM.
  - `description` string — The description of the LLM.
  - `enabled` boolean — Indicates whether the LLM is enabled.
  - `default` boolean — Whether this is the default LLM. Queries use the default LLM when they do not specify a generator.
  - `capabilities` LLMCapabilities — The capabilities of a Large Language Model. If you do not provide capabilities when you create an LLM, the platform infers them from the model name and provider type. Fields you provide explicitly override the inferred defaults.
    - `image_support` boolean — Whether the model supports image inputs.
    - `context_limit` integer — Maximum context window size in tokens.
    - `tool_calling` boolean — Whether the model supports tool/function calling.
    - `structured_outputs` boolean — Whether the model supports structured output generation.
    - `requires_role_alternation` boolean — Whether the model requires strict role alternation in conversations. When true, the platform groups consecutive messages of the same role together.
  - `ownership` 'platform' | 'customer' — Indicates whether the LLM is provided by the platform or created by the customer. Platform LLMs are pre-configured and cannot be modified or deleted. Customer LLMs are created and managed by the customer.
  - `type` 'openai-compatible' | 'openai-responses' | 'vertex-ai' | 'anthropic', required — The provider type. It determines which authentication and configuration apply. Responses always include this field. If a stored LLM cannot be mapped to one of these values, the request fails with HTTP 500.
  - `model` string — Provider-specific model identifier (e.g. `gpt-4o`, `claude-3-5-sonnet-20241022`, `gemini-2.5-flash`).
  - `uri` string, uri — The API endpoint URI configured for this LLM.
  - `headers` object — Additional HTTP headers configured for requests to the LLM API. Not applicable to `vertex-ai`.
  - `idle_timeout_seconds` integer, nullable — The maximum time in seconds that the platform waits for the model to send data before it closes the stale connection. During streaming, this is the SSE idle timeout. If no new server-sent events arrive within this window, the stream closes with an error. For non-streaming requests, where the model sends the entire response at once, this is the maximum time to wait for that response. If unset, the platform uses its default read timeout for the provider. On update, omit the field to keep the configured value, or send an explicit null to clear it.
  - `requests_per_second` integer — The maximum number of requests per second for this LLM. The platform omits this field when the LLM has no limit.
  - `auth` union — The authentication configuration for an LLM. It is a union over every variant any LLM provider accepts. In `GET` responses, secret fields contain the literal string `****`.
    - object — Bearer token authentication
      - `type` string, required — Must be "bearer" for bearer token auth
      - `token` string, required — The bearer token to use for authentication
    - object — Custom header-based authentication
      - `type` string, required — Must be "header" for header-based auth
      - `header` string, required — The header name to use (e.g. x-api-key)
      - `value` string, required — The header value to use
    - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
      - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
      - `client_id` string, required — The OAuth2 client ID.
      - `client_secret` string, required — The OAuth2 client secret.
      - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
      - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
    - object — AWS Bedrock authentication with explicit IAM credentials
      - `type` string, required — Must be "bedrock_static_iam" for static AWS IAM credentials
      - `aws_access_key_id` string, required — AWS access key ID
      - `aws_secret_access_key` string, required — AWS secret access key
      - `region` string, required — AWS region for Bedrock
    - object — AWS Bedrock API key authentication
      - `type` string, required — Must be "bedrock_api_key" for Bedrock API key auth
      - `api_key` string, required — The Bedrock API key
      - `region` string, required — AWS region for Bedrock
    - object — Google Cloud Vertex AI service account authentication
      - `type` string, required — Must be "vertex_service_account" for service account auth
      - `key_json` string, required — The service account key JSON
      - `project` string, required — GCP project ID
      - `region` string, required — GCP region for Vertex AI
    - object — Google Cloud Vertex AI access token authentication
      - `type` string, required — Must be "vertex_access_token" for access token auth
      - `access_token` string, required — The GCP access token
      - `project` string, required — GCP project ID
      - `region` string, required — GCP region for Vertex AI
    - object — API key authentication for Vertex AI
      - `type` string, required — Must be "api_key" for API key authentication
      - `api_key` string, required — The Google Cloud API key for authentication
    - object — Service account authentication for Vertex AI
      - `type` string, required — Must be "service_account" for service account authentication
      - `key_json` string, required — The service account JSON key file contents as a string
  - `prompts` Prompt[] — List of prompts that the model can use. This is deprecated; see `/v2/generation_presets` instead.
    - `id` string, required — The ID of the prompt.
    - `name` string, required — Name of the prompt. This is used as the `prompt_name` in a query.
    - `description` string — The description of the prompt.
    - `enabled` boolean — Indicates whether the prompt is enabled.
    - `default` boolean — Indicates if this prompt is the default prompt used with the LLM.

## Other responses

- `400` — Invalid request body
- `403` — Permissions do not allow creating an LLM

---

[API](https://skmtc.net/vectara/apis/vectara-rest-api-v2.md) · [All operations](https://skmtc.net/vectara/apis/vectara-rest-api-v2/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/vectara/vectara-rest-api-v2/revisions/a95087fe3a20/schema)