Sign in

v34

latestOpenAPI 3.1.0Apache-2.0raw.githubusercontent.com2026-08-0332109227.7 KB
inference

OpenAI-compatible chat completions. Strict allow-list parser: unknown top-level fields reject with 400 unsupported_field. Streaming (stream: true) is supported and emits SSE chat.completion.chunk events; n > 1 streaming fans candidates out as per-choice_index delta chunks with per-choice closure chunks (each carrying finish_reason) before the single global [DONE]. Content is either a string or an array of content parts; image parts (image_url / input_image) with a base64 data: URI are accepted for generation models declaring inputs.image (non-vision/encode-only models reject 400 unsupported_field; remote URLs reject 400 invalid_request). lora_adapter is forwarded to the worker as a top-level lora_path generation kwarg.

post/v1/chat/completions

Headers

X-SIE-MACHINE-PROFILEstring nullable

Preferred GPU or machine profile

X-SIE-Poolstring nullable

Explicit pool routing override

X-SIE-SDK-Versionstring nullable

Client SDK version for skew warnings

Request body

best_ofinteger nullable

Generate this many candidates and return the top n by cumulative logprob. Integer in [1, 128]. Cross-field rule: best_of >= n (otherwise 400 invalid_request). Rejected with 400 unsupported_field when stream: true (mirrors OpenAI).

{"stackTrail":"components:schemas:ChatCompletionRequest:properties:chat_template_kwargs","oasType":"schema","type":"unknown","description":"Per-request overrides for kwargs passed to the tokenizer's\n``apply_chat_template`` call. Object whose entries are merged on\ntop of the model YAML's ``chat_template_kwargs`` (the YAML wins\nfor keys present in both — adjust the YAML when a request needs\nto defeat a baked-in default). Typical use: pass\n``{\"enable_thinking\": false}`` to the Qwen3 family to suppress\n``<think>`` reasoning on a per-request basis. Absent → only the\nmodel YAML's defaults apply."}
frequency_penaltynumber float nullable

OpenAI frequency_penalty in [-2.0, 2.0]; out-of-range or non-numeric values yield 400 invalid_request.

logit_biasobject nullable

OpenAI logit_bias — {token_id_string: bias_float}. Keys must parse as integer token ids; values must be finite numbers in [-100.0, 100.0]. Map size capped at 1024 keys (request rejects with 400 invalid_request beyond the cap).

logprobsboolean nullable

Return per-token logprobs. Boolean. When true, the chosen token's logprob (and optionally a top-N list via top_logprobs) rides on each choices[].logprobs entry.

lora_adapterstring nullable

SIE extension: non-empty served-name of a LoRA adapter declared in the model profile's lora_paths. Absent → the base model. Unknown name → 400 with param:"lora_adapter". The gateway forwards it to the worker as a top-level lora_path generation kwarg (SGLang then selects the adapter by served name; the sampling-params placement is an SGLang implementation detail, not part of the SIE wire contract).

max_completion_tokensinteger nullable

Preferred output-token cap. Falls back to max_tokens when absent. When BOTH are omitted the gateway applies a default (1024, override via SIE_GATEWAY_DEFAULT_MAX_TOKENS) rather than rejecting — matching OpenAI, where this field is optional.

max_tokensinteger nullable

Legacy compatibility — max_completion_tokens wins when both are present. Optional; see max_completion_tokens for the behaviour when neither is supplied.

min_tokensinteger nullable

SGLang sampling_params.min_new_tokens: integer >= 0 capping how many tokens the model must emit before any stop condition can fire. Use to work around models that occasionally emit the stop token as the very first decoded token (e.g. Qwen3.6 thinking-off under greedy decode). Absent → sampler default (no minimum).

modelstring required
ninteger nullable

Number of candidate completions in [1, 128]. n>1 returns a multi-entry choices array; streaming with n>1 is supported (per-choice_index delta chunks + per-choice closure chunks ride before the single global [DONE]).

parallel_tool_callsboolean nullable

OpenAI parallel_tool_calls — boolean controlling whether the model may emit multiple tool calls per turn.

presence_penaltynumber float nullable

OpenAI presence_penalty in [-2.0, 2.0]; same validation as frequency_penalty.

prompt_cache_keystring nullable

Prompt-cache hint; plumbed onto the work envelope and ignored by the worker on the chat-completions surface.

repetition_penaltynumber float nullable

Non-OpenAI repetition_penalty: float in (0.0, 2.0] (1.0 = no penalty). Absent → sampler default.

routing_keystring nullable

Routing affinity hint; same plumbing as prompt_cache_key.

safety_identifierstring nullable

Accepted and silently ignored (never logged, never forwarded).

seedinteger nullable

Optional signed 64-bit per-request sampling seed. Reproducibility is best effort, not guaranteed, and depends on the active generation backend and deployment configuration. Non-integer or out-of-range values reject with 400 invalid_request.

streamboolean nullable

SSE streaming. When true, the response is a stream of chat.completion.chunk events terminated by data: [DONE]. For n > 1: per-choice_index delta chunks include a per-choice delta.role:"assistant" once per choice; per-choice closure chunks carry the finish_reason for that choice before the global [DONE]. Non-boolean values reject with 400 invalid_request.

temperaturenumber float nullable

Sampling temperature. Finite number >= 0; non-finite values reject.

top_kinteger nullable

Non-OpenAI top_k (Together / Fireworks / vLLM extension): integer >= 1. Absent → top-k disabled (model default).

top_logprobsinteger nullable

Number of alternative top tokens to return alongside each chosen token's logprob. Integer in [0, 20]. Requires logprobs: true when > 0 (OpenAI rule; 400 invalid_request otherwise).

top_pnumber float nullable

Nucleus sampling. Finite number in (0, 1].

userstring nullable

OpenAI user — Sensitive PII. Accepted-and-dropped: debug-logged only, never persisted, never forwarded to the worker.

Response

Chat completion response

createdinteger required

Epoch seconds.

idstring required

Always "chatcmpl-<request_id>".

modelstring required
objectstring required

Always "chat.completion" on the non-streaming endpoint.