v1

latestOpenAPI 3.1.02026-07-1387366.9 KB

Chat Completions

Chat completions endpoint for Inflection hosted models.

⚠️ This endpoint is under active development to enable all Inflection model and config types.

post/v1/chat/completions

Request body

modelstring nullable

frequency_penaltynumber nullable

logit_biasobject nullable

logprobsboolean nullable

top_logprobsinteger nullable

max_tokensinteger nullable

max_completion_tokensinteger nullable

ninteger nullable

presence_penaltynumber nullable

seedinteger nullable

streamboolean nullable

temperaturenumber nullable

top_pnumber nullable

parallel_tool_callsboolean nullable

userstring nullable

best_ofinteger nullable

use_beam_searchboolean

top_kinteger nullable

min_pnumber nullable

repetition_penaltynumber nullable

length_penaltynumber

stop_token_idsinteger[] nullable

include_stop_str_in_outputboolean

ignore_eosboolean

min_tokensinteger

skip_special_tokensboolean

spaces_between_special_tokensboolean

truncate_prompt_tokensinteger nullable

prompt_logprobsinteger nullable

allowed_token_idsinteger[] nullable

bad_wordsstring[]

echoboolean

If true, the new message will be prepended with the last message if they belong to the same role.

add_generation_promptboolean

If true, the generation prompt will be added to the chat template. This is a parameter used by chat template in tokenizer config of the model.

continue_final_messageboolean

If this is set, the chat will be formatted so that the final message in the chat is open-ended, without any EOS tokens. The model will continue this message rather than starting a new one. This allows you to "prefill" part of the model's response for it. Cannot be used at the same time as add_generation_prompt.

add_special_tokensboolean

If true, special tokens (e.g. BOS) will be added to the prompt on top of what is added by the chat template. For most models, the chat template takes care of adding the special tokens so this should be set to false (as is the default).

documentsobject[] nullable

A list of dicts representing documents that will be accessible to the model if it is performing RAG (retrieval-augmented generation). If the template does not support RAG, this argument will have no effect. We recommend that each document should be a dict containing "title" and "text" keys.

chat_templatestring nullable

A Jinja template to use for this conversion. As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.

chat_template_kwargsobject nullable

Additional keyword args to pass to the template renderer. Will be accessible by the chat template.

mm_processor_kwargsobject nullable

Additional kwargs to pass to the HF processor.

guided_regexstring nullable

If specified, the output will follow the regex pattern.

guided_choicestring[] nullable

If specified, the output will be exactly one of the choices.

guided_grammarstring nullable

If specified, the output will follow the context free grammar.

structural_tagstring nullable

If specified, the output will follow the structural tag schema.

guided_decoding_backendstring nullable

If specified, will override the default guided decoding backend of the server for this specific request. If set, must be either 'outlines' / 'lm-format-enforcer'

guided_whitespace_patternstring nullable

If specified, will override the default whitespace pattern for guided json decoding.

guided_metaobject nullable

Specify custom arguments for your backend

priorityinteger

The priority of the request (lower means earlier handling; default: 0). Any priority other than 0 will raise an error if the served model does not use priority scheduling.

request_idstring

The request_id related to this request. If the caller does not set it, a random_uuid will be generated. This id is used through out the inference process and return in response.

return_tokens_as_token_idsboolean nullable

If specified with 'logprobs', tokens are represented as strings of the form 'token_id:{token_id}' so that tokens that are not JSON-encodable can be identified.

cache_saltstring nullable

If specified, the prefix cache will be salted with the provided string to prevent an attacker to guess prompts in multi-user environments. The salt should be random, protected from access by 3rd parties, and long enough to be unpredictable (e.g., 43 characters base64-encoded, corresponding to 256 bit). Not supported by vLLM engine V0.

kv_transfer_paramsobject nullable

KVTransfer parameters used for disaggregated serving.

vllm_xargsobject nullable

Additional request parameters with string or numeric values, used by custom extensions.

timeout_secnumber nullable

Response

Returns either the full response (application/json) or streams multiple ChatCompletionStreamResponse objects via text/event-stream. Each event in the stream is a JSON object matching the ChatCompletionStreamResponse schema, and each event is prefixed by data: (typical for Server-Sent Events).

idstring

object'chat.completion'

createdinteger

modelstring required

prompt_logprobsobject[] nullable

kv_transfer_paramsobject nullable

KVTransfer parameters.

system_fingerprintstring nullable

time_infoobject nullable