v1
latestOpenAPI 3.1.02026-07-1754140.5 KBOpenAI-compatible chat endpoint
Headers
Request body
The model to use.
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.
Modify the likelihood of specified tokens appearing in the completion. Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect varies based on the model, but values between -1 and 1 should decrease or increase likelihood of selection; values at the ends of the range, such as -100 or 100 should result in a ban or exclusive selection of the relevant token.
Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message.
An integer specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.
The maximum number of tokens that can be generated.
How many completions to generate for each prompt.
Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.
Changing the seed produces a different response with similar characteristics. Fixing the seed reproduces the same results if all other parameters are also kept constant.
If set, partial message deltas are sent, like in ChatGPT. Tokens are sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE]
What sampling temperature to use, between 0 and 2. Higher values such as 0.8 make the output more random, while lower values such as 0.2 make it more focused and deterministic.
An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. NVIDIA recommends that you alter this option or temperature but not both.
This is currently unsupported. If true, the previous message is emitted again before the current message if they belong to the same role.
If true, special tokens (e.g. BOS) will be added to the prompt on top of what is added by the chat template. For most models, the chat template takes care of adding the special tokens so this should be set to false (as is the default).
A list of dicts representing documents that will be accessible to the model if it is performing RAG (retrieval-augmented generation). If the template does not support RAG, this argument will have no effect. We recommend that each document should be a dict containing "title" and "text" keys.
A Jinja template to use for this conversion. As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.
Additional kwargs to pass to the template renderer. Will be accessible by the chat template.
If specified, will override the default whitespace pattern for guided json decoding.
The image as a string or null.
Number of frames to sample from the video or stream. They will be the input to model.
The return format if there are media file outputs such as images or videos. none means to not return these files; base64 means to return them as base64 string in response content, such as The response content is... <img src="data:image/png;base64,{image_b64}" />; nvcf means to store them in NVCF large storage directory
Response
Successful Response