Sign in

v1

latestOpenAPI 3.1.0Apache-2.02026-07-1783444.0 KB
chat

Create chat completion

This handler performs several key operations:

  1. Forwards the chat completion request to the inference service
  2. Signs the response using the node's keystore
  3. Tracks token usage for the stack

Arguments

  • Extension((stack_small_id, estimated_total_tokens)) - Stack ID and estimated tokens count from middleware
  • state - Application state containing the inference client and keystore
  • payload - The chat completion request body

Returns

Returns a JSON response containing:

  • The inference service's response
  • A cryptographic signature of the response

Errors

Returns a AtomaServiceError::InternalError if:

  • The inference service request fails
  • Response parsing fails
  • Response signing fails
  • Token usage update fails
post/v1/chat/completions

Request body

frequency_penaltynumber float nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far

{"stackTrail":"components:schemas:ChatCompletionRequest:properties:function_call","oasType":"schema","type":"unknown","description":"Controls how the model responds to function calls"}
logit_biasobject nullable

Modify the likelihood of specified tokens appearing in the completion.

Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token.

max_completion_tokensinteger nullable

The maximum number of tokens to generate in the chat completion

max_tokensinteger nullable

The maximum number of tokens to generate in the chat completion

modelstring required

ID of the model to use

ninteger nullable

How many chat completion choices to generate for each input message

parallel_tool_callsboolean nullable

Whether to enable parallel tool calls.

presence_penaltynumber float nullable

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far

seedinteger nullable

If specified, our system will make a best effort to sample deterministically

service_tierstring nullable

Specifies the latency tier to use for processing the request. This parameter is relevant for customers subscribed to the scale tier service:

If set to 'auto', and the Project is Scale tier enabled, the system will utilize scale tier credits until they are exhausted. If set to 'auto', and the Project is not Scale tier enabled, the request will be processed using the default service tier with a lower uptime SLA and no latency guarantee. If set to 'default', the request will be processed using the default service tier with a lower uptime SLA and no latency guarantee. When not set, the default behavior is 'auto'.

stopstring[] nullable

Up to 4 sequences where the API will stop generating further tokens

streamboolean nullable

Whether to stream back partial progress

temperaturenumber float nullable

What sampling temperature to use, between 0 and 2

top_logprobsinteger nullable

An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

top_pnumber float nullable

An alternative to sampling with temperature

userstring nullable

A unique identifier representing your end-user

Example request

{
  "max_completion_tokens": 4096,
  "max_tokens": 4096,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "n": 1,
  "seed": 123,
  "temperature": 0.7,
  "top_logprobs": 1,
  "top_p": 1,
  "user": "user-1234"
}

Response

Chat completion successful

createdinteger required

The Unix timestamp (in seconds) of when the chat completion was created.

idstring required

A unique identifier for the chat completion.

modelstring required

The model used for the chat completion.

objectstring required

The object of the chat completion.

service_tierstring nullable

The service tier of the chat completion.

system_fingerprintstring nullable

The system fingerprint for the completion, if applicable.