Sign in

v1

latestOpenAPI 3.1.02026-07-1766222595.7 KB

Batch Completions

post/v2/batch-completions

Request body

input_data_pathstring nullable

Path to the input file. The input file should be a JSON file of type List[CreateBatchCompletionsRequestContent].

output_data_pathstring required

Path to the output file. The output file will be a JSON file of type List[CompletionOutput].

labelsobject

Labels to attach to the batch inference job.

data_parallelisminteger nullable

Number of replicas to run the batch inference. More replicas are slower to schedule but faster to inference.

max_runtime_secinteger nullable

Maximum runtime of the batch inference in seconds. Default to one day.

prioritystring nullable

Priority of the batch inference job. Default to None.

gpusinteger nullable

Number of GPUs to use for the batch inference.

gpu_type'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb'

Lists allowed GPU types for Launch.

nodes_per_workerinteger nullable

Number of nodes per worker for the batch inference.

Example request

{
  "content": [
    {
      "max_tokens": 16,
      "n": 1,
      "suffix": "test.",
      "temperature": 1,
      "top_p": 1,
      "user": "user-1234"
    }
  ],
  "model_config": {
    "model": "mixtral-8x7b-instruct"
  }
}

Response

Successful Response

job_idstring required
input_data_pathstring nullable

Path to the input file. The input file should be a JSON file of type List[CreateBatchCompletionsRequestContent].

output_data_pathstring required

Path to the output file. The output file will be a JSON file of type List[CompletionOutput].

prioritystring nullable

Priority of the batch inference job. Default to None.

status'queued' | 'running' | 'completed' | 'failed' | 'cancelled' | 'unknown' required
created_atstring required
expires_atstring required
completed_atstring nullable required
metadataobject nullable required

Example response

{
  "model_config": {
    "model": "mixtral-8x7b-instruct"
  }
}