v1

latestOpenAPI 3.1.02026-07-1766222595.7 KB

Create Batch Completions

post/v1/llm/batch-completions

input_data_pathstring nullable

Path to the input file. The input file should be a JSON file of type List[CreateBatchCompletionsRequestContent].

output_data_pathstring required

Path to the output file. The output file will be a JSON file of type List[CompletionOutput].

labelsobject

Labels to attach to the batch inference job.

data_parallelisminteger nullable

Number of replicas to run the batch inference. More replicas are slower to schedule but faster to inference.

max_runtime_secinteger nullable

Maximum runtime of the batch inference in seconds. Default to one day.

prioritystring nullable

Priority of the batch inference job. Default to None.

gpusinteger nullable

Number of GPUs to use for the batch inference.

Lists allowed GPU types for Launch.

nodes_per_workerinteger nullable

Number of nodes per worker for the batch inference.

{
  "model_config": {
    "model": "mixtral-8x7b-instruct"
  }
}

Successful Response

job_idstring required