--- title: "Create Batch Completions" method: POST path: "/v1/llm/batch-completions" --- # Create Batch Completions `POST /v1/llm/batch-completions` ## Request body - CreateBatchCompletionsV1Request — Request object for batch completions. - `input_data_path` string, nullable — Path to the input file. The input file should be a JSON file of type List[CreateBatchCompletionsRequestContent]. - `output_data_path` string, required — Path to the output file. The output file will be a JSON file of type List[CompletionOutput]. - `labels` object — Labels to attach to the batch inference job. - `data_parallelism` integer, nullable — Number of replicas to run the batch inference. More replicas are slower to schedule but faster to inference. - `max_runtime_sec` integer, nullable — Maximum runtime of the batch inference in seconds. Default to one day. - `priority` string, nullable — Priority of the batch inference job. Default to None. - `tool_config` ToolConfig — Configuration for tool use. NOTE: this config is highly experimental and signature will change significantly in future iterations. - `name` string, required - `max_iterations` integer, nullable - `execution_timeout_seconds` integer, nullable - `should_retry_on_error` boolean, nullable - `cpus` union — CPUs to use for the batch inference. - string - integer - number - `gpus` integer, nullable — Number of GPUs to use for the batch inference. - `memory` union — Amount of memory to use for the batch inference. - string - integer - number - `gpu_type` 'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb' — Lists allowed GPU types for Launch. - `storage` union — Storage to use for the batch inference. - string - integer - number - `nodes_per_worker` integer, nullable — Number of nodes per worker for the batch inference. - `content` CreateBatchCompletionsV1RequestContent - `prompts` string[], required - `max_new_tokens` integer, required - `temperature` number, required - `stop_sequences` string[], nullable - `return_token_log_probs` boolean, nullable - `presence_penalty` number, nullable - `frequency_penalty` number, nullable - `top_k` integer, nullable - `top_p` number, nullable - `skip_special_tokens` boolean, nullable - `model_config` CreateBatchCompletionsV1ModelConfig, required - `max_model_len` integer, nullable — Model context length, If unspecified, will be automatically derived from the model config - `max_num_seqs` integer, nullable — Maximum number of sequences per iteration - `enforce_eager` boolean, nullable — Always use eager-mode PyTorch. If False, will use eager mode and CUDA graph in hybrid for maximal perforamnce and flexibility - `trust_remote_code` boolean, nullable — Whether to trust remote code from Hugging face hub. This is only applicable to models whose code is not supported natively by the transformers library (e.g. deepseek). Default to False. - `pipeline_parallel_size` integer, nullable — Number of pipeline stages. Default to None. - `tensor_parallel_size` integer, nullable — Number of tensor parallel replicas. Default to None. - `quantization` string, nullable — Method used to quantize the weights. If None, we first check the `quantization_config` attribute in the model config file. If that is None, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. - `disable_log_requests` boolean, nullable — Disable logging requests. Default to None. - `chat_template` string, nullable — A Jinja template to use for this endpoint. If not provided, will use the chat template from the checkpoint - `tool_call_parser` string, nullable — Tool call parser - `enable_auto_tool_choice` boolean, nullable — Enable auto tool choice - `load_format` string, nullable — The format of the model weights to load. * "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. * "pt" will load the weights in the pytorch bin format. * "safetensors" will load the weights in the safetensors format. * "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. * "dummy" will initialize the weights with random values, which is mainly for profiling. * "tensorizer" will load the weights using tensorizer from CoreWeave. See the Tensorize vLLM Model script in the Examples section for more information. * "bitsandbytes" will load the weights using bitsandbytes quantization. - `config_format` string, nullable — The config format which shall be loaded. Defaults to 'auto' which defaults to 'hf'. - `tokenizer_mode` string, nullable — Tokenizer mode. 'auto' will use the fast tokenizer ifavailable, 'slow' will always use the slow tokenizer, and'mistral' will always use the tokenizer from `mistral_common`. - `limit_mm_per_prompt` string, nullable — Maximum number of data instances per modality per prompt. Only applicable for multimodal models. - `max_num_batched_tokens` integer, nullable — Maximum number of batched tokens per iteration - `tokenizer` string, nullable — Name or path of the huggingface tokenizer to use. - `dtype` string, nullable — Data type for model weights and activations. The 'auto' option will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - `seed` integer, nullable — Random seed for the model. - `revision` string, nullable — The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. - `code_revision` string, nullable — The specific revision to use for the model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. - `rope_scaling` object, nullable — Dictionary containing the scaling configuration for the RoPE embeddings. When using this flag, don't update `max_position_embeddings` to the expected new maximum. - `tokenizer_revision` string, nullable — The specific tokenizer version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. - `quantization_param_path` string, nullable — Path to JSON file containing scaling factors. Used to load KV cache scaling factors into the model when KV cache type is FP8_E4M3 on ROCm (AMD GPU). In the future these will also be used to load activation and weight scaling factors when the model dtype is FP8_E4M3 on ROCm. - `max_seq_len_to_capture` integer, nullable — Maximum sequence len covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. Additionally for encoder-decoder models, if the sequence length of the encoder input is larger than this, we fall back to the eager mode. - `disable_sliding_window` boolean, nullable — Whether to disable sliding window. If True, we will disable the sliding window functionality of the model. If the model does not support sliding window, this argument is ignored. - `skip_tokenizer_init` boolean, nullable — If true, skip initialization of tokenizer and detokenizer. - `served_model_name` string, nullable — The model name used in metrics tag `model_name`, matches the model name exposed via the APIs. If multiple model names provided, the first name will be used. If not specified, the model name will be the same as `model`. - `override_neuron_config` object, nullable — Initialize non default neuron config or override default neuron config that are specific to Neuron devices, this argument will be used to configure the neuron config that can not be gathered from the vllm arguments. - `mm_processor_kwargs` object, nullable — Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. - `block_size` integer, nullable — Size of a cache block in number of tokens. - `gpu_memory_utilization` number, nullable — Fraction of GPU memory to use for the vLLM execution. - `swap_space` number, nullable — Size of the CPU swap space per GPU (in GiB). - `cache_dtype` string, nullable — Data type for kv cache storage. - `num_gpu_blocks_override` integer, nullable — Number of GPU blocks to use. This overrides the profiled num_gpu_blocks if specified. Does nothing if None. - `enable_prefix_caching` boolean, nullable — Enables automatic prefix caching. - `model` string, required — ID of the model to use. - `checkpoint_path` string, nullable — Path to the checkpoint to load the model from. - `num_shards` integer, nullable — Suggested number of shards to distribute the model. When not specified, will infer the number of shards based on model config. System may decide to use a different number than the given value. - `max_context_length` integer, nullable — Maximum context length to use for the model. Defaults to the max allowed by the model. Deprecated in favor of max_model_len. - `response_role` string, nullable — Role of the response in the conversation. Only supported in chat completions. - `labels` object — Labels to attach to the batch inference job. ## Response `200` Successful Response - CreateBatchCompletionsV1Response - `job_id` string, required ## Other responses - `422` — Validation Error --- [API](https://skmtc.net/scaleapi/apis/launch.md) · [All operations](https://skmtc.net/scaleapi/apis/launch/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/scaleapi/launch/versions/8333576dbe43/schema)