---
title: "Update Model Endpoint"
method: PUT
path: "/v1/llm/model-endpoints/{model_endpoint_name}"
---

# Update Model Endpoint

`PUT /v1/llm/model-endpoints/{model_endpoint_name}`

Updates an LLM endpoint for the current user.

## Path parameters

- `model_endpoint_name` string, required

## Request body

- union
  - UpdateVLLMModelEndpointRequest
    - `quantize` 'bitsandbytes' | 'awq'
    - `checkpoint_path` string, nullable
    - `post_inference_hooks` string[], nullable
    - `cpus` union
      - string
      - integer
      - number
    - `gpus` integer, nullable
    - `memory` union
      - string
      - integer
      - number
    - `gpu_type` 'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb' — Lists allowed GPU types for Launch.
    - `storage` union
      - string
      - integer
      - number
    - `nodes_per_worker` integer, nullable
    - `optimize_costs` boolean, nullable
    - `prewarm` boolean, nullable
    - `high_priority` boolean, nullable
    - `billing_tags` object, nullable
    - `default_callback_url` string, nullable
    - `default_callback_auth` union
      - CallbackBasicAuth
        - `kind` 'basic', required
        - `username` string, required
        - `password` string, required
      - CallbackmTLSAuth
        - `kind` 'mtls', required
        - `cert` string, required
        - `key` string, required
    - `public_inference` boolean, nullable
    - `task_expires_seconds` integer, nullable — For async endpoints, how long a task can wait in queue before expiring (in seconds).
    - `chat_template_override` string, nullable — A Jinja template to use for this endpoint. If not provided, will use the chat template from the checkpoint
    - `enable_startup_metrics` boolean, nullable — Enable startup metrics collection via OpenTelemetry. When enabled, emits traces and metrics for download, Python init, and vLLM init phases.
    - `queue_message_timeout_seconds` integer, nullable — For async endpoints, the queue message visibility/lock timeout in seconds. Controls how long a worker has to process a message before it becomes visible again (SQS VisibilityTimeout / ASB lock_duration). Note: Azure Service Bus has a maximum of 300 seconds; values above this will be clamped.
    - `model_name` string, nullable
    - `source` 'hugging_face'
    - `inference_framework` 'vllm'
    - `inference_framework_image_tag` string, nullable
    - `num_shards` integer, nullable
    - `metadata` object, nullable
    - `force_bundle_recreation` boolean, nullable
    - `min_workers` integer, nullable
    - `max_workers` integer, nullable
    - `per_worker` integer, nullable
    - `labels` object, nullable
    - `max_gpu_memory_utilization` number, nullable — Maximum GPU memory utilization for the batch inference. Default to 90%. Deprecated in favor of specifying this in VLLMModelConfig
    - `attention_backend` string, nullable — Attention backend to use for vLLM. Default to None.
    - `max_model_len` integer, nullable — Model context length, If unspecified, will be automatically derived from the model config
    - `max_num_seqs` integer, nullable — Maximum number of sequences per iteration
    - `enforce_eager` boolean, nullable — Always use eager-mode PyTorch. If False, will use eager mode and CUDA graph in hybrid for maximal perforamnce and flexibility
    - `trust_remote_code` boolean, nullable — Whether to trust remote code from Hugging face hub. This is only applicable to models whose code is not supported natively by the transformers library (e.g. deepseek). Default to False.
    - `pipeline_parallel_size` integer, nullable — Number of pipeline stages. Default to None.
    - `tensor_parallel_size` integer, nullable — Number of tensor parallel replicas. Default to None.
    - `quantization` string, nullable — Method used to quantize the weights. If None, we first check the `quantization_config` attribute in the model config file. If that is None, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights.
    - `disable_log_requests` boolean, nullable — Disable logging requests. Default to None.
    - `chat_template` string, nullable — A Jinja template to use for this endpoint. If not provided, will use the chat template from the checkpoint
    - `tool_call_parser` string, nullable — Tool call parser
    - `enable_auto_tool_choice` boolean, nullable — Enable auto tool choice
    - `load_format` string, nullable — The format of the model weights to load. * "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. * "pt" will load the weights in the pytorch bin format. * "safetensors" will load the weights in the safetensors format. * "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. * "dummy" will initialize the weights with random values, which is mainly for profiling. * "tensorizer" will load the weights using tensorizer from CoreWeave. See the Tensorize vLLM Model script in the Examples section for more information. * "bitsandbytes" will load the weights using bitsandbytes quantization.
    - `config_format` string, nullable — The config format which shall be loaded. Defaults to 'auto' which defaults to 'hf'.
    - `tokenizer_mode` string, nullable — Tokenizer mode. 'auto' will use the fast tokenizer ifavailable, 'slow' will always use the slow tokenizer, and'mistral' will always use the tokenizer from `mistral_common`.
    - `limit_mm_per_prompt` string, nullable — Maximum number of data instances per modality per prompt. Only applicable for multimodal models.
    - `max_num_batched_tokens` integer, nullable — Maximum number of batched tokens per iteration
    - `tokenizer` string, nullable — Name or path of the huggingface tokenizer to use.
    - `dtype` string, nullable — Data type for model weights and activations. The 'auto' option will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models.
    - `seed` integer, nullable — Random seed for reproducibility.
    - `revision` string, nullable — The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
    - `code_revision` string, nullable — The specific revision to use for the model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
    - `rope_scaling` object, nullable — Dictionary containing the scaling configuration for the RoPE embeddings. When using this flag, don't update `max_position_embeddings` to the expected new maximum.
    - `tokenizer_revision` string, nullable — The specific tokenizer version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
    - `quantization_param_path` string, nullable — Path to JSON file containing scaling factors. Used to load KV cache scaling factors into the model when KV cache type is FP8_E4M3 on ROCm (AMD GPU). In the future these will also be used to load activation and weight scaling factors when the model dtype is FP8_E4M3 on ROCm.
    - `max_seq_len_to_capture` integer, nullable — Maximum sequence len covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. Additionally for encoder-decoder models, if the sequence length of the encoder input is larger than this, we fall back to the eager mode.
    - `disable_sliding_window` boolean, nullable — Whether to disable sliding window. If True, we will disable the sliding window functionality of the model. If the model does not support sliding window, this argument is ignored.
    - `skip_tokenizer_init` boolean, nullable — If true, skip initialization of tokenizer and detokenizer.
    - `served_model_name` string, nullable — The model name used in metrics tag `model_name`, matches the model name exposed via the APIs. If multiple model names provided, the first name will be used. If not specified, the model name will be the same as `model`.
    - `override_neuron_config` object, nullable — Initialize non default neuron config or override default neuron config that are specific to Neuron devices, this argument will be used to configure the neuron config that can not be gathered from the vllm arguments.
    - `mm_processor_kwargs` object, nullable — Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor.
    - `block_size` integer, nullable — Size of a cache block in number of tokens.
    - `gpu_memory_utilization` number, nullable — Fraction of GPU memory to use for the vLLM execution.
    - `swap_space` number, nullable — Size of the CPU swap space per GPU (in GiB).
    - `cache_dtype` string, nullable — Data type for kv cache storage.
    - `num_gpu_blocks_override` integer, nullable — Number of GPU blocks to use. This overrides the profiled num_gpu_blocks if specified. Does nothing if None.
    - `enable_prefix_caching` boolean, nullable — Enables automatic prefix caching.
  - UpdateSGLangModelEndpointRequest
    - `quantize` 'bitsandbytes' | 'awq'
    - `checkpoint_path` string, nullable
    - `post_inference_hooks` string[], nullable
    - `cpus` union
      - string
      - integer
      - number
    - `gpus` integer, nullable
    - `memory` union
      - string
      - integer
      - number
    - `gpu_type` 'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb' — Lists allowed GPU types for Launch.
    - `storage` union
      - string
      - integer
      - number
    - `nodes_per_worker` integer, nullable
    - `optimize_costs` boolean, nullable
    - `prewarm` boolean, nullable
    - `high_priority` boolean, nullable
    - `billing_tags` object, nullable
    - `default_callback_url` string, nullable
    - `default_callback_auth` union
      - CallbackBasicAuth
        - `kind` 'basic', required
        - `username` string, required
        - `password` string, required
      - CallbackmTLSAuth
        - `kind` 'mtls', required
        - `cert` string, required
        - `key` string, required
    - `public_inference` boolean, nullable
    - `task_expires_seconds` integer, nullable — For async endpoints, how long a task can wait in queue before expiring (in seconds).
    - `chat_template_override` string, nullable — A Jinja template to use for this endpoint. If not provided, will use the chat template from the checkpoint
    - `enable_startup_metrics` boolean, nullable — Enable startup metrics collection via OpenTelemetry. When enabled, emits traces and metrics for download, Python init, and vLLM init phases.
    - `queue_message_timeout_seconds` integer, nullable — For async endpoints, the queue message visibility/lock timeout in seconds. Controls how long a worker has to process a message before it becomes visible again (SQS VisibilityTimeout / ASB lock_duration). Note: Azure Service Bus has a maximum of 300 seconds; values above this will be clamped.
    - `model_name` string, nullable
    - `source` 'hugging_face'
    - `inference_framework` 'sglang'
    - `inference_framework_image_tag` string, nullable
    - `num_shards` integer, nullable
    - `metadata` object, nullable
    - `force_bundle_recreation` boolean, nullable
    - `min_workers` integer, nullable
    - `max_workers` integer, nullable
    - `per_worker` integer, nullable
    - `labels` object, nullable
    - `trust_remote_code` boolean, nullable — Whether to trust remote code from Hugging face hub. This is only applicable to models whose code is not supported natively by the transformers library (e.g. deepseek). Default to False.
    - `tp_size` integer, nullable — The tensor parallel size.
    - `skip_tokenizer_init` boolean, nullable — If set, skip init tokenizer and pass input_ids in generate request
    - `load_format` string, nullable — The format of the model weights to load.
    - `dtype` string, nullable — Data type for model weights and activations.
    - `kv_cache_dtype` string, nullable — Data type for kv cache storage. "auto" will use model data type.
    - `quantization_param_path` string, nullable — Path to the JSON file containing the KV cache scaling factors.
    - `quantization` string, nullable — The quantization method.
    - `context_length` integer, nullable — The model's maximum context length.
    - `device` string, nullable — The device type.
    - `served_model_name` string, nullable — Override the model name returned by the v1/models endpoint in OpenAI API server.
    - `chat_template` string, nullable — The builtin chat template name or path of the chat template file.
    - `is_embedding` boolean, nullable — Whether to use a CausalLM as an embedding model.
    - `revision` string, nullable — The specific model version to use.
    - `mem_fraction_static` number, nullable — The fraction of the memory used for static allocation.
    - `max_running_requests` integer, nullable — The maximum number of running requests.
    - `max_total_tokens` integer, nullable — The maximum number of tokens in the memory pool.
    - `chunked_prefill_size` integer, nullable — The maximum number of tokens in a chunk for the chunked prefill.
    - `max_prefill_tokens` integer, nullable — The maximum number of tokens in a prefill batch.
    - `schedule_policy` string, nullable — The scheduling policy of the requests.
    - `schedule_conservativeness` number, nullable — How conservative the schedule policy is.
    - `cpu_offload_gb` integer, nullable — How many GBs of RAM to reserve for CPU offloading
    - `prefill_only_one_req` boolean, nullable — If true, we only prefill one request at one prefill batch
    - `stream_interval` integer, nullable — The interval for streaming in terms of the token length.
    - `random_seed` integer, nullable — The random seed.
    - `constrained_json_whitespace_pattern` string, nullable — Regex pattern for syntactic whitespaces allowed in JSON constrained output.
    - `watchdog_timeout` number, nullable — Set watchdog timeout in seconds.
    - `download_dir` string, nullable — Model download directory.
    - `base_gpu_id` integer, nullable — The base GPU ID to start allocating GPUs from.
    - `log_level` string, nullable — The logging level of all loggers.
    - `log_level_http` string, nullable — The logging level of HTTP server.
    - `log_requests` boolean, nullable — Log the inputs and outputs of all requests.
    - `show_time_cost` boolean, nullable — Show time cost of custom marks.
    - `enable_metrics` boolean, nullable — Enable log prometheus metrics.
    - `decode_log_interval` integer, nullable — The log interval of decode batch.
    - `api_key` string, nullable — Set API key of the server.
    - `file_storage_pth` string, nullable — The path of the file storage in backend.
    - `enable_cache_report` boolean, nullable — Return number of cached tokens in usage.prompt_tokens_details.
    - `data_parallel_size` integer, nullable — The data parallelism size.
    - `load_balance_method` string, nullable — The load balancing strategy for data parallelism.
    - `expert_parallel_size` integer, nullable — The expert parallelism size.
    - `dist_init_addr` string, nullable — The host address for initializing distributed backend.
    - `nnodes` integer, nullable — The number of nodes.
    - `node_rank` integer, nullable — The node rank.
    - `json_model_override_args` string, nullable — A dictionary in JSON string format used to override default model configurations.
    - `lora_paths` string[], nullable — The list of LoRA adapters.
    - `max_loras_per_batch` integer, nullable — Maximum number of adapters for a running batch.
    - `attention_backend` string, nullable — Choose the kernels for attention layers.
    - `sampling_backend` string, nullable — Choose the kernels for sampling layers.
    - `grammar_backend` string, nullable — Choose the backend for grammar-guided decoding.
    - `speculative_algorithm` string, nullable — Speculative algorithm.
    - `speculative_draft_model_path` string, nullable — The path of the draft model weights.
    - `speculative_num_steps` integer, nullable — The number of steps sampled from draft model in Speculative Decoding.
    - `speculative_num_draft_tokens` integer, nullable — The number of token sampled from draft model in Speculative Decoding.
    - `speculative_eagle_topk` integer, nullable — The number of token sampled from draft model in eagle2 each step.
    - `enable_double_sparsity` boolean, nullable — Enable double sparsity attention
    - `ds_channel_config_path` string, nullable — The path of the double sparsity channel config
    - `ds_heavy_channel_num` integer, nullable — The number of heavy channels in double sparsity attention
    - `ds_heavy_token_num` integer, nullable — The number of heavy tokens in double sparsity attention
    - `ds_heavy_channel_type` string, nullable — The type of heavy channels in double sparsity attention
    - `ds_sparse_decode_threshold` integer, nullable — The threshold for sparse decoding in double sparsity attention
    - `disable_radix_cache` boolean, nullable — Disable RadixAttention for prefix caching.
    - `disable_jump_forward` boolean, nullable — Disable jump-forward for grammar-guided decoding.
    - `disable_cuda_graph` boolean, nullable — Disable cuda graph.
    - `disable_cuda_graph_padding` boolean, nullable — Disable cuda graph when padding is needed.
    - `disable_outlines_disk_cache` boolean, nullable — Disable disk cache of outlines.
    - `disable_custom_all_reduce` boolean, nullable — Disable the custom all-reduce kernel.
    - `disable_mla` boolean, nullable — Disable Multi-head Latent Attention (MLA) for DeepSeek-V2.
    - `disable_overlap_schedule` boolean, nullable — Disable the overlap scheduler.
    - `enable_mixed_chunk` boolean, nullable — Enable mixing prefill and decode in a batch when using chunked prefill.
    - `enable_dp_attention` boolean, nullable — Enable data parallelism for attention and tensor parallelism for FFN.
    - `enable_ep_moe` boolean, nullable — Enable expert parallelism for moe.
    - `enable_torch_compile` boolean, nullable — Optimize the model with torch.compile.
    - `torch_compile_max_bs` integer, nullable — Set the maximum batch size when using torch compile.
    - `cuda_graph_max_bs` integer, nullable — Set the maximum batch size for cuda graph.
    - `cuda_graph_bs` integer[], nullable — Set the list of batch sizes for cuda graph.
    - `torchao_config` string, nullable — Optimize the model with torchao.
    - `enable_nan_detection` boolean, nullable — Enable the NaN detection for debugging purposes.
    - `enable_p2p_check` boolean, nullable — Enable P2P check for GPU access.
    - `triton_attention_reduce_in_fp32` boolean, nullable — Cast the intermediate attention results to fp32.
    - `triton_attention_num_kv_splits` integer, nullable — The number of KV splits in flash decoding Triton kernel.
    - `num_continuous_decode_steps` integer, nullable — Run multiple continuous decoding steps to reduce scheduling overhead.
    - `delete_ckpt_after_loading` boolean, nullable — Delete the model checkpoint after loading the model.
    - `enable_memory_saver` boolean, nullable — Allow saving memory using release_memory_occupation and resume_memory_occupation
    - `allow_auto_truncate` boolean, nullable — Allow automatically truncating requests that exceed the maximum input length.
    - `enable_custom_logit_processor` boolean, nullable — Enable users to pass custom logit processors to the server.
    - `tool_call_parser` string, nullable — Specify the parser for handling tool-call interactions.
    - `huggingface_repo` string, nullable — The Hugging Face repository ID.
  - UpdateDeepSpeedModelEndpointRequest
    - `quantize` 'bitsandbytes' | 'awq'
    - `checkpoint_path` string, nullable
    - `post_inference_hooks` string[], nullable
    - `cpus` union
      - string
      - integer
      - number
    - `gpus` integer, nullable
    - `memory` union
      - string
      - integer
      - number
    - `gpu_type` 'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb' — Lists allowed GPU types for Launch.
    - `storage` union
      - string
      - integer
      - number
    - `nodes_per_worker` integer, nullable
    - `optimize_costs` boolean, nullable
    - `prewarm` boolean, nullable
    - `high_priority` boolean, nullable
    - `billing_tags` object, nullable
    - `default_callback_url` string, nullable
    - `default_callback_auth` union
      - CallbackBasicAuth
        - `kind` 'basic', required
        - `username` string, required
        - `password` string, required
      - CallbackmTLSAuth
        - `kind` 'mtls', required
        - `cert` string, required
        - `key` string, required
    - `public_inference` boolean, nullable
    - `task_expires_seconds` integer, nullable — For async endpoints, how long a task can wait in queue before expiring (in seconds).
    - `chat_template_override` string, nullable — A Jinja template to use for this endpoint. If not provided, will use the chat template from the checkpoint
    - `enable_startup_metrics` boolean, nullable — Enable startup metrics collection via OpenTelemetry. When enabled, emits traces and metrics for download, Python init, and vLLM init phases.
    - `queue_message_timeout_seconds` integer, nullable — For async endpoints, the queue message visibility/lock timeout in seconds. Controls how long a worker has to process a message before it becomes visible again (SQS VisibilityTimeout / ASB lock_duration). Note: Azure Service Bus has a maximum of 300 seconds; values above this will be clamped.
    - `model_name` string, nullable
    - `source` 'hugging_face'
    - `inference_framework` 'deepspeed'
    - `inference_framework_image_tag` string, nullable
    - `num_shards` integer, nullable
    - `metadata` object, nullable
    - `force_bundle_recreation` boolean, nullable
    - `min_workers` integer, nullable
    - `max_workers` integer, nullable
    - `per_worker` integer, nullable
    - `labels` object, nullable
  - UpdateTextGenerationInferenceModelEndpointRequest
    - `quantize` 'bitsandbytes' | 'awq'
    - `checkpoint_path` string, nullable
    - `post_inference_hooks` string[], nullable
    - `cpus` union
      - string
      - integer
      - number
    - `gpus` integer, nullable
    - `memory` union
      - string
      - integer
      - number
    - `gpu_type` 'nvidia-tesla-t4' | 'nvidia-ampere-a10' | 'nvidia-ampere-a100' | 'nvidia-ampere-a100e' | 'nvidia-hopper-h100' | 'nvidia-hopper-h100-1g20gb' | 'nvidia-hopper-h100-3g40gb' — Lists allowed GPU types for Launch.
    - `storage` union
      - string
      - integer
      - number
    - `nodes_per_worker` integer, nullable
    - `optimize_costs` boolean, nullable
    - `prewarm` boolean, nullable
    - `high_priority` boolean, nullable
    - `billing_tags` object, nullable
    - `default_callback_url` string, nullable
    - `default_callback_auth` union
      - CallbackBasicAuth
        - `kind` 'basic', required
        - `username` string, required
        - `password` string, required
      - CallbackmTLSAuth
        - `kind` 'mtls', required
        - `cert` string, required
        - `key` string, required
    - `public_inference` boolean, nullable
    - `task_expires_seconds` integer, nullable — For async endpoints, how long a task can wait in queue before expiring (in seconds).
    - `chat_template_override` string, nullable — A Jinja template to use for this endpoint. If not provided, will use the chat template from the checkpoint
    - `enable_startup_metrics` boolean, nullable — Enable startup metrics collection via OpenTelemetry. When enabled, emits traces and metrics for download, Python init, and vLLM init phases.
    - `queue_message_timeout_seconds` integer, nullable — For async endpoints, the queue message visibility/lock timeout in seconds. Controls how long a worker has to process a message before it becomes visible again (SQS VisibilityTimeout / ASB lock_duration). Note: Azure Service Bus has a maximum of 300 seconds; values above this will be clamped.
    - `model_name` string, nullable
    - `source` 'hugging_face'
    - `inference_framework` 'text_generation_inference'
    - `inference_framework_image_tag` string, nullable
    - `num_shards` integer, nullable
    - `metadata` object, nullable
    - `force_bundle_recreation` boolean, nullable
    - `min_workers` integer, nullable
    - `max_workers` integer, nullable
    - `per_worker` integer, nullable
    - `labels` object, nullable

## Response `200`

Successful Response

- UpdateLLMModelEndpointV1Response
  - `endpoint_creation_task_id` string, required

## Other responses

- `422` — Validation Error

---

[API](https://skmtc.net/scaleapi/apis/launch.md) · [All operations](https://skmtc.net/scaleapi/apis/launch/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/scaleapi/launch/versions/8333576dbe43/schema)