Sign in

v1

latestOpenAPI 3.1.02026-07-17122076.0 KB
Summarization

Summarize a video file

Run video file summarization. For stream summarization use POST /v1/generate_captions and POST /v1/stream_summarize instead.

post/v1/summarize

Request body

min_tokensinteger

Minimum number of tokens to generate before the model is allowed to stop. Used with ignore_eos for fixed-length generation.

ignore_eosboolean

If true, ignore end-of-sequence token and continue generating until max_tokens. Useful for benchmarking with fixed output length.

urlstring nullable

URL of the video to summarize

system_promptstring

System prompt for the VLM. To enable reasoning with Cosmos Reason1, add <think></think> and <answer></answer> tags to the system prompt.

promptstring

Prompt for summary generation

modelstring required

Model to use for this query.

max_tokensinteger

The maximum number of tokens to generate in any given call.

temperaturenumber

The sampling temperature to use for text generation. The higher the temperature value is, the less deterministic the output text will be.

top_pnumber

The top-p sampling mass used for text generation. The top-p value determines the probability mass that is sampled at sampling time.

top_knumber

The number of highest probability vocabulary tokens to keep for top-k-filtering

seedinteger

Seed value

chunk_durationinteger

Chunk videos into chunkDuration seconds. Set 0 for no chunking

chunk_overlap_durationinteger

Chunk Overlap Duration Time in Seconds. Set 0 for no overlap

summary_durationinteger

Summarize every summaryDuration seconds of the video. Applicable to live streams only.

num_frames_per_chunkinteger

DEPRECATED: Use num_frames_per_second_or_fixed_frames_chunk instead. Number of frames per chunk to use for the VLM.

vlm_input_widthinteger

VLM Input Width

vlm_input_heightinteger

VLM Input Height

enable_audioboolean

Enable transcription of the audio stream in the media

enable_reasoningboolean

Enable reasoning for VLM captions generation

num_frames_per_second_or_fixed_frames_chunknumber

Number of frames per second (if use_fps_for_chunking=true) or fixed number of frames per chunk (if use_fps_for_chunking=false).

use_fps_for_chunkingboolean

If true, use num_frames_per_second_or_fixed_frames_chunk as FPS. If false, use it as a fixed frame count per chunk.

creation_timestring

Creation time of the media in ISO 8601 format (e.g. 2024-06-09T18:32:11.123Z). If provided, offsets frame timestamps in the response.

alert_categorystring

Alert category for VLM captions (e.g. 'Worker PPE Violation').

mm_processor_kwargsobject

Additional keyword arguments for the multimodal processor (e.g., size, shortest_edge, longest_edge).

custom_metadataobject

Custom metadata to be added to the summarization request. This is a JSON object with key-value pairs. Custom metadata is supported only with user managed milvus db collections.

delete_external_collectionboolean

Delete the external collection at the end of the summarization request

schemastring

JSON schema for structured output extraction from video content

batch_response_methodstring

Method for batch response processing

scenariostring required

Scenario or use case context for the summarization

eventsstring[] required

List of events to detect or extract from the video

auto_generate_promptboolean

Enable automatic prompt generation based on schema and events

override_vlm_promptboolean

Override the VLM prompt with the user supplied prompt

enable_vlm_structured_outputboolean

Enable VLM structured output

objects_of_intereststring[]

List of objects of interest to detect or extract from the video

Example request

{
  "min_tokens": 100,
  "ignore_eos": true,
  "url": "https://www.example.com/video.mp4",
  "system_prompt": "You are a helpful assistant. Answer the user's question.",
  "prompt": "Write a concise and clear dense caption for the provided warehouse video",
  "model": "cosmos-reason1",
  "max_tokens": 512,
  "temperature": 0.2,
  "top_p": 1,
  "top_k": 100,
  "seed": 10,
  "chunk_duration": 60,
  "chunk_overlap_duration": 10,
  "summary_duration": 60,
  "media_info": {
    "end_offset": 4000000000
  },
  "num_frames_per_chunk": 10,
  "vlm_input_width": 256,
  "vlm_input_height": 256,
  "enable_audio": true,
  "enable_reasoning": true,
  "num_frames_per_second_or_fixed_frames_chunk": 1,
  "use_fps_for_chunking": true,
  "creation_time": "2024-06-09T18:32:11.123Z",
  "alert_category": "Worker PPE Violation",
  "mm_processor_kwargs": {
    "shortest_edge": 384
  },
  "schema": "{\"type\": \"object\", \"properties\": {\"events\": {\"type\": \"array\"}}}",
  "batch_response_method": "json_schema",
  "scenario": "warehouse",
  "events": [
    "fire",
    "theft",
    "accident"
  ],
  "auto_generate_prompt": true,
  "override_vlm_prompt": true,
  "enable_vlm_structured_output": true,
  "objects_of_interest": [
    "person",
    "car",
    "bicycle"
  ]
}

Response

Successful Response.

idstring uuid required

Unique ID for the query

video_idstring uuid required

Unique ID for the video

createdinteger required

The Unix timestamp (in seconds) of when the chat completion/summary request was created.

modelstring required

The model used for the chat completion/summarization.

object'chat.completion' | 'summarization.completion' | 'summarization.progressing' | 'vlm_captions.completion' | 'vlm_captions.progressing' required

Completion object type.

Example response

{
  "choices": [
    {
      "index": 1,
      "message": {
        "content": "Some summary of the video",
        "tool_calls": [
          {
            "alert": {
              "ntpTimestamp": "2024-05-30T01:41:25.000Z",
              "offset": 20
            }
          }
        ]
      }
    }
  ],
  "created": 1717405636,
  "model": "cosmos-reason1",
  "media_info": {
    "start_timestamp": "2024-05-30T01:41:25.000Z",
    "end_timestamp": "2024-05-30T02:14:51.000Z"
  },
  "usage": {
    "query_processing_time": 78,
    "total_chunks_processed": 10,
    "summary_tokens": 100,
    "aggregation_tokens": 100,
    "summary_requests": 10,
    "summary_latency": 1,
    "aggregation_latency": 1
  }
}