Sign in

v1

latestOpenAPI 3.1.02026-07-17122076.0 KB
Stream

Start stream captioning

Fire-and-forget: kicks off VLM captioning on RTVI for a stream that was previously added via RTVI stream/add. Returns immediately once RTVI acknowledges the request.

post/v1/generate_captions

Request body

idstring uuid required

Stream ID (from RTVI stream/add).

modelstring required

Model to use for caption generation.

promptstring

VLM prompt for caption generation.

system_promptstring

System prompt for the VLM.

chunk_durationinteger

Chunk videos into chunkDuration seconds. 0 for no chunking.

chunk_overlap_durationinteger

Chunk overlap duration in seconds. 0 for no overlap.

num_frames_per_second_or_fixed_frames_chunknumber nullable

Number of frames per second or fixed frames per chunk.

use_fps_for_chunkingboolean

Use FPS for chunking instead of fixed frame count.

max_tokensinteger nullable

Maximum number of tokens to generate per chunk.

temperaturenumber nullable

Sampling temperature for VLM text generation.

top_pnumber nullable

Top-p sampling mass for VLM text generation.

top_knumber nullable

Top-k filtering for VLM text generation.

seedinteger nullable

Seed value for reproducibility.

enable_reasoningboolean

Enable reasoning for VLM captions generation.

enable_audioboolean

Enable audio ASR alongside video captioning.

vlm_input_widthinteger

VLM input width (0 = model default).

vlm_input_heightinteger

VLM input height (0 = model default).

mm_processor_kwargsobject nullable

Optional multimodal processor kwargs.

alert_categorystring nullable

Alert category for structured captioning.

creation_timestring nullable

ISO 8601 creation timestamp for the stream.

scenariostring

Scenario description for auto-prompt generation.

eventsstring[]

Event types for auto-prompt generation.

objects_of_intereststring[]

Objects of interest for auto-prompt generation.

enable_vlm_structured_outputboolean

Enable structured VLM output (controls auto-prompt format).

override_vlm_promptboolean

If true, use prompt as-is instead of auto-generating from scenario/events.

Example request

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "model": "cosmos-reason1",
  "prompt": "Write a concise and clear dense caption for the provided video",
  "chunk_duration": 60,
  "chunk_overlap_duration": 10,
  "max_tokens": 512,
  "temperature": 0.2,
  "top_p": 1,
  "top_k": 100,
  "seed": 10,
  "vlm_input_width": 256,
  "vlm_input_height": 256
}

Response

Captioning started.

idstring required

Stream ID that captioning was started for.

statusstring required

Status of the captioning request.

modelstring required

Model used for caption generation.

Example response

{
  "status": "accepted"
}