Sign in

v29

latestOpenAPI 3.1.0raw.githubusercontent.com2026-03-233765161.7 KB
v1 modals

Unified Contextual Memory Search (Text + Multimodal)

Retrieves relevant memories based on text conversation context and/or multimodal inputs.

You can provide **text messages**, **images**, **video**, **audio**, or any combination.
The system finds memories semantically relevant to the provided inputs.

**Universal Base Params:**
- user_id (str, required): Restrict search to this user
- project_id (str, required): Restrict search to this project
- persona_id (str, optional): Use persona's context if provided

**Input — at least one required:**
- messages (array, optional): Text conversation context
- video_base64 (str, optional): Base64 encoded video
- image_base64 (str, optional): Base64 encoded image
- audio_base64 (str, optional): Base64 encoded audio

**Search Config:**
- include_modalities (array, optional): Filter results by type: ["text", "image", "video"]

**Response:**
- new_prompt (str): Enhanced prompt with retrieved memory context
- raw_results (dict): Structured memory data from retrieval
- entity_images (dict, optional): Reference images for matched entities
- success (bool): True if query succeeded

Returns 200 OK with memory data. Requires authentication.
post/v1/modal/query

Request body

user_idstring required

Restrict search to this user (required)

project_idstring nullable

Restrict search to this project (required)

persona_idstring nullable

Optional persona ID. If provided, query uses persona's context

text_inputstring nullable

Text input to search against. The system finds memories relevant to this text.

video_base64string nullable

Base64 encoded video content

image_base64string nullable

Base64 encoded image content

audio_base64string nullable

Base64 encoded audio content (supports webm, wav, mp3, mp4, and other formats)

include_modalitiesstring[] nullable

Filter results by type: ['text', 'image', 'video']

session_idstring nullable

Optional session identifier for conversation context

Example request

{
  "include_modalities": [
    "text",
    "video"
  ],
  "messages": [
    {
      "content": "What did we decide about the animation speed last time?",
      "role": "user"
    }
  ],
  "project_id": "proj_ABC",
  "user_id": "user_123"
}

Response

Successful Response

new_promptstring required

Enhanced prompt with retrieved memory context

raw_resultsobject

Raw results from the retrieval process

entity_imagesobject nullable

Reference images for matched entities (entity_name -> base64 image)

successboolean

Whether the query was processed successfully

Example response

{
  "new_prompt": "User preferences: likes morning coffee...",
  "raw_results": {
    "episodic": [],
    "identity": [],
    "preferences": []
  },
  "success": true
}