--- title: "Update Storage Connection" method: PATCH path: "/v1/organizations/connections/{connection_identifier}" tags: ["Organization Connections"] --- # Update Storage Connection `PATCH /v1/organizations/connections/{connection_identifier}` Update connection metadata or credentials. Allows partial updates to connection metadata without changing credentials. Credentials can be updated via provider_config. All changes are logged in audit trail. **What You Can Update:** - Connection name and description - Metadata tags - Status (active/suspended) - Provider credentials (via provider_config) **Example:** ```bash curl -X PATCH "http://localhost:8000/v1/organizations/connections/conn_abc123" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Updated Drive Name", "status": "suspended" }' ``` ## Path parameters - `connection_identifier` string, required — Connection identifier - either connection ID (conn_...) or name. The system will automatically resolve names to IDs. ## Request body - StorageConnectionUpdateRequest — Request payload for updating storage connection metadata. Allows partial updates to connection metadata without changing credentials. Credentials can be updated via provider_config. **What You Can Update:** - Connection name and description - Metadata tags - Status (active/suspended) - Provider credentials (via provider_config) **Examples:** ```python # Update name and description { "name": "Updated Drive Name", "description": "New description" } # Suspend connection { "status": "suspended", "is_active": False } # Refresh credentials { "provider_config": { "credentials": {...} } } ``` - `name` string, nullable — OPTIONAL. New name for the connection. Must be unique within the organization if provided. Format: 1-100 characters. - `description` string, nullable — OPTIONAL. New description for the connection. Set to empty string to clear existing description. Format: Up to 500 characters. - `metadata` object, nullable — OPTIONAL. New metadata dictionary. Replaces existing metadata entirely (partial updates not supported). Set to empty dict {} to clear all metadata. - `status` 'PENDING' | 'QUEUED' | 'IN_PROGRESS' | 'PROCESSING' | 'COMPLETED' | 'COMPLETED_WITH_ERRORS' | 'FAILED' | 'CANCELED' | 'INTERRUPTED' | 'UNKNOWN' | 'SKIPPED' | 'DRAFT' | 'ACTIVE' | 'ARCHIVED' | 'SUSPENDED' | 'DEACTIVATED' — Enumeration of task statuses for tracking asynchronous operations. Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution. Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces) Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted ACTIVE: Entity is active and operational (for buckets, collections, etc.) ARCHIVED: Entity has been archived SUSPENDED: Entity has been temporarily suspended Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state. Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately. Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling - `is_active` boolean, nullable — OPTIONAL. Quick boolean flag for filtering. True when status is ACTIVE, False otherwise. Automatically maintained when status changes. - `provider_config` object, nullable — OPTIONAL. Updated provider configuration including credentials. Replaces entire provider_config (partial updates not supported). SECURITY: Sensitive fields are encrypted at rest. ## Response `200` Successful Response - StorageConnectionModel — Canonical representation of an external storage provider connection. Storage connections enable Mixpeek to access external cloud storage providers (Google Drive, S3, etc.) for automated file ingestion and synchronization. Each connection represents a configured integration with credentials, health monitoring, and usage tracking. Lifecycle States: - ACTIVE: Connection is healthy and ready for sync operations - SUSPENDED: Temporarily disabled by user (credentials preserved) - FAILED: Health checks failing (may need credential refresh) - ARCHIVED: Permanently retired (cannot be reactivated) Security: - Sensitive credential fields are encrypted at rest using MongoDB client-side field level encryption (CSFLE) - Credentials never appear in API responses or logs - Failed authentication attempts are logged in last_error - Consecutive failures trigger automatic suspension Use Cases: - Connect to team Google Drive for document ingestion - Sync files from customer S3 buckets - Monitor and process uploaded media files - Schedule periodic sync operations Health Monitoring: - Automatic health checks validate connectivity and credentials - consecutive_failures tracks authentication/network issues - Auto-disable after 5 consecutive failures to prevent lockout - last_error stores diagnostic information for debugging - `connection_id` string — Unique identifier for the storage connection. Auto-generated with 'conn_' prefix followed by secure random token. Format: conn_{15-character alphanumeric}. Used for API operations and audit trails. - `internal_id` string, required — REQUIRED. Organization internal identifier for multi-tenancy scoping. All connection operations are scoped to this organization. Format: int_{24-character secure token}. - `provider_type` 'google_drive' | 's3' | 'snowflake' | 'sharepoint' | 'tigris' | 'postgresql' | 'instagram' | 'tiktok' | 'rss' | 'http_api' | 'box' | 'brightdata' | 'backblaze' | 'mux' | 'email' | 'supabase' | 'iconik', required — Supported external storage providers for ingestion and sync. Mixpeek can connect to external storage providers to automatically ingest objects and keep them synchronized with your namespaces. Providers: GOOGLE_DRIVE: Google Drive and Google Workspace shared drives. - Authentication: Service account or OAuth2 - Features: Shared drive support, real-time sync, metadata preservation - Use cases: Marketing assets, team documents, knowledge bases - Limitations: Rate limits apply (10,000 requests/100 seconds per user) S3: Amazon S3 and S3-compatible storage (MinIO, DigitalOcean Spaces, etc). - Authentication: Access keys or IAM role assumption - Features: Bucket notifications, prefix filtering, versioning support - Use cases: Data lakes, video archives, ML datasets, backups - Limitations: IAM role assumption preferred over access keys SNOWFLAKE: Snowflake data warehouse tables. - Authentication: Key pair or username/password - Features: Incremental sync via watermarks, row-level mapping, schema introspection - Use cases: Customer data tables, product catalogs, transaction logs, metadata tables - Limitations: Each row becomes one object; large tables require incremental column SHAREPOINT: Microsoft SharePoint and OneDrive for Business. - Authentication: Azure AD OAuth2 (client credentials or delegated) - Features: Site/drive selection, folder sync, delta queries for incremental sync - Use cases: Enterprise documents, team collaboration files, compliance archives - Limitations: Requires Azure AD app registration; throttling limits apply Connection Requirements: - Valid credentials with read access to target files/buckets - Network connectivity from Mixpeek infrastructure - Appropriate IAM policies or share permissions configured Examples: - Use GOOGLE_DRIVE for syncing team marketing materials - Use S3 for ingesting video archives from data lakes - Use S3 with IAM role for secure production deployments - Use SHAREPOINT for syncing enterprise SharePoint document libraries TIGRIS: Tigris Data globally distributed object storage (S3-compatible). - Authentication: Access keys (same format as S3) - Features: S3-compatible API, global distribution, zero egress fees - Use cases: Globally distributed media, low-latency content delivery - Endpoint: https://fly.storage.tigris.dev POSTGRESQL: PostgreSQL relational database. - Authentication: Username/password - Features: SQL queries, incremental sync via watermarks, row-level mapping - Use cases: Customer data tables, product catalogs, transaction logs - Limitations: Each row becomes one object; large tables require incremental column INSTAGRAM: Instagram Business/Creator accounts via Meta Graph API. - Authentication: OAuth 2.0 via Meta Developer Console - Features: Media sync (posts, reels, carousels), engagement metrics, captions - Use cases: Social media content analysis, talent scouting, competitor monitoring - Limitations: Requires Instagram Business/Creator account; long-lived tokens (60 days) need refresh TIKTOK: TikTok accounts via TikTok Content API. - Authentication: OAuth 2.0 via TikTok Login Kit - Features: Video sync, engagement metrics (likes, views, shares), descriptions - Use cases: Social media content analysis, creator discovery, trend monitoring - Limitations: Access tokens expire in 24h (refresh_token flow); rate limits apply RSS: RSS/Atom feed entries. - Authentication: Optional HTTP headers (most feeds are public) - Features: Entry-level sync with title, author, categories, content - Use cases: News monitoring, blog ingestion, content aggregation - Limitations: No pagination; feeds are fetched in full each poll HTTP_API: Arbitrary REST/HTTP JSON APIs. - Authentication: Optional HTTP headers (API keys, Bearer tokens, etc.) - Features: Configurable JSONPath to items array, dedup via item ID field, incremental sync via timestamp field, GET or POST methods, JSON or JSONL responses - Use cases: Public APIs (Hacker News, GitHub), private APIs (Stripe, internal services), any JSON endpoint that returns a list of items - Limitations: No built-in pagination; API is fetched in full each poll BOX: Box cloud content management and file sharing. - Authentication: OAuth 2.0 (JWT or Client Credentials Grant with CCG) - Features: Folder sync, enterprise content management, metadata, versioning - Use cases: Enterprise document management, compliance archives, collaboration files - Limitations: Rate limits apply (10 API calls per second per user) BRIGHTDATA: BrightData web data platform for dataset collection and web scraping. - Authentication: API token from BrightData dashboard - Features: Pre-built datasets (LinkedIn, Amazon, etc.), custom scrapers, scheduled collection, geo-targeting, output format selection - Use cases: Competitor monitoring, market research, lead generation, e-commerce pricing, social media data ingestion - Limitations: Dataset availability depends on subscription; rate limits vary BACKBLAZE: Backblaze B2 Cloud Storage (S3-compatible object storage). - Authentication: Application key ID + application key from Backblaze console - Features: S3-compatible API, auto-discovers regional endpoint via B2 auth, full bucket/prefix sync, include/exclude patterns, modified_since filtering - Use cases: Cold storage sync, media archives, backup ingestion - Limitations: Buckets are region-specific; key must have listFiles/readFiles MUX: Mux video infrastructure platform. - Authentication: Access token ID + access token secret (HTTP Basic Auth) - Features: Video asset listing, static rendition download (MP4), metadata sync - Use cases: Video library ingestion, media asset processing, content analysis - Limitations: Assets must be in 'ready' status; static renditions must be enabled EMAIL: Inbound email connector (push-based). - Authentication: Per-connection webhook signing secret (HMAC-SHA256) - Features: MIME parsing, attachment extraction, sender allowlist, raw .eml archival - Use cases: Document intake via email forwarding, compliance mailbox, support inbox - Limitations: Requires external inbound email service (SES/Postmark/CloudMailin) SUPABASE: Supabase Storage (S3-compatible object storage built on open source). - Authentication: Dedicated S3 access keys (recommended) OR service_role JWT as session token (access_key_id=project_ref, secret=anon_key, session_token=service_role) - Features: S3-compatible API, per-project endpoint auto-derived from project_ref, bucket-scoped sync, include/exclude patterns, modified_since incremental sync - Use cases: Syncing user-uploaded assets from Supabase-backed apps, unified search across Supabase Storage + other sources - Limitations: Endpoint is project-specific (https://.storage.supabase.co); S3 keys must be generated from the Supabase dashboard (Storage → S3 Access Keys) - `provider_config` union, required — REQUIRED. Provider-specific configuration payload including credentials. Type depends on provider_type (GoogleDriveConfig, S3Config, etc.). SECURITY: Sensitive credential fields are encrypted at rest via MongoDB client-side field level encryption (CSFLE). Credentials never appear in API responses or logs. See provider_configs.py for detailed schemas. - GoogleDriveConfig — Google Drive and Google Workspace shared drive configuration. This configuration enables Mixpeek to connect to Google Drive for automated file ingestion and synchronization. Supports both personal Drive and Google Workspace shared drives (formerly Team Drives). Authentication Options: - Service Account: Recommended for production. No user interaction required. - OAuth2: Suitable for personal Drive access or development. Requirements: - Google Drive API enabled in Google Cloud Console - Appropriate authentication credentials configured - Files/folders shared with the service account or OAuth user - Network connectivity to drive.googleapis.com Use Cases: - Sync marketing materials from shared drives - Ingest documents from team collaboration folders - Monitor and process uploaded media files - Archive and search historical documents - `provider_type` 'google_drive' - `credentials` union, required — REQUIRED. Authentication credentials for Google Drive API access. Choose service_account for production (recommended) or oauth for personal access. The 'type' field determines which credential type is used. - GoogleDriveServiceAccountCredentials — Credentials for Google Drive service account authentication. Service accounts provide server-to-server authentication for Google Drive without requiring user interaction. They are ideal for automated sync operations. Prerequisites: - Create a service account in Google Cloud Console - Enable Google Drive API for the project - Download the JSON key file - Share target Drive files/folders with the service account email Security: - private_key field is encrypted at rest using MongoDB client-side field level encryption - Credentials never appear in logs or API responses - Use domain-wide delegation for G Suite environments Use Cases: - Automated ingestion pipelines from shared drives - Scheduled sync operations without user interaction - Service-to-service integration for enterprise deployments - `type` 'service_account' - `project_id` string, required — REQUIRED. Google Cloud project ID where the service account was created. Found in the JSON key file as 'project_id'. Format: lowercase alphanumeric with hyphens (e.g., 'my-project-123'). - `private_key_id` string, required — REQUIRED. Unique identifier for the private key. Found in the JSON key file as 'private_key_id'. Format: 40-character hexadecimal string. - `private_key` string, required — REQUIRED. PEM-encoded RSA private key for authentication. Found in the JSON key file as 'private_key'. SECURITY: This field is encrypted at rest. Never log or expose this value. Format: Must include BEGIN/END PRIVATE KEY markers. - `client_email` string, required — REQUIRED. Service account email address. Found in the JSON key file as 'client_email'. Share Drive files/folders with this email to grant access. Format: {account-name}@{project-id}.iam.gserviceaccount.com - `client_id` string, required — REQUIRED. Numeric service account identifier. Found in the JSON key file as 'client_id'. Format: 21-digit numeric string. - GoogleDriveOAuthCredentials — Credentials for Google Drive OAuth2 user authentication. OAuth2 credentials provide access to Google Drive on behalf of a specific user. This authentication method is suitable when accessing personal Drive files or when service account delegation is not available. Prerequisites: - Create OAuth 2.0 credentials in Google Cloud Console - Configure authorized redirect URIs - Complete OAuth consent flow to obtain refresh token - Ensure appropriate OAuth scopes are granted (drive.readonly or drive) Security: - client_secret and refresh_token are encrypted at rest - Access tokens are automatically refreshed and cached temporarily - Credentials are scoped to the user who granted consent Use Cases: - Personal Drive file access for individual users - Prototyping and development without service account setup - Environments where service account delegation is not feasible Limitations: - Requires user interaction during initial setup - Access is limited to files the consenting user can access - Refresh tokens can be revoked by the user - `type` 'oauth' - `client_id` string, required — REQUIRED. OAuth 2.0 client ID from Google Cloud Console. Found in the API credentials section. Format: {id}.apps.googleusercontent.com - `client_secret` string, required — REQUIRED. OAuth 2.0 client secret from Google Cloud Console. SECURITY: This field is encrypted at rest. Never log or expose this value. Format: Alphanumeric string from Google Cloud Console. - `refresh_token` string, required — REQUIRED. Long-lived refresh token obtained during OAuth consent flow. Used to automatically obtain new access tokens without user interaction. SECURITY: Encrypted at rest. Can be revoked by user at any time. Obtain via: Complete OAuth flow with drive.readonly or drive scope. - `shared_drive_id` string, nullable — NOT REQUIRED. Google Workspace shared drive (Team Drive) identifier. When provided, sync operations are scoped to this shared drive only. When omitted, syncs from 'My Drive' of the authenticated account. Find ID: Open shared drive in browser, copy ID from URL. Format: 0A{alphanumeric-string} - `impersonate_user` string, nullable — NOT REQUIRED. Email address to impersonate when using service account credentials. Requires domain-wide delegation to be enabled for the service account. Used in G Suite environments to access files as a specific user. When omitted, uses the service account's own access. Format: Valid email address in the G Suite domain. - S3Config — Amazon S3 and S3-compatible storage provider configuration. This configuration enables Mixpeek to connect to Amazon S3 or S3-compatible storage services (MinIO, DigitalOcean Spaces, Wasabi, Backblaze B2, etc.) for automated object ingestion and synchronization. Authentication Methods: 1. IAM Role Assumption (RECOMMENDED for production): - Most secure option with automatic credential rotation - No long-lived credentials shared - Ideal for customer-owned S3 buckets 2. Access Keys: - Simpler setup for development and testing - Works with S3-compatible services - Requires manual credential rotation Requirements: - Valid AWS credentials or IAM role configuration - S3 bucket with appropriate permissions (s3:GetObject, s3:ListBucket) - Network connectivity to S3 endpoint - Correct region configuration Supported Services: - Amazon S3 (all regions) - MinIO (self-hosted or cloud) - DigitalOcean Spaces - Wasabi Cloud Storage - Backblaze B2 - Any S3-compatible storage with compatible API Use Cases: - Ingest videos from data lakes - Sync images from marketing asset buckets - Process documents from archive storage - Monitor and index uploaded files - Backup and disaster recovery workflows - `provider_type` 's3' - `credentials` union, required — REQUIRED. AWS authentication credentials configuration. Choose 'iam_role' for production deployments (recommended) or 'access_key' for development, testing, or S3-compatible services. The 'type' field determines which credential mechanism is used. - S3AccessKeyCredentials — AWS S3 access key and secret credentials. Access keys provide programmatic access to S3 buckets using long-lived credentials. This authentication method is straightforward but less secure than IAM role assumption. Prerequisites: - IAM user or role with S3 access permissions - Access key and secret key generated in AWS Console - Appropriate bucket policies or IAM policies configured Security Considerations: - Access keys are long-lived and don't automatically expire - secret_access_key is encrypted at rest but should be rotated regularly - Consider using IAM role assumption (S3RoleCredentials) for production - Never commit access keys to version control Use Cases: - Quick prototyping and development - Testing S3 integrations - Temporary credentials with session_token for enhanced security - Accessing S3-compatible services (MinIO, DigitalOcean Spaces) Recommended Alternative: For production deployments, use S3RoleCredentials with IAM role assumption instead of access keys for better security and credential management. - `type` 'access_key' - `access_key_id` string, required — REQUIRED. AWS access key ID for authentication. Format: 20-character alphanumeric string starting with 'AKIA' (long-term) or 'ASIA' (temporary). Obtain from: AWS Console > IAM > Users > Security Credentials - `secret_access_key` string, required — REQUIRED. AWS secret access key for authentication. SECURITY: This field is encrypted at rest. Never log or expose this value. Format: 40-character base64-encoded string. Obtain from: AWS Console when creating/viewing access key (shown only once) - `session_token` string, nullable — NOT REQUIRED. Temporary session token for AWS STS credentials. REQUIRED when using temporary security credentials from AWS STS. NOT REQUIRED for long-term IAM user access keys. SECURITY: Encrypted at rest. Automatically expires after session duration. Format: Base64-encoded string, typically several hundred characters. Use case: Enhanced security with automatic credential rotation - S3RoleCredentials — AWS S3 IAM role assumption credentials (RECOMMENDED for production). IAM role assumption provides secure, temporary credentials for accessing customer S3 buckets without sharing long-lived access keys. This is the recommended authentication method for production deployments. How It Works: 1. Customer creates an IAM role in their AWS account 2. Role trust policy allows Mixpeek AWS account to assume the role 3. External ID provides additional security against confused deputy attacks 4. Mixpeek assumes the role and receives temporary credentials (auto-renewed) 5. Temporary credentials are used to access the customer's S3 bucket Prerequisites: 1. Create IAM role in customer AWS account 2. Attach policy granting s3:GetObject, s3:ListBucket permissions 3. Configure trust relationship to allow Mixpeek account 4. Use organization-specific external_id for security 5. Share role ARN with Mixpeek Security Advantages: - No long-lived credentials shared with third parties - Temporary credentials automatically rotate (1-hour sessions by default) - Customer retains full control and can revoke access anytime - External ID prevents confused deputy attacks - Audit trail in CloudTrail for all access Use Cases: - Production deployments accessing customer S3 buckets - Enterprise integrations requiring strong security - Multi-tenant environments with customer-owned storage - Compliance-sensitive workloads (HIPAA, SOC 2, etc.) - `type` 'iam_role' - `role_arn` string, required — REQUIRED. Amazon Resource Name (ARN) of the IAM role to assume. This role must exist in the customer's AWS account and have a trust relationship configured to allow Mixpeek to assume it. Format: arn:aws:iam::{account-id}:role/{role-name} Example trust policy should allow principal: arn:aws:iam::{mixpeek-account}:root Recommended role name: mixpeek-storage-sync-role - `external_id` string, required — REQUIRED. External ID for secure role assumption (prevents confused deputy attacks). This value should be unique to your organization and kept confidential. Mixpeek provides this value during connection setup. Must match the ExternalId condition in the role's trust policy. Format: Recommended pattern is mixpeek-{organization_id} Security: Include this in the trust policy Condition statement - `region` string, required — REQUIRED. AWS region where the S3 bucket is located. Must match the bucket's actual region to avoid routing errors. For S3-compatible services, use their documented region value or 'us-east-1' as a default if regions are not applicable. Format: AWS region code (e.g., us-east-1, eu-west-1) - `endpoint_url` string, nullable — NOT REQUIRED for AWS S3 (uses default AWS endpoints). REQUIRED for S3-compatible services to specify custom endpoint URL. Must be a valid HTTPS or HTTP URL without trailing slash. Examples: - MinIO: https://minio.example.com - DigitalOcean Spaces: https://nyc3.digitaloceanspaces.com - Wasabi: https://s3.wasabisys.com - `use_ssl` boolean — Whether to use TLS/SSL encryption for connections to S3. RECOMMENDED: Always True for production environments. Set to False only for local development with unencrypted endpoints. Default: True - `verify_ssl` boolean — Whether to verify TLS/SSL certificates when connecting. RECOMMENDED: Always True for production to prevent MITM attacks. Set to False only for development with self-signed certificates. Requires use_ssl=True to have any effect. Default: True - SnowflakeConfig — Snowflake data warehouse configuration for table-based sync. Enables syncing Snowflake table rows as JSON objects in Mixpeek buckets. Each row becomes one object, with incremental sync via watermark columns. Authentication Options: - Key Pair: Recommended for production (secure, password-less) - Username/Password: Fallback option (simpler setup) Requirements: - Snowflake account with read access to target tables - Warehouse with compute resources - SELECT permissions on target tables/schemas - USAGE permissions on database, schema, warehouse Use Cases: - Sync customer data tables for AI/ML pipelines - Ingest product catalog for search/recommendations - Process transaction logs for analytics - Mirror metadata tables for vector search Example: Production setup with key pair auth: ```python config = { "provider_type": "snowflake", "credentials": { "type": "key_pair", "username": "MIXPEEK_SYNC", "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n", }, "account": "xy12345.us-east-1", "warehouse": "MIXPEEK_SYNC_WH", "database": "PRODUCTION", "schema": "PUBLIC", "role": "SYNC_ROLE", "incremental_column": "updated_at", "primary_key_columns": ["id"], } ``` - `provider_type` 'snowflake' - `credentials` union, required — REQUIRED. Authentication credentials for Snowflake. Choose key_pair for production (recommended) or username_password for simpler setup. The 'type' field determines which credential mechanism is used. - SnowflakeKeyPairCredentials — Snowflake key pair authentication (RECOMMENDED for production). Key pair authentication provides secure, password-less access to Snowflake. The private key is encrypted at rest using MongoDB CSFLE. Prerequisites: 1. Generate RSA key pair (2048-bit minimum) 2. Extract public key and assign to Snowflake user 3. Store private key securely (encrypted) Security: - Private key encrypted at rest via CSFLE - No password exposure - Key rotation supported - Recommended for production Example: Generate key pair: ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub ``` Assign public key to Snowflake user: ```sql ALTER USER mixpeek_sync SET RSA_PUBLIC_KEY='MIIBIjANBg...'; ``` - `type` 'key_pair' - `username` string, required — Snowflake username for authentication - `private_key` string, required — REQUIRED. PEM-encoded RSA private key for authentication. SECURITY: This field is encrypted at rest via CSFLE. Never log or expose. Format: -----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY----- - `private_key_passphrase` string, nullable — NOT REQUIRED. Passphrase for encrypted private key. SECURITY: Encrypted at rest if provided. Use only if private key is passphrase-protected. - SnowflakeUsernamePasswordCredentials — Snowflake username/password authentication (FALLBACK option). Traditional username/password authentication for Snowflake. Less secure than key pair authentication but simpler to set up. Security: - Password encrypted at rest via CSFLE - Consider using key pair auth for production - Enable MFA on Snowflake user account - `type` 'username_password' - `username` string, required — REQUIRED. Snowflake username (case-insensitive). - `password` string, required — REQUIRED. Snowflake password for authentication. SECURITY: This field is encrypted at rest via CSFLE. Never log or expose. - `account` string, required — REQUIRED. Snowflake account identifier. Format: {account_locator}.{cloud_region} or {org_name}-{account_name} Find in: Snowflake UI > Account dropdown > Account URL - `warehouse` string, required — REQUIRED. Warehouse name for compute resources. Must have USAGE privilege on this warehouse. Warehouse will be used for all sync queries. Consider: Use dedicated warehouse for sync operations to isolate costs. - `database` string, nullable — NOT REQUIRED if fully qualified table name used in source_path. Database name for default context. Can be omitted if source_path uses {DATABASE}.{SCHEMA}.{TABLE} format. Must have USAGE privilege if specified. - `schema` string, nullable — NOT REQUIRED if fully qualified table name used in source_path. Schema name for default context. Can be omitted if source_path uses {SCHEMA}.{TABLE} or {DATABASE}.{SCHEMA}.{TABLE}. Must have USAGE privilege if specified. - `role` string, nullable — NOT REQUIRED. Snowflake role to use for operations. If omitted, uses user's default role. Role must have SELECT on target tables and USAGE on database/schema/warehouse. Best practice: Create dedicated read-only role for sync operations. - `incremental_column` string, nullable — NOT REQUIRED. Column name for incremental sync watermark. Should be a TIMESTAMP, TIMESTAMP_NTZ, or DATE column that tracks row modifications. When set, only rows with {incremental_column} > last_sync_watermark are synced. Common values: updated_at, modified_at, last_updated, ingestion_timestamp. If omitted, full table scan on every sync (not recommended for large tables). - `primary_key_columns` string[], nullable — NOT REQUIRED. Column names forming the primary key for stable object IDs. Used to generate deterministic file_id for deduplication. If omitted, uses hash of entire row content (less stable). Recommendation: Always specify for production to ensure idempotent syncs. - `query_timeout_seconds` integer — Query timeout in seconds. Prevents long-running queries from blocking sync operations. Default: 300 seconds (5 minutes). Increase for large tables or complex queries. - `fetch_size` integer — Number of rows to fetch per network round-trip. Higher values reduce network overhead but increase memory usage. Default: 1000 rows. Tune based on row size and available memory. - SharePointConfig — Microsoft SharePoint and OneDrive for Business configuration. Enables Mixpeek to connect to SharePoint sites, document libraries, and OneDrive for Business for automated file ingestion and synchronization. Architecture: SharePoint hierarchy: Tenant → Sites → Drives (Document Libraries) → Items - site_id: Identifies the SharePoint site - drive_id: Identifies a specific document library within the site - folder_path: Path within the document library Authentication Methods: 1. Client Credentials (RECOMMENDED for production): - App-only access without user interaction - Requires admin consent for Graph API permissions - Access level determined by app permissions 2. Delegated: - User-level access via OAuth consent - Access limited to user's permissions - Requires refresh token management Requirements: - Azure AD application registration - Microsoft Graph API permissions (Sites.Read.All, Files.Read.All) - Network connectivity to graph.microsoft.com Use Cases: - Sync SharePoint document libraries - Ingest enterprise collaboration files - Monitor and process uploaded documents - Archive compliance-sensitive materials Example: ```python config = { "provider_type": "sharepoint", "credentials": { "type": "client_credentials", "tenant_id": "12345678-...", "client_id": "87654321-...", "client_secret": "your-secret", }, "site_id": "contoso.sharepoint.com,guid1,guid2", "drive_id": "b!abc123...", "folder_path": "/Shared Documents/Marketing", } ``` - `provider_type` 'sharepoint' - `credentials` union, required — REQUIRED. Azure AD authentication credentials. Choose client_credentials for production (app-only) or delegated for user-level access. The 'type' field determines which authentication flow is used. - SharePointClientCredentials — SharePoint/OneDrive client credentials (app-only) authentication. Client credentials flow provides application-level access without user interaction. Recommended for automated sync operations in enterprise environments. Prerequisites: 1. Register an application in Azure AD (portal.azure.com) 2. Grant Microsoft Graph API permissions: - Sites.Read.All (Application) - for SharePoint site access - Files.Read.All (Application) - for file access 3. Admin consent granted for the permissions 4. Generate client secret Security: - client_secret encrypted at rest via CSFLE - Provides application-level access to all sites (based on permissions) - No user context - accesses files as the application itself - Consider using certificate-based auth for production Use Cases: - Automated enterprise-wide document ingestion - Background sync without user interaction - Multi-tenant applications with admin consent - `type` 'client_credentials' - `tenant_id` string, required — REQUIRED. Azure AD tenant ID (directory ID). Find in: Azure Portal > Azure Active Directory > Overview. Format: UUID (e.g., '12345678-1234-1234-1234-123456789abc') - `client_id` string, required — REQUIRED. Azure AD application (client) ID. Find in: Azure Portal > App Registrations > Your App > Overview. Format: UUID - `client_secret` string, required — REQUIRED. Azure AD client secret for authentication. SECURITY: This field is encrypted at rest via CSFLE. Never log or expose. Generate in: Azure Portal > App Registrations > Your App > Certificates & secrets. Note: Secrets expire; consider using certificates for production. - SharePointDelegatedCredentials — SharePoint/OneDrive delegated (user) authentication via OAuth2. Delegated flow provides access on behalf of a specific user. Useful when you need to access files with user-level permissions. Prerequisites: 1. Register an application in Azure AD 2. Grant Microsoft Graph API permissions: - Sites.Read.All (Delegated) or Sites.Selected - Files.Read.All (Delegated) or Files.Read 3. Configure redirect URI for OAuth flow 4. Complete OAuth consent to obtain refresh token Security: - client_secret and refresh_token encrypted at rest - Access scoped to what the consenting user can access - Refresh tokens can be revoked by user or admin Use Cases: - Personal OneDrive access - User-specific SharePoint sites - Respecting per-user permissions - `type` 'delegated' - `tenant_id` string, required — REQUIRED. Azure AD tenant ID. Use 'common' for multi-tenant apps, or specific tenant ID for single-tenant. - `client_id` string, required — REQUIRED. Azure AD application (client) ID. - `client_secret` string, required — REQUIRED. Azure AD client secret. SECURITY: Encrypted at rest via CSFLE. - `refresh_token` string, required — REQUIRED. OAuth2 refresh token obtained from consent flow. SECURITY: Encrypted at rest. Can be revoked by user. Obtain via: Complete OAuth flow with Files.Read.All scope. - `site_id` string, nullable — NOT REQUIRED if using personal OneDrive. SharePoint site identifier for targeting a specific site. Format: '{hostname},{site-collection-id},{web-id}' or site URL. Find via: Microsoft Graph API GET /sites?search={keyword} Example: 'contoso.sharepoint.com,12345678-...,87654321-...' - `drive_id` string, nullable — NOT REQUIRED if you want to use the default document library. Specific drive (document library) ID within the site. Find via: GET /sites/{site-id}/drives Format: Base64-encoded ID starting with 'b!' - `folder_path` string, nullable — NOT REQUIRED. Path within the drive to sync from. If omitted, syncs from the root of the drive. Format: Forward-slash separated path (e.g., '/Documents/Marketing'). Note: Leading slash is optional. - TigrisConfig — Tigris Data globally distributed object storage configuration. Tigris is an S3-compatible object storage service with automatic global distribution, zero egress fees, and built-in CDN capabilities. It uses the AWS S3 API, making integration straightforward. Key Features: - S3-compatible API (drop-in replacement) - Automatic global data distribution - Zero egress fees - Built-in CDN and caching - Strong consistency guarantees Authentication: - Access keys only (similar to S3 access keys) - No IAM role assumption (Tigris is not AWS) Requirements: - Tigris account with access keys - Bucket created in Tigris console - Network connectivity to fly.storage.tigris.dev Use Cases: - Globally distributed media assets - Low-latency content delivery worldwide - Cost-effective storage with no egress fees - S3-compatible workflows outside AWS - `provider_type` 'tigris' - `credentials` TigrisAccessKeyCredentials, required — Tigris Data access key credentials. Tigris uses S3-compatible authentication with access keys. Credentials can be obtained from the Tigris dashboard at https://console.tigris.dev. Prerequisites: - Create a Tigris account at https://www.tigrisdata.com - Create a bucket in the Tigris console - Generate access keys from the dashboard Security: - secret_access_key is encrypted at rest using MongoDB CSFLE - Rotate keys regularly via the Tigris dashboard - Use bucket-scoped keys when possible for least privilege Use Cases: - Globally distributed object storage - Low-latency content delivery - S3-compatible workflows with zero egress fees - `type` 'access_key' - `access_key_id` string, required — REQUIRED. Tigris access key ID for authentication. Obtain from: Tigris Console > Access Keys - `secret_access_key` string, required — REQUIRED. Tigris secret access key for authentication. SECURITY: This field is encrypted at rest. Never log or expose this value. Obtain from: Tigris Console when creating access key (shown only once) - `region` string — Region for Tigris. Typically 'auto' for automatic global distribution. Tigris automatically distributes data globally, so region is usually 'auto'. Default: auto - `endpoint_url` string — Tigris S3-compatible endpoint URL. Default: https://fly.storage.tigris.dev This is the standard Tigris endpoint and usually doesn't need to be changed. - PostgreSQLConfig — PostgreSQL database configuration for table-based sync and SQL queries. Enables syncing PostgreSQL table rows as JSON objects and running SQL queries via the SQL Lookup retriever stage. Each row becomes one object, with incremental sync via watermark columns. Authentication: - Username/Password: Standard PostgreSQL authentication Requirements: - PostgreSQL 12+ recommended - Read access to target tables - Network connectivity to PostgreSQL server Use Cases: - Sync customer data tables for AI/ML pipelines - Run SQL lookups to enrich documents in retriever pipelines - Ingest product catalog for search/recommendations - Process transaction logs for analytics Example: ```python config = { "provider_type": "postgresql", "credentials": { "type": "username_password", "username": "mixpeek_sync", "password": "secure_password", }, "host": "db.example.com", "port": 5432, "database": "production", "schema": "public", "ssl_mode": "require", } ``` - `provider_type` 'postgresql' - `credentials` PostgreSQLCredentials, required — PostgreSQL username/password authentication. Standard username/password authentication for PostgreSQL databases. Password is encrypted at rest using MongoDB CSFLE. Security: - password field is encrypted at rest via CSFLE - Consider using SSL mode 'require' for production - Use dedicated read-only database user for sync operations - `type` 'username_password' - `username` string, required — REQUIRED. PostgreSQL username for authentication. - `password` string, required — REQUIRED. PostgreSQL password for authentication. SECURITY: This field is encrypted at rest via CSFLE. Never log or expose. - `host` string, required — REQUIRED. PostgreSQL server hostname or IP address. Examples: 'localhost', 'db.example.com', '192.168.1.100' - `port` integer — PostgreSQL server port. Default: 5432 (standard PostgreSQL port) - `database` string, required — REQUIRED. Database name to connect to. User must have CONNECT privilege on this database. - `schema` string, nullable — Schema name for default context. Default: 'public'. User must have USAGE privilege on this schema. - `ssl_mode` string — SSL/TLS connection mode. Options: 'disable', 'allow', 'prefer', 'require', 'verify-ca', 'verify-full'. Default: 'prefer'. RECOMMENDED: Use 'require' or stricter for production environments. - `incremental_column` string, nullable — NOT REQUIRED. Column name for incremental sync watermark. Should be a TIMESTAMP or DATE column that tracks row modifications. Common values: updated_at, modified_at, last_updated. If omitted, full table scan on every sync. - `primary_key_columns` string[], nullable — NOT REQUIRED. Column names forming the primary key for stable object IDs. Used to generate deterministic file_id for deduplication. If omitted, uses hash of entire row content. - `query_timeout_seconds` integer — Query timeout in seconds. Default: 300 seconds (5 minutes). Increase for large tables or complex queries. - `fetch_size` integer — Number of rows to fetch per batch. Higher values reduce network overhead but increase memory usage. Default: 1000 rows. - InstagramConfig — Instagram Business/Creator account configuration via Meta Graph API. Enables Mixpeek to sync media (posts, reels, carousels) from Instagram accounts using the official Instagram Graph API. Each media item becomes a bucket object with metadata (caption, likes, comments, permalink). Authentication: - OAuth 2.0 via Meta Developer Console - Long-lived tokens (60 days) with automatic refresh Requirements: - Instagram Business or Creator account - Facebook Page linked to the Instagram account - Meta app with instagram_basic permission Use Cases: - Sync creator content for talent analysis - Monitor competitor ad creatives - Build social media content libraries for multimodal search - `provider_type` 'instagram' - `credentials` InstagramOAuthCredentials, required — Credentials for Instagram Graph API OAuth2 authentication. Instagram uses Meta's OAuth2 flow to obtain access tokens. The flow produces a short-lived token which is exchanged for a long-lived token (valid for 60 days). Long-lived tokens can be refreshed before expiry. Prerequisites: - Meta Developer account with an Instagram app - Instagram Business or Creator account linked to a Facebook Page - App must have instagram_basic permission approved Security: - client_secret and access_token encrypted at rest via CSFLE - Long-lived tokens must be refreshed before 60-day expiry - Token refresh happens automatically during sync execution - `type` 'oauth' - `client_id` string, required — Instagram App ID from Meta Developer Console. - `client_secret` string, required — SECURITY: Encrypted at rest via CSFLE. Instagram App Secret from Meta Developer Console. - `access_token` string, required — SECURITY: Encrypted at rest. Long-lived Instagram access token (60-day validity). - `token_expires_at` string, date-time, nullable — Token expiration timestamp. Used to trigger proactive refresh. - `instagram_user_id` string, required — Instagram User ID of the account that authorized access. - `instagram_username` string, nullable — Instagram username for display purposes. - `scopes` string[] — OAuth scopes granted during authorization. - `business_account_id` string, nullable — Instagram Business Account ID if different from the user ID. Auto-populated from OAuth flow. - TikTokConfig — TikTok account configuration via TikTok Content API. Enables Mixpeek to sync videos from TikTok accounts using the official TikTok API. Each video becomes a bucket object with metadata (description, likes, views, shares, comments). Authentication: - OAuth 2.0 via TikTok Login Kit - Access tokens expire in 24h (refresh_token flow) Requirements: - TikTok Developer account with approved app - user.info.basic and video.list scopes Use Cases: - Sync creator video content for talent analysis - Monitor trending content and competitors - Build video content libraries for multimodal search - `provider_type` 'tiktok' - `credentials` TikTokOAuthCredentials, required — Credentials for TikTok Content API OAuth2 authentication. TikTok uses OAuth2 via TikTok Login Kit. Access tokens are short-lived (24 hours) and must be refreshed using the refresh token. Prerequisites: - TikTok Developer account with an approved app - App must have user.info.basic and video.list scopes Security: - client_secret, access_token, and refresh_token encrypted at rest via CSFLE - Access tokens expire in 24h; refresh tokens used for renewal - Token refresh happens automatically during sync execution - `type` 'oauth' - `client_key` string, required — TikTok App Client Key from Developer Portal. - `client_secret` string, required — SECURITY: Encrypted at rest via CSFLE. TikTok App Client Secret from Developer Portal. - `access_token` string, required — SECURITY: Encrypted at rest. TikTok access token (24h validity). - `refresh_token` string, required — SECURITY: Encrypted at rest. Used to obtain new access tokens. - `token_expires_at` string, date-time, nullable — Access token expiration timestamp. - `open_id` string, required — TikTok user's open_id from the authorization response. - `scopes` string[] — OAuth scopes granted during authorization. - RSSConfig — RSS/Atom feed configuration. source_path = feed URL. - `provider_type` 'rss' - `credentials` RSSHttpHeaderCredentials — Optional HTTP credentials for private RSS feeds. - `type` 'http_headers' - `headers` object — HTTP headers for feed requests (e.g., Authorization). - `user_agent` string — User-Agent header for feed requests. - `request_timeout` integer — HTTP request timeout in seconds. - HTTPAPIConfig — REST/HTTP JSON API configuration. source_path = API URL. Enables Mixpeek to sync items from any JSON REST API endpoint. Each item in the API response becomes a bucket object with its JSON body stored as the blob. Authentication: - Optional HTTP headers (API keys, Bearer tokens, etc.) Requirements: - API must return JSON (or JSONL) response - Response must contain an array of items (at root or nested path) - Each item must have a unique ID field for deduplication Use Cases: - Sync data from public APIs (Hacker News, GitHub, etc.) - Ingest records from internal REST services - Poll third-party APIs (Stripe, Shopify, etc.) - Monitor any JSON endpoint for new items - `provider_type` 'http_api' - `credentials` HTTPAPIHeaderCredentials — Optional HTTP credentials for HTTP API endpoints. Supports arbitrary headers for authentication (API keys, Bearer tokens, etc.). Same pattern as RSS credentials but for REST API endpoints. Security: - Header values containing secrets are encrypted at rest via CSFLE - Common patterns: Authorization: Bearer , X-API-Key: - `type` 'http_headers' - `headers` object — HTTP headers for API requests (e.g., Authorization, X-API-Key). - `http_method` 'GET' | 'POST' — HTTP method to use when calling the API. - `request_body` object, nullable — Optional JSON body for POST requests. - `items_path` string — Dot-notation path to the array of items in the JSON response. Examples: 'hits' for {hits: [...]}, 'data.items' for {data: {items: [...]}}. Leave empty if the response is a top-level array. - `item_id_field` string, required — REQUIRED. Field name in each item used as a unique ID for deduplication. Examples: 'id', 'objectID', 'uuid'. - `item_modified_field` string, nullable — Optional field name for a timestamp used for incremental sync. Supports ISO 8601 strings and Unix timestamps. Examples: 'updated_at', 'created_at', 'modified'. - `response_content_type` 'json' | 'jsonl' — Response format: 'json' (default) or 'jsonl' (newline-delimited JSON). - `user_agent` string — User-Agent header for API requests. - `request_timeout` integer — HTTP request timeout in seconds. - BoxConfig — Box cloud content management and file sharing configuration. Enables Mixpeek to connect to Box for automated file ingestion and synchronization. Supports enterprise-grade content management features including folder sync, metadata, versioning, and retention policies. Authentication Methods: 1. OAuth 2.0 (for user-level access): - Standard OAuth flow with access/refresh tokens - Access scoped to the authorizing user's content 2. Client Credentials Grant (CCG) (RECOMMENDED for production): - Server-to-server without user interaction - Acts as service account or specific user - Requires admin authorization in Box Admin Console 3. JWT (for high-security enterprise): - RSA key pair for signing JWT assertions - No user interaction required - Highest security option Requirements: - Box Developer account with an application - Application authorized in Box Admin Console (for CCG/JWT) - Network connectivity to api.box.com Use Cases: - Sync enterprise document libraries - Ingest compliance and legal documents - Monitor collaboration folders for new content - Archive and search enterprise content - `provider_type` 'box' - `credentials` union, required — REQUIRED. Box authentication credentials. Choose 'oauth' for user-level access, 'ccg' for server-to-server (recommended), or 'jwt' for high-security enterprise. The 'type' field determines which authentication flow is used. - BoxOAuthCredentials — Credentials for Box OAuth 2.0 authentication. Box supports OAuth 2.0 with access and refresh tokens. The refresh token is used to automatically obtain new access tokens without user interaction. Prerequisites: - Create a Box application at https://developer.box.com - Configure OAuth 2.0 with the appropriate scopes - Complete the OAuth consent flow to obtain tokens Security: - client_secret, access_token, and refresh_token encrypted at rest via CSFLE - Access tokens expire in ~60 minutes; refresh tokens used for renewal - Token refresh happens automatically during sync execution - `type` 'oauth' - `client_id` string, required — REQUIRED. Box application client ID. Found in: Box Developer Console > Your App > Configuration > OAuth 2.0 Credentials. - `client_secret` string, required — REQUIRED. Box application client secret. SECURITY: Encrypted at rest via CSFLE. Never log or expose. Found in: Box Developer Console > Your App > Configuration. - `access_token` string, required — REQUIRED. Box OAuth 2.0 access token. SECURITY: Encrypted at rest. Expires in ~60 minutes. - `refresh_token` string, required — REQUIRED. Box OAuth 2.0 refresh token for automatic token renewal. SECURITY: Encrypted at rest. Single-use; new one issued on each refresh. - BoxCCGCredentials — Credentials for Box Client Credentials Grant (CCG) authentication. CCG provides server-to-server authentication without user interaction. Recommended for enterprise and automated sync operations. Prerequisites: - Create a Box application with Server Authentication (Client Credentials Grant) - Authorize the application in the Box Admin Console - Optionally configure an enterprise or user ID to act as Security: - client_secret encrypted at rest via CSFLE - No user tokens involved; app authenticates as itself or as a user/enterprise - `type` 'ccg' - `client_id` string, required — REQUIRED. Box application client ID from Developer Console. - `client_secret` string, required — REQUIRED. Box application client secret. SECURITY: Encrypted at rest via CSFLE. - `enterprise_id` string, nullable — Enterprise ID to authenticate as. Required when using CCG to act as the enterprise (service account). Find in: Box Admin Console > Enterprise Settings. - `user_id` string, nullable — User ID to authenticate as. Used when the app needs to act as a specific managed user. Mutually exclusive with enterprise_id for token acquisition. - BoxJWTCredentials — Credentials for Box JWT (JSON Web Token) authentication. JWT provides server-to-server authentication using a public/private key pair. Recommended for enterprise integrations requiring high security. Prerequisites: - Create a Box application with Server Authentication (with JWT) - Generate a public/private key pair in the Developer Console - Authorize the application in the Box Admin Console - Download the JSON configuration file Security: - private_key encrypted at rest via CSFLE - RSA key pair used for signing JWT assertions - No user interaction required - `type` 'jwt' - `client_id` string, required — REQUIRED. Box application client ID. - `client_secret` string, required — REQUIRED. Box application client secret. SECURITY: Encrypted at rest via CSFLE. - `enterprise_id` string, required — REQUIRED. Box enterprise ID for JWT authentication. Find in: Box Admin Console > Enterprise Settings. - `jwt_key_id` string, required — REQUIRED. Public key ID registered with Box. Found in the JSON config file as 'publicKeyID'. - `private_key` string, required — REQUIRED. PEM-encoded RSA private key for JWT signing. SECURITY: Encrypted at rest via CSFLE. Never log or expose. Found in the JSON config file as 'privateKey'. - `private_key_passphrase` string, nullable — Passphrase for the private key if it is encrypted. SECURITY: Encrypted at rest via CSFLE. - `folder_id` string — Box folder ID to sync from. Default '0' is the root folder. Find folder ID: Open folder in Box web UI, copy the numeric ID from the URL. Example URL: https://app.box.com/folder/123456789 → folder_id='123456789' - BrightDataConfig — BrightData web data platform configuration for dataset and scraper syncs. Enables Mixpeek to collect data from BrightData's pre-built datasets (LinkedIn, Amazon, Google Maps, etc.) or custom web scrapers. Each sync triggers a new dataset snapshot, polls until ready, then downloads and ingests each row as a bucket object. Authentication: - API token from BrightData dashboard Requirements: - Active BrightData subscription with Datasets or Web Scraper access - API token with appropriate dataset permissions - Network connectivity to api.brightdata.com Use Cases: - Ingest LinkedIn company/people datasets for lead generation - Monitor e-commerce pricing via Amazon product datasets - Collect Google Maps business data for location intelligence - Run custom web scrapers and process results as documents - Aggregate public social media data for trend analysis - `provider_type` 'brightdata' - `credentials` BrightDataCredentials, required — API token credentials for BrightData platform. Security: - api_token is encrypted at rest via CSFLE - Tokens can be created/rotated in the BrightData dashboard - `type` 'api_token' - `api_token` string, required — REQUIRED. BrightData API token. SECURITY: Encrypted at rest. Never log or expose. Find in: BrightData Dashboard > Account > API Token. - `customer_id` string, nullable — NOT REQUIRED. BrightData customer ID (zone-level auth). Required when using zone-specific API access. Find in: BrightData Dashboard > your zone settings. - `default_output_format` 'jsonl' | 'json' | 'csv' | 'ndjson' — Output format for downloaded dataset results. 'jsonl' (default) streams one JSON object per line. 'json' returns a full JSON array. 'csv' returns comma-separated values. 'ndjson' is equivalent to jsonl. - `country` string, nullable — NOT REQUIRED. ISO 3166-1 alpha-2 country code for geo-targeted data collection. When set, the dataset snapshot targets data from this country. Examples: 'us', 'gb', 'de', 'jp'. - `inputs` object[], nullable — NOT REQUIRED. Default seed inputs for datasets that require input URLs or records (e.g. LinkedIn company profiles). Each item is a JSON object passed to the BrightData trigger API. Overridden per-call when inputs are supplied at sync time. - BackblazeConfig — Backblaze B2 Cloud Storage configuration. Backblaze B2 is an S3-compatible object storage service. This provider auto-discovers the regional S3 endpoint via the B2 authorization API, so users only need to provide their key ID and application key. Authentication: - Application key ID + application key from Backblaze console Requirements: - Active Backblaze account with B2 Cloud Storage enabled - Application key with listBuckets, listFiles, and readFiles capabilities - (Optional) Restrict key to specific bucket for scoped access Use Cases: - Cold storage archive ingestion - Media file sync from Backblaze-hosted assets - Backup data processing - Cost-effective large-file storage sync - `provider_type` 'backblaze' - `credentials` BackblazeCredentials, required — Application key credentials for Backblaze B2 Cloud Storage. Security: - application_key is encrypted at rest via CSFLE - Keys can be created/rotated in the Backblaze console - `type` 'application_key' - `key_id` string, required — REQUIRED. Backblaze B2 application key ID. Found in: Backblaze Console > App Keys > keyID column. - `application_key` string, required — REQUIRED. Backblaze B2 application key (secret). SECURITY: Encrypted at rest. Shown only once when key is created. Found in: Backblaze Console > App Keys — copy immediately on creation. - MuxConfig — Mux video infrastructure platform configuration. Mux is a video API platform that provides video hosting, encoding, and streaming. This provider syncs video assets from a Mux account into Mixpeek for processing and analysis. Authentication: - Access token ID + secret from Mux dashboard (HTTP Basic Auth) Requirements: - Active Mux account with video assets - Access token with Mux Video read permissions - Assets must have playback IDs for download Use Cases: - Video library ingestion and analysis - Media asset processing pipelines - Content moderation and classification - Video search and discovery - `provider_type` 'mux' - `credentials` MuxCredentials, required — Access token credentials for Mux video infrastructure platform. Security: - access_token_secret is encrypted at rest via CSFLE - Tokens can be created/rotated in the Mux dashboard - `type` 'access_token' - `access_token_id` string, required — REQUIRED. Mux access token ID. Found in: Mux Dashboard > Settings > Access Tokens. - `access_token_secret` string, required — REQUIRED. Mux access token secret. SECURITY: Encrypted at rest. Shown only once when created. Found in: Mux Dashboard > Settings > Access Tokens — copy immediately on creation. - `webhook_secret` string, nullable — NOT REQUIRED. Mux webhook signing secret for verifying inbound webhook events (e.g. video.asset.deleted, which cascades deletes to Mixpeek bucket objects synced from this connection). Set this to enable cascade deletion when assets are removed from Mux. SECURITY: Encrypted at rest. Find in: Mux Dashboard > Settings > Webhooks > {webhook} > Signing Secret. After saving, register the URL https://api.mixpeek.com/v1/webhooks/mux/{connection_id} in your Mux dashboard. - IconikConfig — Iconik DAM platform configuration. Iconik is a cloud-native DAM (Digital Asset Management) platform. This provider syncs video/image/audio assets from an Iconik account into Mixpeek for processing and analysis. Authentication: - App-ID + Auth-Token (HTTP headers) Requirements: - Active Iconik account with API application configured - Auth token with asset read permissions Use Cases: - Video ad library ingestion and analysis - Creative asset search and discovery - Talent/cast identification across ad library - `provider_type` 'iconik' - `credentials` IconikCredentials, required — App-ID + Auth-Token credentials for Iconik DAM platform. Security: - auth_token is encrypted at rest via CSFLE - Tokens are created in Iconik Admin > Settings > API Applications - `type` 'app_token' - `app_id` string, required — REQUIRED. Iconik application ID. Found in: Iconik Admin > Settings > API Applications. - `auth_token` string, required — REQUIRED. Iconik auth token (JWT or API token). SECURITY: Encrypted at rest. Found in: Iconik Admin > Settings > API Applications > Token. - `webhook_secret` string, nullable — NOT REQUIRED. Webhook secret for verifying inbound Iconik webhook events. Set this to enable webhook-driven sync (asset create/update/delete). SECURITY: Encrypted at rest. - `base_url` string, nullable — Iconik API base URL. Defaults to https://app.iconik.io. Override for on-premise Iconik installations. - `metadata_view_id` string, nullable — Iconik metadata view ID for fetching custom metadata fields. Found in: Iconik Admin > Metadata > Views. Set to 'none' to skip metadata view fetches entirely. - `rate_limit_rps` number, nullable — Max requests per second to the Iconik API (shared across all sync shards). - `rate_limit_redis_url` string, nullable — Redis URL for distributed rate limiting across workers. - SharedOrganizationsConnectionsProviderConfigsEmailConfig — Inbound email connector configuration (push-based). Customers receive a dedicated inbound email address per connection. Emails forwarded to that address are parsed (MIME), attachments are extracted as blobs, and the whole message becomes a bucket object. Authentication: - Inbound webhook verified via HMAC-SHA256 signing secret Requirements: - External inbound email service (SES Inbound / Postmark / CloudMailin) configured to POST parsed emails to the webhook URL Use Cases: - Compliance document intake (legal, healthcare) - Customer support email ingestion - Secure document forwarding pipelines - Media collection via email attachments - `provider_type` 'email' - `credentials` EmailCredentials — Webhook signing credentials for inbound email verification. Security: - webhook_secret is encrypted at rest via CSFLE - Used to verify HMAC-SHA256 signatures on inbound email webhooks - `type` 'webhook_secret' - `webhook_secret` string — Shared secret for verifying inbound email webhook signatures. Auto-generated on connection creation if left empty. SECURITY: Encrypted at rest via CSFLE. - `inbound_address` string, nullable — System-assigned inbound email address. Auto-provisioned on connection creation. Read-only. - `allowed_senders` string[] — Sender allowlist. Emails from senders not matching this list are rejected. Supports exact addresses (user@company.com) and domain wildcards (*@company.com). Empty list = accept all senders. - `store_raw_eml` boolean — Store the raw .eml file as an additional blob for chain of custody. - object - `name` string, required — REQUIRED. Human-readable connection name for identification. Displayed in dashboards, sync logs, and API responses. Must be unique within the organization for clarity. Format: 1-100 characters, descriptive of the connection's purpose. - `description` string, nullable — NOT REQUIRED. Optional description explaining the connection's purpose and scope. Helpful for team collaboration and documentation. Format: Up to 500 characters. - `status` 'PENDING' | 'QUEUED' | 'IN_PROGRESS' | 'PROCESSING' | 'COMPLETED' | 'COMPLETED_WITH_ERRORS' | 'FAILED' | 'CANCELED' | 'INTERRUPTED' | 'UNKNOWN' | 'SKIPPED' | 'DRAFT' | 'ACTIVE' | 'ARCHIVED' | 'SUSPENDED' | 'DEACTIVATED' — Enumeration of task statuses for tracking asynchronous operations. Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution. Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces) Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted ACTIVE: Entity is active and operational (for buckets, collections, etc.) ARCHIVED: Entity has been archived SUSPENDED: Entity has been temporarily suspended Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state. Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately. Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling - `is_active` boolean — Quick boolean flag for filtering active connections in queries. True when status is ACTIVE, False for SUSPENDED/FAILED/ARCHIVED. Maintained automatically when status changes. Use for efficient filtering: db.connections.find({'is_active': True}) - `last_used_at` string, date-time, nullable — NOT REQUIRED. UTC timestamp of the most recent successful sync operation. Updated automatically after each successful file sync/list operation. None if connection has never been used. Useful for identifying stale connections and usage analytics. - `last_error` string, nullable — NOT REQUIRED. Most recent error message from failed health check or sync. Populated when authentication fails, network errors occur, or permissions denied. None when connection is healthy. Format: Error message truncated to 1000 characters. Used for diagnostics and troubleshooting. - `consecutive_failures` integer — Counter tracking consecutive failed health checks or sync attempts. Incremented on each failure, reset to 0 on success. Used to implement automatic connection suspension. Auto-suspend after 5 consecutive failures to prevent account lockout. Range: 0 to infinity (typically 0-10). - `created_at` string, date-time — UTC timestamp when the connection was created. Auto-generated using shared.utilities.helpers.current_time(). Immutable after creation. Format: ISO 8601 datetime. - `updated_at` string, date-time — UTC timestamp of the most recent update to the connection. Updated automatically on any field modification. Tracks configuration changes, status updates, and credential refreshes. Format: ISO 8601 datetime. - `created_by_user_id` string, required — REQUIRED. User identifier of the user who created this connection. Used for audit trails and permission checks. Format: usr_{15-character alphanumeric}. Immutable after creation. - `metadata` object — Arbitrary key-value metadata provided by the user. Useful for tagging, categorization, and custom annotations. NOT REQUIRED - defaults to empty dictionary. Common uses: team tags, cost center codes, project identifiers. ## Other responses - `400` — Bad Request - `401` — Unauthorized - `403` — Forbidden - `404` — Not Found - `422` — Validation Error - `500` — Internal Server Error --- [API](https://skmtc.net/mixpeek/apis/mixpeek-api.md) · [All operations](https://skmtc.net/mixpeek/apis/mixpeek-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mixpeek/mixpeek-api/revisions/220a3b263fda/schema)