List Storage Connections

Supported external storage providers for ingestion and sync.

Mixpeek can connect to external storage providers to automatically ingest objects and keep them synchronized with your namespaces.

Providers: GOOGLE_DRIVE: Google Drive and Google Workspace shared drives. - Authentication: Service account or OAuth2 - Features: Shared drive support, real-time sync, metadata preservation - Use cases: Marketing assets, team documents, knowledge bases - Limitations: Rate limits apply (10,000 requests/100 seconds per user)

S3: Amazon S3 and S3-compatible storage (MinIO, DigitalOcean Spaces, etc).
    - Authentication: Access keys or IAM role assumption
    - Features: Bucket notifications, prefix filtering, versioning support
    - Use cases: Data lakes, video archives, ML datasets, backups
    - Limitations: IAM role assumption preferred over access keys

SNOWFLAKE: Snowflake data warehouse tables.
    - Authentication: Key pair or username/password
    - Features: Incremental sync via watermarks, row-level mapping, schema introspection
    - Use cases: Customer data tables, product catalogs, transaction logs, metadata tables
    - Limitations: Each row becomes one object; large tables require incremental column

SHAREPOINT: Microsoft SharePoint and OneDrive for Business.
    - Authentication: Azure AD OAuth2 (client credentials or delegated)
    - Features: Site/drive selection, folder sync, delta queries for incremental sync
    - Use cases: Enterprise documents, team collaboration files, compliance archives
    - Limitations: Requires Azure AD app registration; throttling limits apply

Connection Requirements: - Valid credentials with read access to target files/buckets - Network connectivity from Mixpeek infrastructure - Appropriate IAM policies or share permissions configured

Examples: - Use GOOGLE_DRIVE for syncing team marketing materials - Use S3 for ingesting video archives from data lakes - Use S3 with IAM role for secure production deployments - Use SHAREPOINT for syncing enterprise SharePoint document libraries

TIGRIS: Tigris Data globally distributed object storage (S3-compatible).
    - Authentication: Access keys (same format as S3)
    - Features: S3-compatible API, global distribution, zero egress fees
    - Use cases: Globally distributed media, low-latency content delivery
    - Endpoint: https://fly.storage.tigris.dev

POSTGRESQL: PostgreSQL relational database.
    - Authentication: Username/password
    - Features: SQL queries, incremental sync via watermarks, row-level mapping
    - Use cases: Customer data tables, product catalogs, transaction logs
    - Limitations: Each row becomes one object; large tables require incremental column

INSTAGRAM: Instagram Business/Creator accounts via Meta Graph API.
    - Authentication: OAuth 2.0 via Meta Developer Console
    - Features: Media sync (posts, reels, carousels), engagement metrics, captions
    - Use cases: Social media content analysis, talent scouting, competitor monitoring
    - Limitations: Requires Instagram Business/Creator account; long-lived tokens (60 days) need refresh

TIKTOK: TikTok accounts via TikTok Content API.
    - Authentication: OAuth 2.0 via TikTok Login Kit
    - Features: Video sync, engagement metrics (likes, views, shares), descriptions
    - Use cases: Social media content analysis, creator discovery, trend monitoring
    - Limitations: Access tokens expire in 24h (refresh_token flow); rate limits apply

RSS: RSS/Atom feed entries.
    - Authentication: Optional HTTP headers (most feeds are public)
    - Features: Entry-level sync with title, author, categories, content
    - Use cases: News monitoring, blog ingestion, content aggregation
    - Limitations: No pagination; feeds are fetched in full each poll

HTTP_API: Arbitrary REST/HTTP JSON APIs.
    - Authentication: Optional HTTP headers (API keys, Bearer tokens, etc.)
    - Features: Configurable JSONPath to items array, dedup via item ID field,
      incremental sync via timestamp field, GET or POST methods, JSON or JSONL responses
    - Use cases: Public APIs (Hacker News, GitHub), private APIs (Stripe, internal services),
      any JSON endpoint that returns a list of items
    - Limitations: No built-in pagination; API is fetched in full each poll

BOX: Box cloud content management and file sharing.
    - Authentication: OAuth 2.0 (JWT or Client Credentials Grant with CCG)
    - Features: Folder sync, enterprise content management, metadata, versioning
    - Use cases: Enterprise document management, compliance archives, collaboration files
    - Limitations: Rate limits apply (10 API calls per second per user)

BRIGHTDATA: BrightData web data platform for dataset collection and web scraping.
    - Authentication: API token from BrightData dashboard
    - Features: Pre-built datasets (LinkedIn, Amazon, etc.), custom scrapers,
      scheduled collection, geo-targeting, output format selection
    - Use cases: Competitor monitoring, market research, lead generation,
      e-commerce pricing, social media data ingestion
    - Limitations: Dataset availability depends on subscription; rate limits vary

BACKBLAZE: Backblaze B2 Cloud Storage (S3-compatible object storage).
    - Authentication: Application key ID + application key from Backblaze console
    - Features: S3-compatible API, auto-discovers regional endpoint via B2 auth,
      full bucket/prefix sync, include/exclude patterns, modified_since filtering
    - Use cases: Cold storage sync, media archives, backup ingestion
    - Limitations: Buckets are region-specific; key must have listFiles/readFiles

MUX: Mux video infrastructure platform.
    - Authentication: Access token ID + access token secret (HTTP Basic Auth)
    - Features: Video asset listing, static rendition download (MP4), metadata sync
    - Use cases: Video library ingestion, media asset processing, content analysis
    - Limitations: Assets must be in 'ready' status; static renditions must be enabled

EMAIL: Inbound email connector (push-based).
    - Authentication: Per-connection webhook signing secret (HMAC-SHA256)
    - Features: MIME parsing, attachment extraction, sender allowlist, raw .eml archival
    - Use cases: Document intake via email forwarding, compliance mailbox, support inbox
    - Limitations: Requires external inbound email service (SES/Postmark/CloudMailin)

SUPABASE: Supabase Storage (S3-compatible object storage built on open source).
    - Authentication: Dedicated S3 access keys (recommended) OR service_role JWT as
      session token (access_key_id=project_ref, secret=anon_key, session_token=service_role)
    - Features: S3-compatible API, per-project endpoint auto-derived from project_ref,
      bucket-scoped sync, include/exclude patterns, modified_since incremental sync
    - Use cases: Syncing user-uploaded assets from Supabase-backed apps, unified
      search across Supabase Storage + other sources
    - Limitations: Endpoint is project-specific (https://<ref>.storage.supabase.co);
      S3 keys must be generated from the Supabase dashboard (Storage → S3 Access Keys)

Enumeration of task statuses for tracking asynchronous operations.

Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution.

Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces)

Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted

ACTIVE: Entity is active and operational (for buckets, collections, etc.)
ARCHIVED: Entity has been archived
SUSPENDED: Entity has been temporarily suspended

Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state.

Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately.

Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling

OPTIONAL. Filter by active flag. True: Returns only active connections (status=ACTIVE). False: Returns only inactive connections (SUSPENDED/FAILED/ARCHIVED). If not provided, returns connections of all active states.

Total number of connections matching the filters (before pagination). Use this to calculate total pages and display pagination controls.