---
title: "Create pipeline"
method: POST
path: "/v2/pipelines"
tags: ["Pipelines"]
---

# Create pipeline

`POST /v2/pipelines`

Creates a new pipeline that continuously ingests data from a source system and sends each record to an agent for processing. Each source record creates a new agent session. A pipeline is distinct from an agent schedule, which is a recurring single execution of an agent. It is also distinct from a connector, which is a bidirectional chat integration like Slack. A pipeline is an automated, one-directional flow of all source data through an agent.

## Headers

- `Request-Timeout` integer
- `Request-Timeout-Millis` integer

## Request body

- CreatePipelineRequest — Request to create a new pipeline.
  - `key` string — The unique key for the pipeline. You can provide the key, or the platform generates one.
  - `name` string, required — The human-readable name of the pipeline.
  - `description` string
  - `source` union, required — The source system to ingest data from.
    - SharepointSourceConfiguration — Base SharePoint source configuration with all fields optional.
      - `type` string, required
      - `site_url` string, required — The full URL of the SharePoint site.
      - `tenant_id` string, required — Azure AD tenant ID.
      - `client_id` string, required — Azure AD application (client) ID for OAuth.
      - `client_secret` string, required — Azure AD client secret. Encrypted at rest and not returned in responses.
      - `drive_id` string — The ID of a specific document library. If omitted, the default document library is used.
      - `folder_path` string — Optional folder path to scope ingestion to a subdirectory.
      - `source_record_metadata` SourceRecordMetadata — Metadata recorded on each ingested record: source-system fields, owner-attached fields, and access-control grants. It is set as the `source_record_metadata` field in each record's session metadata, where the pipeline agent can access it. When supplied on a source configuration, user-provided values take precedence over connector-derived ones. `system_metadata` and `user_metadata` entries override derived entries key by key. Derived keys not named are kept. A provided `acl_metadata` replaces ACL extraction entirely.
        - `system_metadata` object — Metadata the source system manages, such as size, timestamps, or eTag.
        - `user_metadata` object — Metadata the owner of the source object attached, such as S3 object tags or SharePoint custom columns.
        - `acl_metadata` AclMetadata — Access-control grants recorded on each ingested record. Grants are extracted from the source system or supplied via a source's `source_record_metadata.acl_metadata`. A supplied value always takes priority over extraction. Grants are independent and additive: the effective audience is the union of the user lists, the group lists, `public_access`, and `org_wide_access`. An omitted list means the source does not track that concept for the record. An empty list means it was tracked and nobody holds the grant. User and group identifiers are stored verbatim. Use the identifier form your query-time access filter presents (e.g. email addresses for users). Supply the querying user's groups at query time, because the platform does not expand group membership. Grants are recorded at ingestion. Enforcing them at query time is up to the query's access filtering.
          - `owners` string[] — User identifiers that own the record.
          - `editors` string[] — User identifiers granted edit access.
          - `commenters` string[] — User identifiers granted comment access.
          - `readers` string[] — User identifiers granted read access.
          - `group_editors` string[] — Group identifiers granted edit access.
          - `group_commenters` string[] — Group identifiers granted comment access.
          - `group_readers` string[] — Group identifiers granted read access.
          - `public_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
          - `org_wide_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
    - S3SourceConfiguration — Base S3 source configuration with all fields optional.
      - `type` string, required
      - `bucket` string, required — The S3 bucket name.
      - `prefix` string — Key prefix to scope ingestion to a subset of objects.
      - `region` string, required — The region of the S3-compatible service.
      - `endpoint_url` string — Custom endpoint URL for S3-compatible services. If omitted, defaults to AWS S3.
      - `access_key_id` string, required — AWS access key ID. Encrypted at rest and not returned in responses.
      - `secret_access_key` string, required — AWS secret access key. Encrypted at rest and not returned in responses.
      - `pos_regex` string[] — Regex patterns matched against the full object key. A key must fully match at least one pattern to be ingested. An empty list applies no positive filter.
      - `neg_regex` string[] — Regex patterns matched against the full object key. A key that fully matches any pattern is skipped.
      - `source_record_metadata` SourceRecordMetadata — Metadata recorded on each ingested record: source-system fields, owner-attached fields, and access-control grants. It is set as the `source_record_metadata` field in each record's session metadata, where the pipeline agent can access it. When supplied on a source configuration, user-provided values take precedence over connector-derived ones. `system_metadata` and `user_metadata` entries override derived entries key by key. Derived keys not named are kept. A provided `acl_metadata` replaces ACL extraction entirely.
        - `system_metadata` object — Metadata the source system manages, such as size, timestamps, or eTag.
        - `user_metadata` object — Metadata the owner of the source object attached, such as S3 object tags or SharePoint custom columns.
        - `acl_metadata` AclMetadata — Access-control grants recorded on each ingested record. Grants are extracted from the source system or supplied via a source's `source_record_metadata.acl_metadata`. A supplied value always takes priority over extraction. Grants are independent and additive: the effective audience is the union of the user lists, the group lists, `public_access`, and `org_wide_access`. An omitted list means the source does not track that concept for the record. An empty list means it was tracked and nobody holds the grant. User and group identifiers are stored verbatim. Use the identifier form your query-time access filter presents (e.g. email addresses for users). Supply the querying user's groups at query time, because the platform does not expand group membership. Grants are recorded at ingestion. Enforcing them at query time is up to the query's access filtering.
          - `owners` string[] — User identifiers that own the record.
          - `editors` string[] — User identifiers granted edit access.
          - `commenters` string[] — User identifiers granted comment access.
          - `readers` string[] — User identifiers granted read access.
          - `group_editors` string[] — Group identifiers granted edit access.
          - `group_commenters` string[] — Group identifiers granted comment access.
          - `group_readers` string[] — Group identifiers granted read access.
          - `public_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
          - `org_wide_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
    - GoogleDriveSourceConfiguration — Base Google Drive source configuration. The `scopes` list enumerates one or more starting points. Each entry is either a shared drive (`shared`) or a single user's My Drive accessed via domain-wide delegation (`my_drive`). The service account configured here must have read access to every listed drive. It must also be authorized for each impersonated user.
      - `type` string, required
      - `scopes` GoogleDriveScope[], required — Drive starting points to ingest under this pipeline.
        - union — Which Drive content to ingest, discriminated by `type`. `shared` ingests from a Google Workspace shared drive where the service account is a member. `my_drive` ingests from a single user's My Drive via domain-wide delegation.
          - SharedGoogleDriveScope — Ingests from a Google Workspace shared drive. Add the service account (`client_email`) as a member of the shared drive, or of the linked folder, with at least Viewer access. This scope does not require domain-wide delegation.
            - `type` string, required
            - `url` string, required — URL of the folder to ingest. Use a shared drive's root URL (`https://drive.google.com/drive/folders/<drive_id>`) to enumerate the entire drive. Use any subfolder URL (`https://drive.google.com/drive/folders/<folder_id>`) to scope ingestion to that subtree. The connector walks every descendant subfolder beneath the linked target.
          - MyGoogleDriveScope — Ingests from a single user's My Drive via Google Workspace domain-wide delegation. Requires the Workspace admin to authorize the service account's client ID with the `drive.readonly` OAuth scope.
            - `type` string, required
            - `subject_email` string, required — User email address whose My Drive the service account impersonates.
            - `url` string — Optional URL of a folder within the user's My Drive (`https://drive.google.com/drive/folders/<folder_id>`) to narrow ingestion to that subtree. The connector walks all descendant subfolders and indexes every file within. If omitted, every accessible file in the user's My Drive is enumerated.
      - `client_email` string, required — The Google Cloud service account's email address (the `client_email` field of the service account JSON key). For `shared` scopes, grant this email read access to each drive or folder you want to ingest.
      - `private_key` string, required — The service account's PEM-formatted RSA private key (the `private_key` field of the service account JSON key, including the `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----` markers and embedded newlines). Encrypted at rest and not returned in responses.
    - BoxSourceConfiguration — Base Box source configuration. Ingests files from a Box enterprise starting at `folder_id`. The connector walks every descendant subfolder. It inherits each item's Box collaborations as document-level ACL.
      - `type` string, required
      - `folder_id` string — The Box folder ID to ingest from. A Box folder URL (`https://app.box.com/folder/<id>`) is also accepted, in which case the ID is taken from the URL. Defaults to the enterprise root (ID `"0"`).
      - `auth` BoxCcgAuth, required — Box server-to-server authentication via the Client Credentials Grant. Authorize the app for the enterprise in the Box Admin Console.
        - `type` string, required
        - `client_id` string, required — Box app client ID.
        - `client_secret` string, required — Box app client secret. Encrypted at rest and not returned in responses.
        - `enterprise_id` string, required — Box enterprise ID the app is authorized for.
    - WolkenKbSourceConfiguration — Base Wolken ServiceDesk knowledge-base source configuration. Ingests knowledge-base articles through the API family selected by `kb_api`. Each article's audience attributes are carried as document metadata for attribute-based filtering. Requires Wolken credentials with read access to the knowledge-base listing and article-detail endpoints of the selected family. With `data_api`, the Wolken server filters articles by the configured status, validation, and level filters. Incremental sync with `data_api` also requires the listing to support update-time filters. With `kb_module`, articles are listed per category. Update-time bounds are applied after listing. Supported filters vary by Wolken deployment.
      - `type` string, required
      - `api_endpoint` string, required — Base URL of the Wolken API.
      - `domain` string, required — The value of the `domain` header issued with your Wolken API credentials. Wolken deployments vary in whether this is a tenant name or a full hostname. Use the exact value from your credential handoff.
      - `kb_api` 'data_api' | 'kb_module' — The Wolken API family used to read the knowledge base. `data_api` reads through the provisioned data API endpoint and supports server-side update-time filters. `kb_module` reads through the Knowledge Base module REST API. That API lists articles by category and offers no server-side filters, so update detection happens after listing. The Wolken administrator determines which family a deployment provisions.
      - `category_ids` string[] — Wolken knowledge-base category IDs to restrict ingestion to. If unset, articles of every category are ingested. Category IDs are specific to your Wolken deployment and can be read from the Knowledge Base module category listing. Applies to `kb_module` only.
      - `article_ids` string[] — Wolken knowledge-base article IDs to restrict ingestion to. If unset, all articles matching the other configured filters are ingested.
      - `article_statuses` WolkenArticleStatus[] — Article lifecycle statuses to ingest. If unset, articles of every status are ingested. An article Wolken reports no status for is always ingested.
      - `status_id` integer — Lifecycle status to filter articles by. If unset, articles of every lifecycle status are ingested. Status IDs are specific to your Wolken deployment. Applies to `data_api` only.
      - `validation_status_id` integer — Validation status to filter articles by. If unset, articles of every validation status are ingested. Validation status IDs are specific to your Wolken deployment. Applies to `data_api` only.
      - `level_id` integer — Audience level to filter articles by. If unset, articles of every audience level are ingested. Level IDs are specific to your Wolken deployment. Applies to `data_api` only.
      - `update_upper_bound_operator` 'lt' | 'lte' — Operator used for the upper bound when filtering articles by when they were last updated. `lt` is strict less-than and `lte` is less-than-or-equal. Applies to `data_api` only.
      - `article_url_template` string — Template for each article's public URL. The `{article_number}` placeholder is replaced with the article number. When omitted, the URL returned by the Wolken listing is used. If the listing returns no URL, the article has no public URL recorded.
      - `client_id` string, required — Wolken OAuth client ID.
      - `service_account` string, required — Wolken service account the data API calls run as.
      - `auth_code` string, required — The auth code Wolken issues for the instance, used with `refresh_token` to obtain access tokens. Encrypted at rest and not returned in responses.
      - `refresh_token` string, required — Long-lived refresh token exchanged for access tokens. Encrypted at rest and not returned in responses.
    - WolkenFormsSourceConfiguration — Base Wolken ServiceDesk forms source configuration. Ingests the end-user-facing service catalog forms of a Wolken ServiceDesk instance, covering both incident forms and service request forms. Each form becomes one document carrying the form's title, description, and category. When `include_form_details` is true, the document also carries the form's FAQs, field definitions, and option values. Requires Wolken credentials with read access to the service catalog listing endpoint. When `include_form_details` is true, the credentials also need read access to the form metadata, special instructions, and lookup value endpoints.
      - `type` string, required — Discriminator value identifying this source configuration as `wolken_forms`.
      - `base_url` string, required — Base URL of the Wolken API.
      - `domain` string, required — The value of the `domain` header issued with your Wolken API credentials. Wolken deployments vary in whether this is a tenant name or a full hostname. Use the exact value from your credential handoff.
      - `include_form_details` boolean — Whether each form document includes the form's FAQs, field definitions, field help text, and the selectable values of its dropdown and lookup fields. When false, documents are built from the service catalog listing alone.
      - `item_url_template` string — Template for the portal URL of a form. The `{item_id}` placeholder is replaced with the form's item ID. When omitted, documents carry no URL metadata.
      - `client_id` string, required — Wolken OAuth client ID.
      - `service_account` string, required — Wolken service account the API calls run as. Sent as a header on API requests.
      - `auth_code` string, required — The auth code Wolken issues for the instance, used with `refresh_token` to obtain access tokens. Encrypted at rest and not returned in responses.
      - `refresh_token` string, required — Long-lived refresh token exchanged for access tokens. Encrypted at rest and not returned in responses.
    - ConfluenceSourceConfiguration — Base Confluence source configuration. Ingests pages from the configured spaces and inherits each page's read restrictions and space permissions as document ACL. Works against both Confluence Cloud and Data Center.
      - `type` string, required
      - `base_url` string, required — Base URL of your Confluence instance, e.g. `https://your-domain.atlassian.net` for Cloud or `https://confluence.example.com` for Data Center. The Cloud `/wiki` context path may be included or omitted; the platform adds it when omitted.
      - `deployment` 'cloud' | 'data_center' — Confluence deployment type. `cloud` for Atlassian-hosted Confluence Cloud. `data_center` for a self-hosted Confluence Data Center or legacy Server instance.
      - `space_keys` string[] — Space keys to ingest. When omitted, every global space the authenticated account can read is ingested.
      - `auth` union — Authentication configuration for connecting to a remote service.
        - object — Bearer token authentication
          - `type` string, required — Must be "bearer" for bearer token auth
          - `token` string, required — The bearer token to use for authentication
        - object — Custom header-based authentication
          - `type` string, required — Must be "header" for header-based auth
          - `header` string, required — The header name to use (e.g. x-api-key)
          - `value` string, required — The header value to use
        - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
          - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
          - `client_id` string, required — The OAuth2 client ID.
          - `client_secret` string, required — The OAuth2 client secret.
          - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
          - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
    - FluidtopicsSourceConfiguration — Base Fluid Topics source configuration. Ingests content from a Fluid Topics tenant via the Knowledge Hub REST API. `content_scope` selects which layer to ingest: - `documents`: standalone items. - `topics`: the sections inside maps, each carrying its parent map's identity and classification. - `maps`: whole maps, one record each. Every record carries its full Fluid Topics metadata, including classification and entitlement fields, as document metadata for attribute-based filtering and access control. Requires a Fluid Topics API key with read access to the configured content.
      - `type` string
      - `base_url` string, required — Base URL of the Fluid Topics tenant.
      - `api_key` string, required — Fluid Topics API key, sent as an HTTP Bearer authorization token. Generate one in the Fluid Topics administration interface under Integrations, API keys — see https://doc.fluidtopics.com/r/Fluid-Topics-Configuration-and-Administration-Guide/Configure-a-Fluid-Topics-tenant/Integrations/API-keys. Encrypted at rest and not returned in responses.
      - `content_scope` 'documents' | 'topics' | 'maps' — Which Fluid Topics content layer to ingest: - `documents`: enumerates standalone documents. - `topics`: enumerates the topics within maps, iterating map by map so each topic carries its parent map's identity and classification. - `maps`: enumerates maps, emitting one record each. Under `maps`, `query` is ignored. `filters`, `locale`, `include_sources`, and `exclude_sources` apply, matched against each map's metadata.
      - `query` string — Search query used to select the content to ingest. Defaults to `*` (all content).
      - `locale` string — Restricts ingestion to a single content locale (Fluid Topics `contentLocale`), for example `en-US`. When unset, all locales are ingested.
      - `filters` object — Restricts ingestion to records matching the given Fluid Topics metadata, as a map of metadata key to the list of accepted values. For example `{"Disclosure": ["Public"]}` ingests only public content. When unset, no metadata filter is applied. Under `content_scope: maps` the reserved key `id` matches the map's own identifier, so `{"id": ["<map_id>"]}` restricts a run to a single map.
      - `include_sources` string[] — Fluid Topics source ids (`ft:sourceId`) to ingest. A record whose source is not listed is skipped. Applied to every ingestion path regardless of `query` or `filters`. Empty (the default) ingests every source.
      - `exclude_sources` string[] — Fluid Topics source ids (`ft:sourceId`, for example `drupal` or `khoros`) whose records are excluded from ingestion, even when they also match `include_sources`. Applied to every ingestion path regardless of `query` or `filters`. Empty (the default) excludes nothing.
      - `document_fetch_modes` object — How document bodies are fetched in the `documents` scope, as a map of MIME type to fetch mode. Keys match a document's MIME type (`ft:mimeType`) exactly (`application/pdf`), by prefix (`image/*`), or `*` as the catch-all default. Types matching no key use `extracted_text`, the behavior when this map is unset. Ignored when `content_scope` is `topics`.
    - ScimSourceConfiguration — Base SCIM source configuration. Ingests user records from a SCIM 2.0 service provider through the Users endpoint. Each user becomes one document carrying profile, group membership, and enterprise attributes as document metadata for attribute-based filtering.
      - `type` string, required
      - `base_url` string, required — Base URL of the SCIM 2.0 service provider. The Users resource path is appended automatically and must not be included.
      - `auth` union, required — Authentication configuration for connecting to a remote service.
        - object — Bearer token authentication
          - `type` string, required — Must be "bearer" for bearer token auth
          - `token` string, required — The bearer token to use for authentication
        - object — Custom header-based authentication
          - `type` string, required — Must be "header" for header-based auth
          - `header` string, required — The header name to use (e.g. x-api-key)
          - `value` string, required — The header value to use
        - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
          - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
          - `client_id` string, required — The OAuth2 client ID.
          - `client_secret` string, required — The OAuth2 client secret.
          - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
          - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
      - `filter` string — SCIM filter expression applied to the user listing, restricting which users are ingested. Requires a service provider that supports the filter query parameter. When `incremental_strategy` is `last_modified_filter`, the change window is combined with this expression.
      - `attributes` string[] — Attribute names to request from the service provider, passed as the SCIM attributes query parameter. The meta attribute is always requested so that incremental sync can read each user's meta.lastModified timestamp. When unset, the service provider returns its default attribute set. Returned attributes land in two places on each record. They are rendered into the user's profile document, which the agent converts and indexes. They also populate the record's source_record_metadata in the agent session metadata, where they are available for attribute-based filtering and for tool argument overrides. Narrowing this list therefore narrows both what is indexed and what is available as metadata. An attribute the service provider marks as returned only on request is included only when named here.
      - `incremental_strategy` 'client_side' | 'last_modified_filter' — How incremental syncs detect changed users. `last_modified_filter` pushes the change window into the SCIM filter query parameter as a `meta.lastModified` range. `client_side` enumerates every user on each run and evaluates the change window locally. Support for the filter query parameter is optional in the SCIM 2.0 specification. Set `client_side` when the service provider does not support filtering on `meta.lastModified`.
    - WebSourceConfiguration — Shared configuration for all web-based sources (sitemap, crawl, sitemap+crawl). Contains politeness settings, limits including the `max_pages` page count cap, and authentication. Mode-specific fields (urls, sitemap_urls, BFS settings) live on the page source types under `pages_source`.
      - `respect_robots_txt` boolean — If true (default and strongly recommended), the crawler honors `robots.txt` rules and `Crawl-delay` directives. Set to false only if you have explicit authorization to crawl the target site without robots.txt restrictions (e.g., your own domain).
      - `requests_per_second` number, double — Maximum sustained requests per second per host.
      - `max_concurrent` integer — Maximum concurrent in-flight requests per host.
      - `max_pages` integer — Hard cap on the number of pages fetched per run. The default is a safety rail to prevent unbounded crawls. In BFS mode, `max_depth` alone is not a meaningful bound on high-branching-factor sites like wikis. In sitemap mode, a `<sitemap-index>` can chain 50,000 child sitemaps × 50,000 URLs each per the sitemaps.org limits.
      - `js_rendering` boolean — If true, a headless browser renders each page. This is slower, but required for JavaScript-heavy SPAs. Defaults to false. Turn it on if pages return empty content without rendering.
      - `user_agent` string — User-Agent header sent with every request. Must not contain CR or LF.
      - `max_page_bytes` integer — Maximum page body bytes to download. Larger pages are truncated. The 64 KB floor prevents accidentally capping every page below the size of a typical HTML document. The 100 MiB ceiling is far above any plausible single-page payload.
      - `excluded_content_types` string[] — MIME types to skip in addition to always-skipped media types (image/*, video/*, audio/*). Useful to skip e.g. application/zip if a site links to large archives.
      - `auth` union — Authentication configuration for connecting to a remote service.
        - object — Bearer token authentication
          - `type` string, required — Must be "bearer" for bearer token auth
          - `token` string, required — The bearer token to use for authentication
        - object — Custom header-based authentication
          - `type` string, required — Must be "header" for header-based auth
          - `header` string, required — The header name to use (e.g. x-api-key)
          - `value` string, required — The header value to use
        - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
          - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
          - `client_id` string, required — The OAuth2 client ID.
          - `client_secret` string, required — The OAuth2 client secret.
          - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
          - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
      - `source_record_metadata` SourceRecordMetadata — Metadata recorded on each ingested record: source-system fields, owner-attached fields, and access-control grants. It is set as the `source_record_metadata` field in each record's session metadata, where the pipeline agent can access it. When supplied on a source configuration, user-provided values take precedence over connector-derived ones. `system_metadata` and `user_metadata` entries override derived entries key by key. Derived keys not named are kept. A provided `acl_metadata` replaces ACL extraction entirely.
        - `system_metadata` object — Metadata the source system manages, such as size, timestamps, or eTag.
        - `user_metadata` object — Metadata the owner of the source object attached, such as S3 object tags or SharePoint custom columns.
        - `acl_metadata` AclMetadata — Access-control grants recorded on each ingested record. Grants are extracted from the source system or supplied via a source's `source_record_metadata.acl_metadata`. A supplied value always takes priority over extraction. Grants are independent and additive: the effective audience is the union of the user lists, the group lists, `public_access`, and `org_wide_access`. An omitted list means the source does not track that concept for the record. An empty list means it was tracked and nobody holds the grant. User and group identifiers are stored verbatim. Use the identifier form your query-time access filter presents (e.g. email addresses for users). Supply the querying user's groups at query time, because the platform does not expand group membership. Grants are recorded at ingestion. Enforcing them at query time is up to the query's access filtering.
          - `owners` string[] — User identifiers that own the record.
          - `editors` string[] — User identifiers granted edit access.
          - `commenters` string[] — User identifiers granted comment access.
          - `readers` string[] — User identifiers granted read access.
          - `group_editors` string[] — Group identifiers granted edit access.
          - `group_commenters` string[] — Group identifiers granted comment access.
          - `group_readers` string[] — Group identifiers granted read access.
          - `public_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
          - `org_wide_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
      - `type` string, required
      - `pages_source` union, required — How a `web` source discovers URLs, discriminated by `type`: - `sitemap`: reads sitemap.xml only. - `crawl`: does BFS link-following from seed URLs. - `sitemap_crawl`: reads a sitemap and crawls outward from its entries.
        - SitemapPagesSource — Reads pages enumerated by one or more sitemaps (`sitemap.xml`, `sitemap-index.xml`, or gzipped variants). Per-URL change detection uses the sitemap's `<lastmod>`. The crawler does not follow links.
          - `type` string, required
          - `sitemap_urls` WebUrl[], required — Sitemap URLs to read. At least one is required.
        - CrawlPagesSource — Shared BFS settings for crawl-mode page-source variants (CrawlPagesSource, SitemapCrawlPagesSource). Contains depth and URL filters. Seed lists differ per variant and live on the leaf types.
          - `max_depth` integer — Maximum link-following depth from each seed URL.
          - `same_domain_only` boolean — If true, the crawler does not follow links outside the seed URL's registered domain.
          - `pos_regex` string[] — Regex patterns matched against the full URL. A URL must fully match at least one pattern to be crawled. An empty list applies no positive filter.
          - `neg_regex` string[] — Regex patterns matched against the full URL. A URL that fully matches any pattern is skipped.
          - `type` string, required
          - `urls` WebUrl[], required — Seed URLs for link-following crawl. BFS expands from these.
        - SitemapCrawlPagesSource — Shared BFS settings for crawl-mode page-source variants (CrawlPagesSource, SitemapCrawlPagesSource). Contains depth and URL filters. Seed lists differ per variant and live on the leaf types.
          - `max_depth` integer — Maximum link-following depth from each seed URL.
          - `same_domain_only` boolean — If true, the crawler does not follow links outside the seed URL's registered domain.
          - `pos_regex` string[] — Regex patterns matched against the full URL. A URL must fully match at least one pattern to be crawled. An empty list applies no positive filter.
          - `neg_regex` string[] — Regex patterns matched against the full URL. A URL that fully matches any pattern is skipped.
          - `type` string, required
          - `urls` WebUrl[] — Optional additional seed URLs. Sitemap entries are also used as crawl seeds.
          - `sitemap_urls` WebUrl[], required — Sitemap URLs to read; their entries become the initial crawl seeds.
  - `trigger` union, required — Defines when the pipeline runs.
    - CronTriggerConfiguration — Runs the pipeline on a cron schedule.
      - `type` string, required
      - `expression` string, required — A 5-field cron expression (minute, hour, day-of-month, month, day-of-week) in UTC.
    - IntervalTriggerConfiguration — Runs the pipeline at a fixed interval.
      - `type` string, required
      - `duration` string, required — An ISO-8601 duration specifying the interval between runs.
    - ManualTriggerConfiguration — Runs the pipeline only when triggered manually via the trigger endpoint. There is no automatic scheduling.
      - `type` string, required
  - `transform` PipelineTransform, required — Processes source data using an agent. Each run creates one or more agent sessions and sends source records as input for the agent to act on per its instructions.
    - `type` string, required — The type of transform.
    - `agent_key` string, required — A unique key that identifies an agent.
    - `verification` union — Defines how to verify whether a pipeline record was successfully processed by the worker agent. Two strategies are supported: a lightweight condition expression or a full judge agent.
      - ConditionVerification — Verifies the worker agent's output using a UserFn condition expression. The expression is evaluated against the worker agent's session context (output, tools, session metadata). If the expression evaluates to true, the record is considered successfully processed.
        - `type` string, required
        - `expression` string, required — A UserFn boolean expression evaluated against the worker agent's session context.
        - `reason_expression` string — Optional UserFn expression, evaluated against the same session context as `expression`, whose string result explains the outcome. Evaluated on both pass and fail and surfaced as the record event's `reason` (see `AgentVerification` for the equivalent judge-agent field). When the condition fails and this is omitted or evaluates to empty, a default message referencing the expression is used.
      - AgentVerification — Verifies the worker agent's output using a separate judge agent. The judge agent receives a summary of the worker agent's session and must produce a structured output with `{ "success": boolean, "reason": string }`. The judge agent must be configured with a structured output parser matching this schema. When the judge agent's own `run_condition` evaluates to false nothing verifies the record. The record completes with `skipped` true and a `reason` saying verification did not run, and it counts toward `records_skipped` rather than `records_processed`.
        - `type` string, required
        - `agent_key` string, required — A unique key that identifies an agent.
  - `sync_mode` 'incremental' | 'full_refresh' — How the pipeline syncs data from the source. - `incremental`: Only process new or changed records since the last watermark. - `full_refresh`: Process all records from the source on each run.
  - `enabled` boolean
  - `metadata` object

## Response `201`

The created pipeline.

- Pipeline — A pipeline is an automated data flow. It sends all data from a source system to an agent for the agent to act on per its instructions. Each source record creates a new agent session. The agent has full autonomy to decide what to do with each piece of data — index it into a corpus, extract structured information, route it, or discard it. This is distinct from an **agent schedule**, which is a recurring single execution of an agent with a fixed message. A schedule creates one session per trigger. A pipeline creates one session per source record. This is also distinct from an **agent connector** (e.g. Slack), which is a bidirectional integration for interactive chat with an agent. A pipeline is a one-directional automated flow of all data through an agent.
  - `key` string, required — The unique key for the pipeline. You can provide the key, or the platform generates one.
  - `name` string, required — The human-readable name of the pipeline.
  - `description` string — A description of what this pipeline does.
  - `source` union, required — The source system to ingest data from.
    - SharepointSourceConfiguration — Base SharePoint source configuration with all fields optional.
      - `type` string, required
      - `site_url` string, required — The full URL of the SharePoint site.
      - `tenant_id` string, required — Azure AD tenant ID.
      - `client_id` string, required — Azure AD application (client) ID for OAuth.
      - `client_secret` string, required — Azure AD client secret. Encrypted at rest and not returned in responses.
      - `drive_id` string — The ID of a specific document library. If omitted, the default document library is used.
      - `folder_path` string — Optional folder path to scope ingestion to a subdirectory.
      - `source_record_metadata` SourceRecordMetadata — Metadata recorded on each ingested record: source-system fields, owner-attached fields, and access-control grants. It is set as the `source_record_metadata` field in each record's session metadata, where the pipeline agent can access it. When supplied on a source configuration, user-provided values take precedence over connector-derived ones. `system_metadata` and `user_metadata` entries override derived entries key by key. Derived keys not named are kept. A provided `acl_metadata` replaces ACL extraction entirely.
        - `system_metadata` object — Metadata the source system manages, such as size, timestamps, or eTag.
        - `user_metadata` object — Metadata the owner of the source object attached, such as S3 object tags or SharePoint custom columns.
        - `acl_metadata` AclMetadata — Access-control grants recorded on each ingested record. Grants are extracted from the source system or supplied via a source's `source_record_metadata.acl_metadata`. A supplied value always takes priority over extraction. Grants are independent and additive: the effective audience is the union of the user lists, the group lists, `public_access`, and `org_wide_access`. An omitted list means the source does not track that concept for the record. An empty list means it was tracked and nobody holds the grant. User and group identifiers are stored verbatim. Use the identifier form your query-time access filter presents (e.g. email addresses for users). Supply the querying user's groups at query time, because the platform does not expand group membership. Grants are recorded at ingestion. Enforcing them at query time is up to the query's access filtering.
          - `owners` string[] — User identifiers that own the record.
          - `editors` string[] — User identifiers granted edit access.
          - `commenters` string[] — User identifiers granted comment access.
          - `readers` string[] — User identifiers granted read access.
          - `group_editors` string[] — Group identifiers granted edit access.
          - `group_commenters` string[] — Group identifiers granted comment access.
          - `group_readers` string[] — Group identifiers granted read access.
          - `public_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
          - `org_wide_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
    - S3SourceConfiguration — Base S3 source configuration with all fields optional.
      - `type` string, required
      - `bucket` string, required — The S3 bucket name.
      - `prefix` string — Key prefix to scope ingestion to a subset of objects.
      - `region` string, required — The region of the S3-compatible service.
      - `endpoint_url` string — Custom endpoint URL for S3-compatible services. If omitted, defaults to AWS S3.
      - `access_key_id` string, required — AWS access key ID. Encrypted at rest and not returned in responses.
      - `secret_access_key` string, required — AWS secret access key. Encrypted at rest and not returned in responses.
      - `pos_regex` string[] — Regex patterns matched against the full object key. A key must fully match at least one pattern to be ingested. An empty list applies no positive filter.
      - `neg_regex` string[] — Regex patterns matched against the full object key. A key that fully matches any pattern is skipped.
      - `source_record_metadata` SourceRecordMetadata — Metadata recorded on each ingested record: source-system fields, owner-attached fields, and access-control grants. It is set as the `source_record_metadata` field in each record's session metadata, where the pipeline agent can access it. When supplied on a source configuration, user-provided values take precedence over connector-derived ones. `system_metadata` and `user_metadata` entries override derived entries key by key. Derived keys not named are kept. A provided `acl_metadata` replaces ACL extraction entirely.
        - `system_metadata` object — Metadata the source system manages, such as size, timestamps, or eTag.
        - `user_metadata` object — Metadata the owner of the source object attached, such as S3 object tags or SharePoint custom columns.
        - `acl_metadata` AclMetadata — Access-control grants recorded on each ingested record. Grants are extracted from the source system or supplied via a source's `source_record_metadata.acl_metadata`. A supplied value always takes priority over extraction. Grants are independent and additive: the effective audience is the union of the user lists, the group lists, `public_access`, and `org_wide_access`. An omitted list means the source does not track that concept for the record. An empty list means it was tracked and nobody holds the grant. User and group identifiers are stored verbatim. Use the identifier form your query-time access filter presents (e.g. email addresses for users). Supply the querying user's groups at query time, because the platform does not expand group membership. Grants are recorded at ingestion. Enforcing them at query time is up to the query's access filtering.
          - `owners` string[] — User identifiers that own the record.
          - `editors` string[] — User identifiers granted edit access.
          - `commenters` string[] — User identifiers granted comment access.
          - `readers` string[] — User identifiers granted read access.
          - `group_editors` string[] — Group identifiers granted edit access.
          - `group_commenters` string[] — Group identifiers granted comment access.
          - `group_readers` string[] — Group identifiers granted read access.
          - `public_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
          - `org_wide_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
    - GoogleDriveSourceConfiguration — Base Google Drive source configuration. The `scopes` list enumerates one or more starting points. Each entry is either a shared drive (`shared`) or a single user's My Drive accessed via domain-wide delegation (`my_drive`). The service account configured here must have read access to every listed drive. It must also be authorized for each impersonated user.
      - `type` string, required
      - `scopes` GoogleDriveScope[], required — Drive starting points to ingest under this pipeline.
        - union — Which Drive content to ingest, discriminated by `type`. `shared` ingests from a Google Workspace shared drive where the service account is a member. `my_drive` ingests from a single user's My Drive via domain-wide delegation.
          - SharedGoogleDriveScope — Ingests from a Google Workspace shared drive. Add the service account (`client_email`) as a member of the shared drive, or of the linked folder, with at least Viewer access. This scope does not require domain-wide delegation.
            - `type` string, required
            - `url` string, required — URL of the folder to ingest. Use a shared drive's root URL (`https://drive.google.com/drive/folders/<drive_id>`) to enumerate the entire drive. Use any subfolder URL (`https://drive.google.com/drive/folders/<folder_id>`) to scope ingestion to that subtree. The connector walks every descendant subfolder beneath the linked target.
          - MyGoogleDriveScope — Ingests from a single user's My Drive via Google Workspace domain-wide delegation. Requires the Workspace admin to authorize the service account's client ID with the `drive.readonly` OAuth scope.
            - `type` string, required
            - `subject_email` string, required — User email address whose My Drive the service account impersonates.
            - `url` string — Optional URL of a folder within the user's My Drive (`https://drive.google.com/drive/folders/<folder_id>`) to narrow ingestion to that subtree. The connector walks all descendant subfolders and indexes every file within. If omitted, every accessible file in the user's My Drive is enumerated.
      - `client_email` string, required — The Google Cloud service account's email address (the `client_email` field of the service account JSON key). For `shared` scopes, grant this email read access to each drive or folder you want to ingest.
      - `private_key` string, required — The service account's PEM-formatted RSA private key (the `private_key` field of the service account JSON key, including the `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----` markers and embedded newlines). Encrypted at rest and not returned in responses.
    - BoxSourceConfiguration — Base Box source configuration. Ingests files from a Box enterprise starting at `folder_id`. The connector walks every descendant subfolder. It inherits each item's Box collaborations as document-level ACL.
      - `type` string, required
      - `folder_id` string — The Box folder ID to ingest from. A Box folder URL (`https://app.box.com/folder/<id>`) is also accepted, in which case the ID is taken from the URL. Defaults to the enterprise root (ID `"0"`).
      - `auth` BoxCcgAuth, required — Box server-to-server authentication via the Client Credentials Grant. Authorize the app for the enterprise in the Box Admin Console.
        - `type` string, required
        - `client_id` string, required — Box app client ID.
        - `client_secret` string, required — Box app client secret. Encrypted at rest and not returned in responses.
        - `enterprise_id` string, required — Box enterprise ID the app is authorized for.
    - WolkenKbSourceConfiguration — Base Wolken ServiceDesk knowledge-base source configuration. Ingests knowledge-base articles through the API family selected by `kb_api`. Each article's audience attributes are carried as document metadata for attribute-based filtering. Requires Wolken credentials with read access to the knowledge-base listing and article-detail endpoints of the selected family. With `data_api`, the Wolken server filters articles by the configured status, validation, and level filters. Incremental sync with `data_api` also requires the listing to support update-time filters. With `kb_module`, articles are listed per category. Update-time bounds are applied after listing. Supported filters vary by Wolken deployment.
      - `type` string, required
      - `api_endpoint` string, required — Base URL of the Wolken API.
      - `domain` string, required — The value of the `domain` header issued with your Wolken API credentials. Wolken deployments vary in whether this is a tenant name or a full hostname. Use the exact value from your credential handoff.
      - `kb_api` 'data_api' | 'kb_module' — The Wolken API family used to read the knowledge base. `data_api` reads through the provisioned data API endpoint and supports server-side update-time filters. `kb_module` reads through the Knowledge Base module REST API. That API lists articles by category and offers no server-side filters, so update detection happens after listing. The Wolken administrator determines which family a deployment provisions.
      - `category_ids` string[] — Wolken knowledge-base category IDs to restrict ingestion to. If unset, articles of every category are ingested. Category IDs are specific to your Wolken deployment and can be read from the Knowledge Base module category listing. Applies to `kb_module` only.
      - `article_ids` string[] — Wolken knowledge-base article IDs to restrict ingestion to. If unset, all articles matching the other configured filters are ingested.
      - `article_statuses` WolkenArticleStatus[] — Article lifecycle statuses to ingest. If unset, articles of every status are ingested. An article Wolken reports no status for is always ingested.
      - `status_id` integer — Lifecycle status to filter articles by. If unset, articles of every lifecycle status are ingested. Status IDs are specific to your Wolken deployment. Applies to `data_api` only.
      - `validation_status_id` integer — Validation status to filter articles by. If unset, articles of every validation status are ingested. Validation status IDs are specific to your Wolken deployment. Applies to `data_api` only.
      - `level_id` integer — Audience level to filter articles by. If unset, articles of every audience level are ingested. Level IDs are specific to your Wolken deployment. Applies to `data_api` only.
      - `update_upper_bound_operator` 'lt' | 'lte' — Operator used for the upper bound when filtering articles by when they were last updated. `lt` is strict less-than and `lte` is less-than-or-equal. Applies to `data_api` only.
      - `article_url_template` string — Template for each article's public URL. The `{article_number}` placeholder is replaced with the article number. When omitted, the URL returned by the Wolken listing is used. If the listing returns no URL, the article has no public URL recorded.
      - `client_id` string, required — Wolken OAuth client ID.
      - `service_account` string, required — Wolken service account the data API calls run as.
      - `auth_code` string, required — The auth code Wolken issues for the instance, used with `refresh_token` to obtain access tokens. Encrypted at rest and not returned in responses.
      - `refresh_token` string, required — Long-lived refresh token exchanged for access tokens. Encrypted at rest and not returned in responses.
    - WolkenFormsSourceConfiguration — Base Wolken ServiceDesk forms source configuration. Ingests the end-user-facing service catalog forms of a Wolken ServiceDesk instance, covering both incident forms and service request forms. Each form becomes one document carrying the form's title, description, and category. When `include_form_details` is true, the document also carries the form's FAQs, field definitions, and option values. Requires Wolken credentials with read access to the service catalog listing endpoint. When `include_form_details` is true, the credentials also need read access to the form metadata, special instructions, and lookup value endpoints.
      - `type` string, required — Discriminator value identifying this source configuration as `wolken_forms`.
      - `base_url` string, required — Base URL of the Wolken API.
      - `domain` string, required — The value of the `domain` header issued with your Wolken API credentials. Wolken deployments vary in whether this is a tenant name or a full hostname. Use the exact value from your credential handoff.
      - `include_form_details` boolean — Whether each form document includes the form's FAQs, field definitions, field help text, and the selectable values of its dropdown and lookup fields. When false, documents are built from the service catalog listing alone.
      - `item_url_template` string — Template for the portal URL of a form. The `{item_id}` placeholder is replaced with the form's item ID. When omitted, documents carry no URL metadata.
      - `client_id` string, required — Wolken OAuth client ID.
      - `service_account` string, required — Wolken service account the API calls run as. Sent as a header on API requests.
      - `auth_code` string, required — The auth code Wolken issues for the instance, used with `refresh_token` to obtain access tokens. Encrypted at rest and not returned in responses.
      - `refresh_token` string, required — Long-lived refresh token exchanged for access tokens. Encrypted at rest and not returned in responses.
    - ConfluenceSourceConfiguration — Base Confluence source configuration. Ingests pages from the configured spaces and inherits each page's read restrictions and space permissions as document ACL. Works against both Confluence Cloud and Data Center.
      - `type` string, required
      - `base_url` string, required — Base URL of your Confluence instance, e.g. `https://your-domain.atlassian.net` for Cloud or `https://confluence.example.com` for Data Center. The Cloud `/wiki` context path may be included or omitted; the platform adds it when omitted.
      - `deployment` 'cloud' | 'data_center' — Confluence deployment type. `cloud` for Atlassian-hosted Confluence Cloud. `data_center` for a self-hosted Confluence Data Center or legacy Server instance.
      - `space_keys` string[] — Space keys to ingest. When omitted, every global space the authenticated account can read is ingested.
      - `auth` union — Authentication configuration for connecting to a remote service.
        - object — Bearer token authentication
          - `type` string, required — Must be "bearer" for bearer token auth
          - `token` string, required — The bearer token to use for authentication
        - object — Custom header-based authentication
          - `type` string, required — Must be "header" for header-based auth
          - `header` string, required — The header name to use (e.g. x-api-key)
          - `value` string, required — The header value to use
        - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
          - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
          - `client_id` string, required — The OAuth2 client ID.
          - `client_secret` string, required — The OAuth2 client secret.
          - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
          - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
    - FluidtopicsSourceConfiguration — Base Fluid Topics source configuration. Ingests content from a Fluid Topics tenant via the Knowledge Hub REST API. `content_scope` selects which layer to ingest: - `documents`: standalone items. - `topics`: the sections inside maps, each carrying its parent map's identity and classification. - `maps`: whole maps, one record each. Every record carries its full Fluid Topics metadata, including classification and entitlement fields, as document metadata for attribute-based filtering and access control. Requires a Fluid Topics API key with read access to the configured content.
      - `type` string
      - `base_url` string, required — Base URL of the Fluid Topics tenant.
      - `api_key` string, required — Fluid Topics API key, sent as an HTTP Bearer authorization token. Generate one in the Fluid Topics administration interface under Integrations, API keys — see https://doc.fluidtopics.com/r/Fluid-Topics-Configuration-and-Administration-Guide/Configure-a-Fluid-Topics-tenant/Integrations/API-keys. Encrypted at rest and not returned in responses.
      - `content_scope` 'documents' | 'topics' | 'maps' — Which Fluid Topics content layer to ingest: - `documents`: enumerates standalone documents. - `topics`: enumerates the topics within maps, iterating map by map so each topic carries its parent map's identity and classification. - `maps`: enumerates maps, emitting one record each. Under `maps`, `query` is ignored. `filters`, `locale`, `include_sources`, and `exclude_sources` apply, matched against each map's metadata.
      - `query` string — Search query used to select the content to ingest. Defaults to `*` (all content).
      - `locale` string — Restricts ingestion to a single content locale (Fluid Topics `contentLocale`), for example `en-US`. When unset, all locales are ingested.
      - `filters` object — Restricts ingestion to records matching the given Fluid Topics metadata, as a map of metadata key to the list of accepted values. For example `{"Disclosure": ["Public"]}` ingests only public content. When unset, no metadata filter is applied. Under `content_scope: maps` the reserved key `id` matches the map's own identifier, so `{"id": ["<map_id>"]}` restricts a run to a single map.
      - `include_sources` string[] — Fluid Topics source ids (`ft:sourceId`) to ingest. A record whose source is not listed is skipped. Applied to every ingestion path regardless of `query` or `filters`. Empty (the default) ingests every source.
      - `exclude_sources` string[] — Fluid Topics source ids (`ft:sourceId`, for example `drupal` or `khoros`) whose records are excluded from ingestion, even when they also match `include_sources`. Applied to every ingestion path regardless of `query` or `filters`. Empty (the default) excludes nothing.
      - `document_fetch_modes` object — How document bodies are fetched in the `documents` scope, as a map of MIME type to fetch mode. Keys match a document's MIME type (`ft:mimeType`) exactly (`application/pdf`), by prefix (`image/*`), or `*` as the catch-all default. Types matching no key use `extracted_text`, the behavior when this map is unset. Ignored when `content_scope` is `topics`.
    - ScimSourceConfiguration — Base SCIM source configuration. Ingests user records from a SCIM 2.0 service provider through the Users endpoint. Each user becomes one document carrying profile, group membership, and enterprise attributes as document metadata for attribute-based filtering.
      - `type` string, required
      - `base_url` string, required — Base URL of the SCIM 2.0 service provider. The Users resource path is appended automatically and must not be included.
      - `auth` union, required — Authentication configuration for connecting to a remote service.
        - object — Bearer token authentication
          - `type` string, required — Must be "bearer" for bearer token auth
          - `token` string, required — The bearer token to use for authentication
        - object — Custom header-based authentication
          - `type` string, required — Must be "header" for header-based auth
          - `header` string, required — The header name to use (e.g. x-api-key)
          - `value` string, required — The header value to use
        - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
          - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
          - `client_id` string, required — The OAuth2 client ID.
          - `client_secret` string, required — The OAuth2 client secret.
          - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
          - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
      - `filter` string — SCIM filter expression applied to the user listing, restricting which users are ingested. Requires a service provider that supports the filter query parameter. When `incremental_strategy` is `last_modified_filter`, the change window is combined with this expression.
      - `attributes` string[] — Attribute names to request from the service provider, passed as the SCIM attributes query parameter. The meta attribute is always requested so that incremental sync can read each user's meta.lastModified timestamp. When unset, the service provider returns its default attribute set. Returned attributes land in two places on each record. They are rendered into the user's profile document, which the agent converts and indexes. They also populate the record's source_record_metadata in the agent session metadata, where they are available for attribute-based filtering and for tool argument overrides. Narrowing this list therefore narrows both what is indexed and what is available as metadata. An attribute the service provider marks as returned only on request is included only when named here.
      - `incremental_strategy` 'client_side' | 'last_modified_filter' — How incremental syncs detect changed users. `last_modified_filter` pushes the change window into the SCIM filter query parameter as a `meta.lastModified` range. `client_side` enumerates every user on each run and evaluates the change window locally. Support for the filter query parameter is optional in the SCIM 2.0 specification. Set `client_side` when the service provider does not support filtering on `meta.lastModified`.
    - WebSourceConfiguration — Shared configuration for all web-based sources (sitemap, crawl, sitemap+crawl). Contains politeness settings, limits including the `max_pages` page count cap, and authentication. Mode-specific fields (urls, sitemap_urls, BFS settings) live on the page source types under `pages_source`.
      - `respect_robots_txt` boolean — If true (default and strongly recommended), the crawler honors `robots.txt` rules and `Crawl-delay` directives. Set to false only if you have explicit authorization to crawl the target site without robots.txt restrictions (e.g., your own domain).
      - `requests_per_second` number, double — Maximum sustained requests per second per host.
      - `max_concurrent` integer — Maximum concurrent in-flight requests per host.
      - `max_pages` integer — Hard cap on the number of pages fetched per run. The default is a safety rail to prevent unbounded crawls. In BFS mode, `max_depth` alone is not a meaningful bound on high-branching-factor sites like wikis. In sitemap mode, a `<sitemap-index>` can chain 50,000 child sitemaps × 50,000 URLs each per the sitemaps.org limits.
      - `js_rendering` boolean — If true, a headless browser renders each page. This is slower, but required for JavaScript-heavy SPAs. Defaults to false. Turn it on if pages return empty content without rendering.
      - `user_agent` string — User-Agent header sent with every request. Must not contain CR or LF.
      - `max_page_bytes` integer — Maximum page body bytes to download. Larger pages are truncated. The 64 KB floor prevents accidentally capping every page below the size of a typical HTML document. The 100 MiB ceiling is far above any plausible single-page payload.
      - `excluded_content_types` string[] — MIME types to skip in addition to always-skipped media types (image/*, video/*, audio/*). Useful to skip e.g. application/zip if a site links to large archives.
      - `auth` union — Authentication configuration for connecting to a remote service.
        - object — Bearer token authentication
          - `type` string, required — Must be "bearer" for bearer token auth
          - `token` string, required — The bearer token to use for authentication
        - object — Custom header-based authentication
          - `type` string, required — Must be "header" for header-based auth
          - `header` string, required — The header name to use (e.g. x-api-key)
          - `value` string, required — The header value to use
        - object — OAuth 2.0 client credentials authentication. The platform acquires an access token from the token endpoint before connecting to the remote service.
          - `type` string, required — Must be "oauth_client_credentials" for OAuth client credentials auth.
          - `client_id` string, required — The OAuth2 client ID.
          - `client_secret` string, required — The OAuth2 client secret.
          - `token_endpoint` string, uri, required — The OAuth2 token endpoint URL where the platform exchanges credentials for an access token.
          - `scopes` string[] — OAuth2 scopes to request when acquiring the access token.
      - `source_record_metadata` SourceRecordMetadata — Metadata recorded on each ingested record: source-system fields, owner-attached fields, and access-control grants. It is set as the `source_record_metadata` field in each record's session metadata, where the pipeline agent can access it. When supplied on a source configuration, user-provided values take precedence over connector-derived ones. `system_metadata` and `user_metadata` entries override derived entries key by key. Derived keys not named are kept. A provided `acl_metadata` replaces ACL extraction entirely.
        - `system_metadata` object — Metadata the source system manages, such as size, timestamps, or eTag.
        - `user_metadata` object — Metadata the owner of the source object attached, such as S3 object tags or SharePoint custom columns.
        - `acl_metadata` AclMetadata — Access-control grants recorded on each ingested record. Grants are extracted from the source system or supplied via a source's `source_record_metadata.acl_metadata`. A supplied value always takes priority over extraction. Grants are independent and additive: the effective audience is the union of the user lists, the group lists, `public_access`, and `org_wide_access`. An omitted list means the source does not track that concept for the record. An empty list means it was tracked and nobody holds the grant. User and group identifiers are stored verbatim. Use the identifier form your query-time access filter presents (e.g. email addresses for users). Supply the querying user's groups at query time, because the platform does not expand group membership. Grants are recorded at ingestion. Enforcing them at query time is up to the query's access filtering.
          - `owners` string[] — User identifiers that own the record.
          - `editors` string[] — User identifiers granted edit access.
          - `commenters` string[] — User identifiers granted comment access.
          - `readers` string[] — User identifiers granted read access.
          - `group_editors` string[] — Group identifiers granted edit access.
          - `group_commenters` string[] — Group identifiers granted comment access.
          - `group_readers` string[] — Group identifiers granted read access.
          - `public_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
          - `org_wide_access` 'none' | 'reader' | 'commenter' | 'editor' — Access level a grant confers on a record. `none` means the access concept applies to the record but no grant was made. New levels may be added over time. Clients should treat unknown values as opaque.
      - `type` string, required
      - `pages_source` union, required — How a `web` source discovers URLs, discriminated by `type`: - `sitemap`: reads sitemap.xml only. - `crawl`: does BFS link-following from seed URLs. - `sitemap_crawl`: reads a sitemap and crawls outward from its entries.
        - SitemapPagesSource — Reads pages enumerated by one or more sitemaps (`sitemap.xml`, `sitemap-index.xml`, or gzipped variants). Per-URL change detection uses the sitemap's `<lastmod>`. The crawler does not follow links.
          - `type` string, required
          - `sitemap_urls` WebUrl[], required — Sitemap URLs to read. At least one is required.
        - CrawlPagesSource — Shared BFS settings for crawl-mode page-source variants (CrawlPagesSource, SitemapCrawlPagesSource). Contains depth and URL filters. Seed lists differ per variant and live on the leaf types.
          - `max_depth` integer — Maximum link-following depth from each seed URL.
          - `same_domain_only` boolean — If true, the crawler does not follow links outside the seed URL's registered domain.
          - `pos_regex` string[] — Regex patterns matched against the full URL. A URL must fully match at least one pattern to be crawled. An empty list applies no positive filter.
          - `neg_regex` string[] — Regex patterns matched against the full URL. A URL that fully matches any pattern is skipped.
          - `type` string, required
          - `urls` WebUrl[], required — Seed URLs for link-following crawl. BFS expands from these.
        - SitemapCrawlPagesSource — Shared BFS settings for crawl-mode page-source variants (CrawlPagesSource, SitemapCrawlPagesSource). Contains depth and URL filters. Seed lists differ per variant and live on the leaf types.
          - `max_depth` integer — Maximum link-following depth from each seed URL.
          - `same_domain_only` boolean — If true, the crawler does not follow links outside the seed URL's registered domain.
          - `pos_regex` string[] — Regex patterns matched against the full URL. A URL must fully match at least one pattern to be crawled. An empty list applies no positive filter.
          - `neg_regex` string[] — Regex patterns matched against the full URL. A URL that fully matches any pattern is skipped.
          - `type` string, required
          - `urls` WebUrl[] — Optional additional seed URLs. Sitemap entries are also used as crawl seeds.
          - `sitemap_urls` WebUrl[], required — Sitemap URLs to read; their entries become the initial crawl seeds.
  - `trigger` union, required — Defines when the pipeline runs.
    - CronTriggerConfiguration — Runs the pipeline on a cron schedule.
      - `type` string, required
      - `expression` string, required — A 5-field cron expression (minute, hour, day-of-month, month, day-of-week) in UTC.
    - IntervalTriggerConfiguration — Runs the pipeline at a fixed interval.
      - `type` string, required
      - `duration` string, required — An ISO-8601 duration specifying the interval between runs.
    - ManualTriggerConfiguration — Runs the pipeline only when triggered manually via the trigger endpoint. There is no automatic scheduling.
      - `type` string, required
  - `transform` PipelineTransform, required — Processes source data using an agent. Each run creates one or more agent sessions and sends source records as input for the agent to act on per its instructions.
    - `type` string, required — The type of transform.
    - `agent_key` string, required — A unique key that identifies an agent.
    - `verification` union — Defines how to verify whether a pipeline record was successfully processed by the worker agent. Two strategies are supported: a lightweight condition expression or a full judge agent.
      - ConditionVerification — Verifies the worker agent's output using a UserFn condition expression. The expression is evaluated against the worker agent's session context (output, tools, session metadata). If the expression evaluates to true, the record is considered successfully processed.
        - `type` string, required
        - `expression` string, required — A UserFn boolean expression evaluated against the worker agent's session context.
        - `reason_expression` string — Optional UserFn expression, evaluated against the same session context as `expression`, whose string result explains the outcome. Evaluated on both pass and fail and surfaced as the record event's `reason` (see `AgentVerification` for the equivalent judge-agent field). When the condition fails and this is omitted or evaluates to empty, a default message referencing the expression is used.
      - AgentVerification — Verifies the worker agent's output using a separate judge agent. The judge agent receives a summary of the worker agent's session and must produce a structured output with `{ "success": boolean, "reason": string }`. The judge agent must be configured with a structured output parser matching this schema. When the judge agent's own `run_condition` evaluates to false nothing verifies the record. The record completes with `skipped` true and a `reason` saying verification did not run, and it counts toward `records_skipped` rather than `records_processed`.
        - `type` string, required
        - `agent_key` string, required — A unique key that identifies an agent.
  - `sync_mode` 'incremental' | 'full_refresh', required — How the pipeline syncs data from the source. - `incremental`: Only process new or changed records since the last watermark. - `full_refresh`: Process all records from the source on each run.
  - `watermark` PipelineWatermark — Tracks incremental sync progress. The value is opaque and source-specific (e.g. a timestamp or change token).
    - `value` string, required — The current watermark value.
    - `updated_at` string, date-time, required — When the watermark was last advanced.
  - `status` 'initializing' | 'active' | 'paused' | 'error', required — The current operational status of the pipeline.
  - `status_message` string, nullable — Human-readable status details, such as an error description.
  - `enabled` boolean, required — Whether the pipeline is enabled. Disabled pipelines do not run on their trigger schedule.
  - `metadata` object — Arbitrary key-value metadata for the pipeline.
  - `created_at` string, date-time, required — When the pipeline was created.
  - `updated_at` string, date-time — When the pipeline was last updated.

## Other responses

- `400` — The pipeline creation request is malformed or contains an invalid configuration.
- `403` — Permissions do not allow creating pipelines.
- `409` — A pipeline with this key already exists.

---

[API](https://skmtc.net/vectara/apis/vectara-rest-api-v2.md) · [All operations](https://skmtc.net/vectara/apis/vectara-rest-api-v2/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/vectara/vectara-rest-api-v2/revisions/83554494d7f8/schema)