Root model for Orchestra pipeline definitions.
Orchestra pipelines are DAGs (directed acyclic graphs) of task groups that orchestrate jobs
across external systems (dbt, Snowflake, Fivetran, Databricks, AWS, GCP, Azure, etc.).
DAG structure: The 'pipeline' dictionary contains task groups (recommended) or standalone tasks.
Each entry has a user-chosen ID that can be referenced in 'dependsOn' arrays. Task groups are
the primary mechanism for building DAGs -- dependencies between groups define execution order.
Variable system: Use ${{ expression }} syntax throughout the pipeline. Available variable types:
- ENV.*: Environment variables (e.g., ${{ ENV.MY_VAR }})
- ORCHESTRA.*: Global runtime variables -- PIPELINE_RUN_ID, CURRENT_TIME, TASK_RUN_ID
- inputs.*: Runtime inputs (e.g., ${{ inputs.my_input }})
- matrix.*: Matrix iteration values (e.g., ${{ matrix.table_name }})
- ORCHESTRA.PIPELINE_RUN_TASKS['task_id'].OUTPUTS['name']: Upstream task outputs
Built-in functions: format_date(timestamp, format, timezone), add_days(timestamp, delta), add_months(timestamp, delta),
len(list), int(value), str(value).
Trigger model: Triggers use OR-logic -- any single trigger firing (cron schedule, webhook,
pipeline event, or sensor) starts a pipeline run.
Pipeline run states:
Created -> Running -> Succeeded | Failed | Warning | Cancelling -> Cancelled.
Task run states:
Created -> Queued -> Running -> Succeeded | Failed | Skipped | Cancelling -> Cancelled.
Key constraints: Task IDs must be unique across the entire pipeline. Dependencies must not be
circular. JSON keys use snake_case in YAML but are converted to camelCase in the API (except
in 'parameters' blocks and 'inputs').