--- title: "Get dashboard lineage groups" method: GET path: "/dashboard/lineage" tags: ["Dashboard"] --- # Get dashboard lineage groups `GET /dashboard/lineage` Returns lineage groups (sources, destinations, or transforms) with aggregated metrics. Use the group_by parameter to select the grouping dimension. ## Query parameters - `group_by` 'source' | 'destination' | 'transform' | 'source_audit' — Group lineage by entity type - `search` string, nullable — Search filter for group names - `sort` 'name' | 'topic_count' | 'max_latency' | 'total_speed' | 'error_count' — Sort field values for the lineage groups endpoint. - `sort_dir` 'asc' | 'desc' — Sort direction - `page` integer — Page number (1-indexed) - `page_size` integer — Results per page - `include_metrics` boolean — When false, skip ClickHouse metrics and return metrics as null for faster initial render - `topic_name` string, nullable — Filter groups to those with matching topics (substring, case-insensitive) - `source_ids` string, nullable — Comma-separated source entity IDs to filter groups by - `destination_ids` string, nullable — Comma-separated destination entity IDs to filter groups by - `transform_ids` string, nullable — Comma-separated transform entity IDs to filter groups by - `include_topics` boolean — When true, embed first page of enriched topics in each group - `topics_page_size` integer — Max topics to embed per group - `topics_sort` 'byteTotal' | 'latency' | 'speed' | 'lag' | 'name' — Sort field for embedded topics - `topics_sort_dir` 'asc' | 'desc' — Sort direction for embedded topics - `diff_bands` string, nullable — Comma-separated diff% bands for topic filtering (audit tab). Each band is 'min-max' (inclusive). Multi-select: topic passes if its rounded-to-2-decimals max diff% falls in ANY band. Example: '0-0.05,2-10'. - `latency` string, nullable — Filter groups by max_latency_ms using "op:value" syntax (e.g. "gt:300000"). Ops: gt|gte|lt|lte|eq|ne. - `lag` string, nullable — Filter groups by total_lag using "op:value" syntax (e.g. "gt:1000"). Ops: gt|gte|lt|lte|eq|ne. - `snapshot` string, nullable — Filter groups by snapshot state. Comma-separated list: running|pending|completed|failed|not_started. ## Response `200` Successful Response - DashboardLineageRes — Response model for GET /dashboard/lineage. - `page` integer — Current page number - `page_size` integer — Results per page - `total` integer, nullable — Total number of results - `has_next` boolean, nullable — Whether more pages exist - `result` LineageGroupRes[] — List of lineage groups - `id` string, required — Group ID - `entity_id` string, required — Entity ID - `name` string, required — Display name - `type` string, required — Entity type: 'source', 'destination', or 'transform' - `connector` string — Connector type identifier - `status` string — Current status - `topic_count` integer — Number of associated topics. Reflects filtered count when topic_name and include_topics are both set. - `total_topic_count` integer — Total number of topics regardless of topic_name filter - `metrics` LineageGroupMetricsRes — Metrics for a lineage group (source, destination, or transform). - `max_latency_ms` integer — Maximum latency in milliseconds - `total_speed_rps` number — Total speed in records per second - `total_lag` integer — Total consumer lag across all topics (sum) - `max_dest_latency_ms` integer — Worst-partition destination wall-clock consumer latency (ms) across all reachable dest topics. - `topic_count` integer — Number of topics - `error_count` integer — Number of errors - `snapshot_progress` SourceSnapshotProgressRes — Snapshot progress for a source connector. Sourced from the ClickHouse snapshot_progress table via get_snapshot_summary(). - `state` string, required — Snapshot state: running, pending, completed, failed, or not_started - `snapshot_type` string — Snapshot type: incremental or blocking - `total_topic_count` integer — Logical topics (selectors) in the snapshot — what the user sees. Equals total_table_count unless a selector fanned out (sharded/regex). - `remaining_topic_count` integer — Logical topics still to be processed - `done_topic_count` integer — Logical topics in a terminal DONE/COMPLETED state - `cancelled_topic_count` integer — Logical topics cancelled by the user before completing - `failed_topic_count` integer — Logical topics that terminated in a FAILED state - `total_table_count` integer — Physical tables in the snapshot — expanded per shard/regex match. Exceeds total_topic_count only on fanout. - `remaining_table_count` integer — Physical tables still to be processed - `rows_total` integer — Total source rows across all tables (from QA metrics). 0 if unavailable. - `rows_scanned` integer — Total rows scanned across all tables. 0 if unavailable. - `progress_type` 'rows' | 'tables' — Whether percent_complete is based on row counts ('rows') or table counts ('tables'). - `percent_complete` number — Snapshot completion percentage (0-100) - `eta_ms` integer — Estimated milliseconds to completion - `start_timestamp` string, nullable — Snapshot start time (UTC) - `end_timestamp` string, nullable — Snapshot end time (UTC) - `last_audit_at` string, nullable — ISO timestamp of the last completed QA metrics audit for this group. Only present for source_audit groups. - `topics` EmbeddedTopicsRes — Paginated topics embedded within a lineage group. - `result` LineageTopicRes[] — Enriched topics for this group - `id` string, required — Topic entity ID - `name` string, required — Full topic name - `source` TopicSourceRef, required — Reference to the source that produces a topic. - `id` string, required — Source entity ID - `name` string, required — Source display name - `connector` string — Source connector type - `snapshot_progress` SourceSnapshotProgressRes — Snapshot progress for a source connector. Sourced from the ClickHouse snapshot_progress table via get_snapshot_summary(). - `state` string, required — Snapshot state: running, pending, completed, failed, or not_started - `snapshot_type` string — Snapshot type: incremental or blocking - `total_topic_count` integer — Logical topics (selectors) in the snapshot — what the user sees. Equals total_table_count unless a selector fanned out (sharded/regex). - `remaining_topic_count` integer — Logical topics still to be processed - `done_topic_count` integer — Logical topics in a terminal DONE/COMPLETED state - `cancelled_topic_count` integer — Logical topics cancelled by the user before completing - `failed_topic_count` integer — Logical topics that terminated in a FAILED state - `total_table_count` integer — Physical tables in the snapshot — expanded per shard/regex match. Exceeds total_topic_count only on fanout. - `remaining_table_count` integer — Physical tables still to be processed - `rows_total` integer — Total source rows across all tables (from QA metrics). 0 if unavailable. - `rows_scanned` integer — Total rows scanned across all tables. 0 if unavailable. - `progress_type` 'rows' | 'tables' — Whether percent_complete is based on row counts ('rows') or table counts ('tables'). - `percent_complete` number — Snapshot completion percentage (0-100) - `eta_ms` integer — Estimated milliseconds to completion - `start_timestamp` string, nullable — Snapshot start time (UTC) - `end_timestamp` string, nullable — Snapshot end time (UTC) - `metrics` TopicMetricsDetailRes — Detailed metrics for a single topic in lineage view. - `StreamingMilliSecondsBehindSource` integer — Streaming latency in milliseconds - `SnapshotPercentageComplete` integer — Snapshot completion percentage (uppercase) - `recordSendTotal` integer — Total records sent - `sourceRecordWriteTotal` integer — Total source records written - `snapshotPercentageComplete` integer — Snapshot completion percentage (lowercase) - `recordsLag` integer — Consumer records lag - `byteTotal` integer — Total bytes processed - `sourceRecordPollRate` number — Source record poll rate (records/sec) - `sourceQaCount` number, nullable — Source DB row count from metrics_qa. Null when not collected. - `snapshot_status` string, nullable — Latest snapshot status for this topic: running, pending, completed, failed, or null if never snapshotted. - `snapshot_details` TopicSnapshotStatusRes[] — Detailed snapshot history for this topic. Empty list if never snapshotted. - `status` string, required — Snapshot status: running, pending, completed, failed - `start_timestamp` string, nullable — Snapshot start time (UTC) - `end_timestamp` string, nullable — Snapshot end time (UTC) - `duration_ms` integer, nullable — Snapshot duration in milliseconds - `rows_scanned` integer — Number of rows scanned so far - `rows_total` integer — Total number of rows to scan - `trace` string, nullable — Error trace message for failed snapshots - `eta_ms` integer — Estimated milliseconds to completion - `transforms` TopicTransformRef[] — Transforms applied to this topic - `id` string, required — Transform entity ID - `name` string, required — Transform display name - `status` string — Transform status (e.g. RUNNING) - `metrics` TransformMetricsRes — Metrics for a transform in the topic lineage view. - `latency` integer — Transform processing latency in milliseconds - `output_topics` string[] — Output topics from this transform - `transform_type` 'fan_out' | 'sql_join' | 'enrich' | 'enrich_async' | 'map_filter' | 'toast_handling' | 'un_nesting' | 'rollup' | 'topic_router' - `destinations` TopicDestinationRef[] — Destinations consuming this transform's output topics, with lag/count keyed to the output topic. - `id` string, required — Destination entity ID - `name` string, required — Destination display name - `connector` string — Destination connector type - `pipeline_id` string — Pipeline entity ID - `pipeline_status` string — Pipeline status - `lag` integer, nullable — Destination consumer lag. Null when include_metrics=false. - `latency_ms` integer, nullable — Destination wall-clock consumer latency in milliseconds (worst partition). Null when include_metrics=false or no probe sample exists yet. - `qaCount` number, nullable — Destination row count for this topic from metrics_qa. Null when not collected. - `e2e_eta_ms` integer, nullable — Estimated milliseconds for destination to fully consume remaining data. Shown during active snapshots or when lag exceeds threshold. -1 means records are remaining but destination is not consuming (stalled). Null when not applicable. - `destinations` TopicDestinationRef[] — Destinations consuming this topic - `id` string, required — Destination entity ID - `name` string, required — Destination display name - `connector` string — Destination connector type - `pipeline_id` string — Pipeline entity ID - `pipeline_status` string — Pipeline status - `lag` integer, nullable — Destination consumer lag. Null when include_metrics=false. - `latency_ms` integer, nullable — Destination wall-clock consumer latency in milliseconds (worst partition). Null when include_metrics=false or no probe sample exists yet. - `qaCount` number, nullable — Destination row count for this topic from metrics_qa. Null when not collected. - `e2e_eta_ms` integer, nullable — Estimated milliseconds for destination to fully consume remaining data. Shown during active snapshots or when lag exceeds threshold. -1 means records are remaining but destination is not consuming (stalled). Null when not applicable. - `total` integer — Total matching topics in this group - `page` integer — Page number (always 1 for embedded) - `has_next` boolean — Whether more topics exist beyond this page ## Other responses - `422` — Validation Error --- [API](https://skmtc.net/streamkap/apis/streamkap-rest-api.md) · [All operations](https://skmtc.net/streamkap/apis/streamkap-rest-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/streamkap/streamkap-rest-api/revisions/e9ce8221bac6/schema)