Sign in
latestOpenAPI 3.0.32026-08-10119559584.8 KB

6d091bc86a33

Data Egress

Export joined data across multiple DQ topics

Streams the full result set of rows joined across an arbitrary connected subset of the DQ topics (jobs, monitors, rules) in a single pass for bulk extraction into BI tools, data pipelines, and integrations.

The request body is optional: POST with no body (or a body that omits topics) to export all three topics — jobs, monitors, rules — with every other default applied.

Response format (content negotiation). The format is selected by the Accept request header:

  • Accept: application/x-ndjson → newline-delimited JSON (JSON Lines): one JSON object per line. Preferred for programmatic consumers — types are preserved (numbers stay numbers, null stays null) and rows can be parsed incrementally.
  • Accept: text/csv → CSV download. The header row is the qualified field names (e.g. rules.ruleName). List-typed fields (jobs.daysOfWeek, monitors.dimensionNames, rules.dimensionNames) render as pipe-delimited cells (Completeness|Accuracy) so consumers don't have to handle quoted-comma CSV.

When Accept is absent or */*, the server returns application/x-ndjson; any other media type is rejected with HTTP 406. Both formats stream the same rows in the same order; only the encoding differs.

Topics. JOBS, MONITORS, and RULES are mutually joinable — any non-empty combination of them is valid. Quality dimensions are not a topic: the names of the quality dimensions assigned to a monitor (and to a rule — a rule is a RULE-type monitor) are surfaced inline on the row as the monitors.dimensionNames / rules.dimensionNames list field (active assignments only; empty when none are assigned).

To discover what topics can be joined call GET /egress/manifest first. Each topic's joinsTo lists the other topics it can be joined to; your selected topics are valid if they form a connected subgraph over those edges. The manifest also lists each field's filterable flag and allowedOperators, so a client can fully validate a request locally before POSTing.

Filters. Optional predicates in the body's filters array, ANDed together (omit for no filtering beyond RBAC and the default window). Each EgressFilter is a field, an op, and a value whose shape depends on the operator:

  • value (a single scalar) for EQ, NE, LIKE, GT, GTE, LT, LTE
  • values (an array) for IN, NOT_IN
  • neither for IS_NULL, IS_NOT_NULL

field is the qualified name <topic>.<apiName> (e.g. rules.ruleName) and must belong to a selected topic; each field permits only a subset of operators. Call GET /egress/manifest to discover the filterable fields and their allowedOperators. A request fails with HTTP 400 if a filter names an unknown field, uses an operator the field doesn't allow, or references a field whose topic isn't in topics. Filtering on any *.updatedAt field also overrides the default time window (see below).

Ordering. Custom sort keys are not currently a request parameter. Rows are always returned ordered by <granularity>.updatedAt descending — the same field the default time window keys on (rules > monitors > jobs) — followed by a unique tiebreaker for deterministic streaming.

Projection. is not currently a request parameter. Each row is a key/value map containing all projectable fields of the selected topics, in manifest order (fields marked optIn: true are excluded unless explicitly enabled — see catalog assets below). Use GET /egress/manifest to discover the supported topics, fields, operators, and opt-in flags.

Catalog assets (opt-in). Set includeCatalogAssets (query parameter or request-body property — enrichment applies when either is true; default false) to add, to each row, the DGC Catalog assets it maps to. Three field groups are populated, all omitted when the flag is off:

  • Table — jobs.tableCatalogAssetId / ...DisplayName / ...Type: the Catalog Table asset for the row's table. One per row.
  • Column — monitors.columnCatalogAsset* and rules.columnCatalogAsset*: the Column asset for the row's column (monitors.columnName / rules.ruleColumnName). One per row, or null when the row has no column scope (e.g. a dataset-level monitor).
  • Business Rule — monitors.businessRuleLinks: the Business Rule assets linked to the monitor, matched by identity (a rule surfaces here as its RULE-type monitor). Collapsed into one list of "<assetId>::<signifier>" entries, so a monitor linked to several rules stays one row (no row multiplication).

Two query parameters trim the enrichment (both default false): excludeColumnAssets omits the Column group; excludeRuleAssets omits the Business Rule group. Both are no-ops unless includeCatalogAssets is in effect.

Default time window. When the request omits any *.updatedAt filter, the server injects a rolling window on the updatedAt of a selected topic, anchored in the order monitors > rules > jobs. Its length is the window query parameter in days, defaulting to 90. To set an exact range instead, filter explicitly on any *.updatedAt field — the server suppresses the injected window (and ignores window) once any such filter is present. The default-window contract is also surfaced in defaultWindow on the manifest response.

Permissions

  • Required: DATA_QUALITY and DATA_QUALITY_JOB_VIEW. Results are scoped to the jobs the caller can see.
  • Global bypass:
    • DATA_QUALITY and VIEW_PERMISSIONS_VIEW_ALL
    • DATA_QUALITY and RESOURCE_MANAGE_ALL
post/egress/export

Query parameters

windowinteger

Rolling look-back window, in days, for the auto-injected time filter. The server bounds a selected topic's updatedAt to >= now - window days, anchoring on the topic in the order monitors > rules > jobs (monitors.updatedAt whenever monitors are selected, else rules.updatedAt, else jobs.updatedAt). Defaults to 90, capped at 365. Ignored when the request body already carries an explicit *.updatedAt filter — that filter wins.

includeCatalogAssetsboolean

Opt in to catalog-asset enrichment — the query-parameter equivalent of the request body's includeCatalogAssets, usable on a body-less request. Enrichment is applied when this or the body flag is true. Default false. See Catalog assets in the operation description for the fields populated, the row-multiplication caveat, and the cost.

excludeRuleAssetsboolean

When catalog-asset enrichment is in effect (includeCatalogAssets true), skip the monitors Business Rule asset enrichment (monitors.businessRuleLinks). Default false (business-rule links included). Use this to avoid the linked-rule asset lookup when only table and/or column assets are needed. No effect when includeCatalogAssets is not in effect.

excludeColumnAssetsboolean

When catalog-asset enrichment is in effect (includeCatalogAssets true), skip the DGC Column asset enrichment of the monitors and rules topics (monitors.columnCatalogAsset*, rules.columnCatalogAsset*). Default false (column assets included). No effect when includeCatalogAssets is not in effect.

completionMarkerboolean

Append a trailing completion marker so a consumer can distinguish a complete extract from one truncated by a mid-stream failure (or a proxy stream timeout). NDJSON ends with a final {"_complete":true,"rowCount":N} line; CSV ends with a # complete rowCount=N comment line. Default true. Set false for a byte-clean body (e.g. naive CSV readers that don't skip #-prefixed comment lines), at the cost of losing truncation detection.

Request body

topicsEgressTopic[]

Topics to include in the export. Omitted or empty → all three (JOBS, MONITORS, RULES). Today's surface exposes those three; the upper bound has headroom so adding a new topic doesn't break clients.

includeCatalogAssetsboolean

Opt in to catalog-asset enrichment. May also be set via the includeCatalogAssets query parameter (useful on a body-less request); enrichment applies when either is true. When true, each row is enriched with the DGC Catalog assets it maps to: the Table asset on jobs, the Column asset on monitors/rules, and the governing Data Quality Rule asset on monitors. See Catalog assets in the POST /egress/export description for the exact fields and the excludeColumnAssets / excludeRuleAssets switches.

When false (default) no enrichment is performed and those fields are omitted — off by default because enrichment makes external round-trips to DGC. Note a monitor linked to several Data Quality Rule assets emits one row per linked asset; excludeRuleAssets opts out of that group (and its row multiplication).

Response

The egress export, streamed in the format selected by the Accept header (application/x-ndjson by default, or text/csv). The response carries Content-Disposition: attachment so browsers download to disk; the filename extension reflects the negotiated format (.jsonl / .csv).

Streaming semantics. Both response schemas are declared with format: binary even though the bodies are text — this hints to SDK codegen to produce a streaming Resource / InputStream return type instead of buffering the entire payload in memory as a String. Without that hint, regenerated SDKs default to string-buffered responses, which defeats the streaming-export design (and would OOM on large result sets). Treat the body as text at the wire level; treat the SDK return type as a stream.

Stream completeness. Because the body streams, a failure after the first byte cannot change the HTTP status — it surfaces only as a short stream (plus a server-side error log) and can look like a successful, fully-read extract. To let consumers detect this, the stream always ends with an explicit completion marker once every row has been written:

  • NDJSON: a final line {"_complete":true,"rowCount":N} (data rows never carry a _complete key).
  • CSV: a trailing comment line # complete rowCount=N.

A consumer that reaches end-of-stream without seeing the marker must treat the extract as truncated/failed, not complete. rowCount is the number of data rows emitted (excludes the header and marker).