Skip to content

Migration to 0.4.0 (DataProduct + RuleSet only)

cognite-data-quality 0.4.0 removes config_source: yaml. Production deployment uses DataProduct + RuleSet APIs only.

What changed

Before (yaml mode) After (0.4.0)
SHACL uploaded to CDF Files SHACL published to RuleSet API; runtime loads only from ruleset_references or inline shacl_rules
Per-view workflows deployed inline Consolidated workflows per DataProduct (one instance workflow, optional uniqueness and edge-existence workflows)
config_source: yaml (default) config_source: dataproduct (only value)

Runtime handlers load SHACL from RuleSet API (ruleset_references) or inline shacl_rules only — not from CDF Files (shacl_rules_file_external_id was removed in 0.4.0).

Configuration models

Publish path (you own rules)

Keep local views/*.yaml and shacl_rules/*.ttl. Deploy publishes to RuleSet + DataProduct APIs:

# settings.yaml — config_source defaults to dataproduct; omit or set explicitly
workflow:
  external_id_prefix: "dq-shacl"

dataproducts:  # on each view YAML
  - external_id: "my-data-product"
    version: "1.0.0"

Consume path (externally-owned DataProducts)

external_dataproducts:
  - external_id: "equipment-product"
    version: "latest"

No local view YAML required. Optional local YAML merges operational overrides into DMS ViewConfigState.

Operator cutover checklist

  1. Publish SHACL to RuleSet (uniqueness, edge_existence, etc. as needed).
  2. Delete legacy per-view workflows (dq-shacl-{view}, dq-shacl-{view}-uniqueness) and their triggers.
  3. Deploy current release — deploy_validation_infrastructure() writes managed_data_product_external_ids to DMS and runs data_product_sync (default on).
  4. Historic backlog: on DataProduct/ruleset version change, data_product_sync auto-enqueues per-view historic jobs. For cutover on an unchanged version, run deploy_validation_pipeline() per view if static instances must be validated immediately.
  5. Edge_existence: after steps 1–3, verify data_product_sync upserted {dp_wf}-edge-existence + dual triggers per dqs:edgeExistenceConstraint. On the same version-change historic path, orchestrator Step 3b runs edge_existence_baseline (estate-wide) with a shared job_run_id. For an unchanged version that still needs a full edge re-scan, invoke baseline manually (force_rerun). See Edge existence.

Trigger budget

CDF limit: 200 triggers/project. Estimate per DataProduct before enabling edge constraints at scale:

triggers ≈ views × 2 (sync + uniqueness cron)
         + 2 × edge_rules (node + edge per dqs:edgeExistenceConstraint)

Still file-based in 0.4.0

  • RAW tables (tables/*.yaml) — SHACL uploaded to CDF Files; no DataProduct model. RAW-only deploys (no views/timeseries/external DPs) are supported.
  • Rule engine result sync listeners (rule_engine_result_sync/*.yaml)
  • settings.yaml — infrastructure, DMS upsert, external_dataproducts
  • Orchestrator invoke shorthand — optional view/settings YAML in CDF Files for deploy_validation_pipeline(view_config_external_id=...); SHACL still comes from ruleset_references in the resolved payload

Documentation

Operator docs updated for 0.4.0: instance/time series paths no longer document CDF Files SHACL or config_source: yaml. See Deploy Infrastructure and Lifecycle and Rule sources.

For changes in 0.4.1–0.4.3 (sync-cursor orchestration, edge baseline, records streams, dimensions), see Migration 0.4.1+.

Ad-hoc validation

For one-off runs without deploying workflows, use Run validation or Invoke. These are not substitutes for production deploy_validation_infrastructure().

Errors

If settings.yaml still has config_source: yaml, deploy fails with a message pointing to this guide.