Skip to content

Migration notes — 0.4.1 through 0.4.3

Operator-impacting changes after Migration to 0.4.0. For the 0.4.0 DataProduct cutover, start there first.

Summary table

Change Version Operator action
Edge existence + orchestrator Step 3b baseline 0.4.1 Deploy; verify -edge-existence workflow + dual triggers; historic runs trigger baseline (async)
Orchestrator-only sync triggers 0.4.1 Do not expect deploy to create DMS sync triggers — run historic orchestration
Sync cursor reset on historic handoff 0.4.1 Re-run historic if you saw full backlog replay through sync path
rule_set_version = RuleSet ref version 0.4.1 Update dashboards/filters if they assumed DataProduct semver on records
Stream template names in settings 0.4.1 Use BasicArchive / BasicLiveData in records.stream_type
Historic queue live DataProduct payload 0.4.1 Re-enqueue historic if counts were wrong; live API rebuild ignores ViewConfigState overrides
OrchestrationState.edge_baseline_call_ids 0.4.1 Run deploy_validation_infrastructure() after upgrade (ensures containers)
Aligned historic job_run_id (job_{view}_{epoch}) 0.4.1 Verify sync and historic share same jobRunId on records
schema_space brownfield guard 0.4.2 Dedicated sp_dp_* schema space per DataProduct; allow_existing_schema_space override
data_product_sync skip when DP tag matches 0.4.1+ Edge/sync workflow redeploy needs version bump or data_product_sync with force: true
Historic queue enqueue API 0.4.4 Use enqueue_historic_validation() when sync skips deploy; see Migration 0.4.4

Edge existence (0.4.1)

  • Estate-wide baseline: orchestrator Step 3b fires async edge_existence_baseline jobs (one per rule) with shared historic job_run_id — not monitored or auto-retriggered by the orchestrator.
  • Deploy creates {dp_wf}-edge-existence + dual modeling triggers per dqs:edgeExistenceConstraint when data_product_sync deploys workflows.
  • If the DataProduct version is unchanged and workflows already carry [dp:{version}], sync skips redeploy — bump the version or run sync with force: true to refresh edge workflows/triggers.
  • User guide: Edge existence.

Sync-cursor orchestration (0.4.1)

  • Deploy no longer creates instance sync triggers — orchestrator Step 3 only (after timestamp collection, before partitions).
  • Sync trigger and handler use no_backfill; initial_sync_cutoff applies on first handler run when no cursor exists.
  • Cursor resets on historic handoff to avoid backlog replay.
  • Guide: Sync-cursor and historic orchestration.

Records (0.4.1)

  • records.stream_type stores CDF template names (BasicArchive, BasicLiveData), not mutable/immutable.
  • rule_set_version on validation records uses the RuleSet reference version from ruleset_references, not DataProduct semver when they differ.
  • deploy_validation_infrastructure() ensures the records container only — stream creation is separate (ensure_records_infrastructure() / local run_validation() with post_to_records=True).
  • Guide: Records streams, Records output.

Historic validation (0.4.1)

  • HistoricJobQueue rebuilds orchestrator payload from live DataProduct API at job start (not ViewConfigState overrides when rebuild succeeds).
  • Falls back to stored enqueue snapshot if live rebuild fails.
  • Canonical job_run_id from deploy_validation_pipeline() matches queue-enqueued historic runs.

Schema space (0.4.2)

  • DataProduct view YAML schema_space must be a dedicated catalog space (sp_dp_*); operational instance space stays on view.space.
  • Deploy fails if schema_space already exists and is not owned by the DataProduct unless allow_existing_schema_space: true.
  • See Config → DataProduct-sourced view config.

Dimensions (0.4.3)

  • Library indexes dq:dimension only under http://purl.org/cognite/cdf_dq/DataQuality/.
  • Legacy http://example.org/dataquality/ logs a deprecation warning; dimensions do not populate.
  • Guide: Data quality dimensions in SHACL rules.

Upgrade checklist

  1. Publish latest cognite-data-quality to PyPI and run deploy_validation_infrastructure() (redeploys Cognite Functions and ensures containers).
  2. Redeploy validation infrastructure if edge rules, SHACL, or DataProduct version changed — or invoke data_product_sync with force: true when the DP version is unchanged but workflows must refresh.
  3. For each view on sync-cursor: confirm historic orchestration completed and sync trigger is active (check orchestrator warnings).
  4. Spot-check records: ruleSetVersion, jobRunId, dimensions (if applicable).
  5. For edge baseline after timeout: re-invoke edge_existence_baseline manually with the same job_run_id (no orchestrator auto-resume).