Skip to content

Changelog

Each heading is a published package version. The text under it is that version's release notes.

0.5.0b5 — 2026-10-16 · Pre-release

Fixed

  • A SPARQL type check on an incoming relation fails when nothing points at the focus node, so a node with no reverse neighbour is non-conforming.
  • When several nodes point at the focus node on the same relation, a wrong type on any of them fails the check and that neighbour stays bound.
  • Every reverse type check in the query is rewritten, including a filter that ends with a period and a second check in the same query.
  • Auto-loaded reverse edges quote external ids and keep each node's view across load depths, so the neighbour joins the focus node under the view from the pass that loaded it.

0.5.0b4 — 2026-10-15 · Pre-release

Fixed

  • Publishing a ruleset no longer starts a second historic job while one run for that view is still active. A live monitor, or a start still inside the five-minute startup grace, keeps the queue slot. After the grace, a run that never wrote orchestration state or a monitor is marked failed.
  • A normal historic finish still completes from retained orchestration state after the monitor schedule is removed. A failed state or schedule read waits instead of failing or freeing the job. Start times returned as datetimes or epoch milliseconds count the same as ISO strings.
  • Restarting a stale timestamp collection keeps the same orchestration id, drops a checkpoint resume id, and calls the orchestrator on a fresh session.
  • A partial Records post no longer advances the historic partition cursor or the instance sync cursor past instances that were not stored. The next run retries that page. Rule-engine inference records are not written when the validation page will be retried.

0.5.0b3 — 2026-10-12 · Pre-release

Added

  • Packaged Atlas AI rule-authoring agent and SHACL journey skills (Consistency with sh:node, edge existence, uniqueness, time series quality, conditional and chained conditional). Deploy with deploy_atlas_rule_authoring(), or opt in via atlas_rule_authoring settings / deploy_atlas_rule_authoring=True on infrastructure deploy. Skills ship in the wheel under cognite_data_quality/agents/.
  • Usage guides for Atlas rule authoring and for editing Data Products and RuleSets in Flows apps (draft/test vs Publish, multi-view quality.rules).

0.5.0b2 — 2026-10-08 · Pre-release

Added

  • Verbose validation warns when a SPARQL type or existence check follows a direct relation that has no sh:node auto-load. Without that declaration the related instance is not in the graph, so a valid link can fail the check. The warning covers a, rdf:type, full type IRIs, and hyphenated or URI property paths. Runs without verbose stay quiet.

Fixed

  • Upgrading the validation records container no longer flips required identity properties to optional. ruleSetId, ruleSetVersion, and jobRunId on DataQualityValidationRecord stay non-nullable, as do ruleSetId, ruleSetVersion, ruleId, and runId on RuleEngineResult. The previous full-property update sent the default optional flag and failed on projects where those fields were already required.
  • Historic job status listing returns every matching HistoricJobQueue node. The previous 1000-node cap hid later queued, running, and superseded jobs, so a version-change enqueue could miss the previous job in the status list and the queue manager could skip jobs past that page.

Changed

  • Usage guides state that forward relation targets load only when sh:node declares them, and reverse neighbours only when neat:reverseDirectRelation declares them. Raising auto_load_depth alone does not load undeclared neighbours.

0.5.0b1 — 2026-10-05 · Pre-release

Added

  • SHACL constraint display names on validation records: shaclName on each violation and filterable failedConstraintNames (resolved from the shapes graph; uniqueness and edge existence use typed tokens)
  • Record views over DataQualityValidationRecord and RuleEngineResult (version v1, with streamId) so Flows can bind view plus stream
  • Filterable ruleId on uniqueness / aggregate validation records (from dqs:ruleId, a named uniqueness node, or unique_{property})
  • Instance SHACL validation (sync-cursor and historic) supports Data Product views with used_for: edge, validating properties on edge instances over the edges channel; violation records omit focusNodeInstance for edge-sourced findings
  • Deploy results for external_dataproducts include per-view view_deployments (created, updated, or skipped) with view space, so first-time historic validation can run when a view is added to an existing DataProduct workflow.

Fixed

  • Applying the validation records container no longer drops existing properties when adding newly merged ones; the update now always sends the full desired property set.
  • external_dataproducts deploy resolves operational view space (same callback as data_product_sync), so RuleSets targeting sp_ent_ops match and ViewConfigState aliases are not overwritten with the schema space.
  • Same-path / multi SPARQL constraints no longer last-win a single shaclName via path_to_name; violations are matched to distinct sh:name values using indexed sh:message (suffix-stripped). Unmatchable multi-SPARQL omits shaclName instead of falling back to the NodeShape name. Still one Records row per instance.

Changed

  • The supported consumption path for validation output is record view plus stream. Reverse navigation from customer instance views is opt-in through focusNodeInstance. Grouped uniqueness and RAW rows without that relation stay stream-filter only. Grouped uniqueness externalId stays per property (dq_{ruleSetId}_group_{property}_...), so two uniqueness rules on the same property still share one grouped row on a mutable stream even when ruleId differs.
  • Deploy must re-apply the DataQualityValidationRecord container and record views so failedConstraintNames and ruleId exist before ingest. Redeploy the CDF validation function so writers emit the new fields.
  • Records docs: for same-path SPARQL, aggregate/display by failedConstraintNames / per-violation shaclName, not only failedConstraints machine tokens.

0.4.11 — 2026-09-14

Added

  • Historic enqueue accepts force=True to start a new scan for the current DataProduct version, including views that opted out of reprocess-on-update.

Fixed

  • Historic enqueue force=True still skips queued and running jobs so a view never has two active historic orchestrations.
  • Orchestrator force is rejected. Data-product sync force still only redeploys workflows and does not pass enqueue force.

Changed

  • Release pull requests now include the Read the Docs changelog. Publishing no longer pushes that page after the package is uploaded.

0.4.10 — 2026-09-14

Added

  • Pre-persist Save gate (validation_type: pre_persist): overlay proposed instance JSON, hydrate from DMS, and return form-mappable violations and inferences before apply. Catalog identity is Data Product, RuleSet, and view.
  • Stream retention guard: scheduled refresh of instance historic, timeseries, and edge baseline runs before Records stream TTL expires.

Fixed

  • Timeseries SHACL time-window substitution now resolves each -ago placeholder independently, so rules like "24h-ago" are not evaluated over a 1-hour window.
  • Timeseries prefetch passes millisecond timestamps as integers and aligns count granularity for windows longer than 24 hours.
  • Stream retention guard persists timeseries config_name, skips in-progress retention calls, matches instance jobRunId as job_{view}_{epoch}, and enqueues historic refresh with the DataProduct semver.
  • Orchestrator restores ruleset_references on retention refresh payloads.
  • CDF Function local-wheel deploy uses the required function/ prefix in requirements.txt.

Changed

  • Usage documentation is reorganized into focused guides, including Pre-persist Save and stream retention.

0.4.9 — 2026-08-27

Fixed

  • Historic lastUpdatedTime partitions no longer drop instances at index boundaries when the next partition starts at a later timestamp.
  • Scheduled uniqueness mints a per-run job_run_id, and prefers a resolved jobRunId when the snake_case placeholder is unresolved.
  • Orchestrator no longer uses an unsupported HasData aggregate as an instance-count completeness guard.

0.4.8 — 2026-08-21

Added

  • Persist timeseries_workflow.external_id_prefix on DataQualitySettings so runtime and the dashboard can discover custom timeseries workflow IDs.

Fixed

  • data_product_sync no longer looks up timeseries workflows with the instance prefix when a separate timeseries_workflow prefix is configured.

0.4.7 — 2026-08-20

Fixed

  • Scheduled uniqueness cron could complete as Succeeded without posting Records because the cursor-resolved job_run_id was not applied to uniqueness extensions. Redeploy the CDF validation function after this release.

Changed

  • Uniqueness records now set dimensions to Uniqueness (or the shape dq:dimension) instead of nested Consistency. Update dashboard filters that keyed uniqueness findings on Consistency.
  • Uniqueness status=error now fails the CDF function call instead of returning top-level success.

Improved

  • Uniqueness function logs always print errors and an [aggregate_uniqueness] outcome line (returnedBuckets, violatingBuckets, recordsPosted, recordsSuppressed, truncated).

0.4.6 — 2026-08-17

Fixed

  • Historic enqueue and queue manager resolve DataProduct catalog schema spaces to operational DMS spaces via ViewConfigState, so RuleSet matching works for brownfield DPs (sp_ent_dp_* catalog vs sp_ent_ops SHACL targets).

0.4.5 — 2026-08-13

Fixed

  • Skip managed DataProducts without SHACL-ruled views during bulk historic enqueue (warning instead of failing deploy)
  • Match SPARQL sh:target views when loading ruled views from the DataProduct API (enqueue and sync reload)
  • Explicit single-DataProduct enqueue still raises when that product has no SHACL rules

0.4.4 — 2026-08-13

Fixed

  • Upgrade twine to >=7.0.0 so CD can publish packages built with hatchling Core Metadata 2.5 (retry 0.4.4 PyPI upload)

Added

  • enqueue_historic_validation() public API and historic_mode="enqueue" on deploy_validation_pipeline for sequential historic runs from deploy scripts and CI

0.4.3 — 2026-08-10

Fixed

  • dq:dimension lookup now uses the canonical Cognite vocabulary namespace (http://purl.org/cognite/cdf_dq/DataQuality/) so dimensions populate in validation records when rulesets use the correct prefix
  • Consolidated dimension lookup into shared _dimension_lookup.py across instance, sync-cursor, partitioned, and instance-validation handlers
  • SPARQL dq:dimension annotations on parent NodeShapes via BNode parent walk-up

Added

  • docs/usage/dimensions.md guide for annotating SHACL rules with dq:dimension

0.4.2 — 2026-08-06

Fixed

  • Brownfield DataProduct publish with dedicated schema_space: DataProduct version views[] entries now reference views in schema_space (CDF requires views[].space to match schemaSpace or cdf_*, not operational view.space from YAML)
  • data_product_sync restores operational view.space from ViewConfigState when the DataProduct API returns the schema space
  • View definition is copied into schema_space only when missing there (no-op when the view already exists)

Changed

  • Brownfield deploy docs: clarify that view.space stays operational for validation while schema_space is used in the DataProduct API reference

0.4.1 — 2026-08-05

Added

  • DataProduct-scoped consolidated validation workflows, per-view sync lease, and RuleSet-only runtime path (0.4.0 migration)
  • dqs:edgeExistenceConstraint production orchestration: consolidated {dp}-edge-existence workflow with dual modeling triggers (edge_existence_event)
  • Estate-wide edge existence baseline (edge_existence_baseline) via orchestrator Step 3b on DataProduct/RuleSet version change
  • Mutable and immutable Records API streams with auto-detected ingest vs upsert; stable space-qualified record externalIds
  • Brownfield DataProduct publish with dedicated schema_space separate from operational view.space

Changed

  • records.stream_type stores CDF template names (BasicArchive, BasicLiveData, …); legacy mutable/immutable aliases still map at stream creation
  • Instance sync DMS triggers are orchestrator-only (correct job_run_id, cutoff, and lock initialization)
  • Historic job_run_id aligned to job_{view}_{epoch} across queue manager, orchestrator, and deploy helpers

Fixed

  • Sync cursor reset on historic handoff; forward-only no_backfill when no cursor/cutoff
  • data_product_sync no longer redeploys/re-enqueues hourly on an unchanged DataProduct version
  • Orchestrator empty sync-page completion; null data_quality_space on sync lock paths
  • Container property merge preserves nullable flags when adding new DMS properties
  • Handler raises on status: error so CDF marks failed function calls as Failed
  • Handler startup banners for queue manager, sync, and other validation handlers

0.4.1b11 — 2026-08-05 · Pre-release

Fixed

  • Orchestrator timestamp collection infinite loop when DMS sync returns empty pages with an advancing cursor at end of backfill (#320)

Improved

  • Log validation_type and orchestrator-style startup banners for handlers missing clear identification in CDF logs (#319)

0.4.1b10 — 2026-08-05 · Pre-release

Fixed

  • Align historic job_run_id format (job_{view}_{epoch}) across historic queue manager, orchestrator, and deploy helpers (#317)
  • Require explicit job_run_id for sync-cursor and uniqueness validation; remove deploy-time fallback ids that split dashboard runs (#317)
  • Omit static job_run_id from sync/uniqueness trigger input; resolve from cursor handoff or workflow payload (#317)
  • Raise from the unified handler when sub-handlers return status: error, so CDF marks failed function calls as Failed (#317)
  • Resolve null data_quality_space in sync lock load/acquire/release paths (#311)

0.4.1b9 — 2026-08-04 · Pre-release

Fixed

  • _safe_apply_container preserves existing DMS container property definitions (including nullable) when adding new properties. Fixes deploy failure: Cannot change property orchestration_id from non-nullable to nullable in OrchestrationState when upgrading to versions that add edge_baseline_call_ids.
  • Set explicit nullable=False on OrchestrationState.orchestration_id and nullable=True on HistoricJobQueue.orchestration_id for new container creation.

0.4.1b8 — 2026-08-04 · Pre-release

Added

  • Brownfield DataProduct publish: set schema_space to a dedicated space while views remain in their operational DMS space (for example sp_ent_ops). Explicit schema_space is required when view.space is not cdf_*.

Changed

  • DataProduct publish no longer defaults schema_space to view.space for operational spaces, avoiding accidental dedicated-space guard failures on brownfield projects.

Improved

  • Deploy docs include brownfield dataproducts example with separate schema_space and view.space.

0.4.1b7 — 2026-07-10 · Pre-release

Changed

  • Settings records.stream_type now stores CDF stream template names (BasicArchive, BasicLiveData, etc.) instead of mutable/immutable; legacy values still map when creating streams (#300)
  • Records validation uses stream_template in settings instead of stream_type (#299)

Fixed

  • Use RuleSet reference version in records_config.rule_set_version when loading views from the DataProduct API (#296)
  • Release workflow excludes erroneous 0.0.1b* tags so git describe resolves 0.4.1b6 and publishes 0.4.1b7 instead of 0.0.1b2

0.3.8 — 2026-07-10

Fixed

  • Use RuleSet reference version in records_config.rule_set_version instead of DataProduct version when loading views from the DataProduct API (#295)
  • Align orchestrator and deploy pipeline with per-view workflow model (dq-shacl-{view} / dq-shacl-trigger-{view}) on release-0.3.x
  • Delete stale DMS sync triggers for a view before upserting the canonical trigger, preventing duplicate triggers on repeated pipeline runs

0.0.1b2 — 2026-07-10 · Pre-release

Changed

  • Settings records.stream_type now stores CDF stream template names (BasicArchive, BasicLiveData, etc.) instead of mutable/immutable; legacy values still map when creating streams (#300)

Fixed

  • Release workflow restores git describe --exclude '0.3.*' on main after git tag --merged version sort incorrectly picked v0.0.0 and published 0.0.1b1 instead of 0.4.1b7

0.0.1b1 — 2026-07-10 · Pre-release

Changed

  • Settings records.stream_type now stores CDF stream template names (BasicArchive, BasicLiveData, etc.) instead of mutable/immutable; legacy values still map when creating streams (#300)

Fixed

  • Release workflow uses git tag --merged HEAD for branch-scoped version lookup, adds manual retry dispatch, and fails when the computed version already has a release (#304)

0.4.1b6 — 2026-07-09 · Pre-release

Fixed

  • Reset sync cursor on historic handoff so the sync handler no longer replays the entire backlog (#294)
  • Sync-cursor handler opens instances.sync() in no_backfill mode when no saved cursor and no cutoff (forward-only)
  • Avoid KeyError on post-reset empty cursor state

Added

  • Estate-wide edge existence baseline handler (validation_type: edge_existence_baseline) and orchestrator Step 3b (#290)

0.4.1b5 — 2026-07-07 · Pre-release

Fixed

  • Deploy no longer creates DMS instance sync triggers; only the historic orchestrator creates them with the correct job_run_id, initial_sync_cutoff, and sync lock initialization
  • Removed bind_sync_trigger_job_run_id at orchestrator start; sync trigger input is now built entirely by the orchestrator after timestamp collection
  • Release pipeline on main excludes 0.3.* tags so version bumps resolve the correct 0.4.x line

Changed

  • Skips workflow-version patching on the consolidated parametric sync-cursor deploy path

0.3.8b1 — 2026-07-07 · Pre-release

Fixed

  • Deploy no longer creates DMS instance sync triggers; only the historic orchestrator creates them with the correct job_run_id, initial_sync_cutoff, and sync lock initialization
  • Removed bind_sync_trigger_job_run_id at orchestrator start; sync trigger input is now built entirely by the orchestrator after timestamp collection

Changed

  • Skips workflow-version patching on the consolidated parametric sync-cursor deploy path

0.3.7 — 2026-07-07

Fixed

  • Backport #276: stop hourly data_product_sync historic re-enqueue when a completed job already exists for the same view and DataProduct version; only check up-to-date status for views with SHACL rules
  • Backport #278: fix orchestrator sync-trigger crash on missing workflow — resolve sync workflow ID with CDF existence probe, fail fast when workflow version 1 is absent, and make Step 6 non-fatal so historic partitions survive sync setup failures
  • Backport #279: use timestamped job_{view}_{epoch} job_run_id, bind per-view DMS sync triggers at historic orchestrator start, and rebind sync cursor job_run_id with optimistic concurrency
  • Address Gemini review: invoke.py prefix guards, corrected 0.3.7 migration doc, remove dead _check_dataproduct_workflows_deployed, use limit=-1 for trigger listing

0.4.1b4 — 2026-07-06 · Pre-release

Added

  • Edge existence consolidated validation workflow with dual triggers to validate typed-edge cardinality

Fixed

  • Require a dedicated schema space for DataProducts to avoid space conflicts during deploy
  • Orchestrator crash when a sync trigger referenced a missing workflow
  • Historic validation now uses a timestamped job_run_id and binds sync triggers at historic start

0.4.1b3 — 2026-07-03 · Pre-release

Fixed

  • data_product_sync no longer redeploys workflows and re-enqueues historic jobs every hour on an unchanged DataProduct version
  • False redeploys when per-view tag check required workflows for views with no SHACL rules

Changed

  • Always use consolidated DataProduct workflows in dataproduct deploy paths; remove consolidate_per_dataproduct setting
  • Historic queue skips enqueue when a completed job already exists for the same view and DataProduct version

0.4.1b2 — 2026-07-02 · Pre-release

Fixed

  • Deployment fails when persisting managed_data_product_external_ids to DMS: wrap the ID list in a JSON object so scalar json properties accept the value
  • Harden managed DataProduct ID encode/decode so scalar strings are not split into characters

0.4.1b1 — 2026-07-02 · Pre-release

Added

  • DataProduct-scoped validation workflows: consolidated deploy pipeline, per-view sync execution lease, and cursor CAS (Phase 1–3)
  • Mutable and immutable Records API stream support with auto-detected ingest vs upsert
  • Stable validation record externalId now includes instance space
  • Minor + Beta release bump support for minor-line prereleases (0.3.6 → 0.4.1b1)
  • SHACL per-extension dispatch refactor and edge-existence orchestration groundwork
  • DataProduct + RuleSet-only runtime path and 0.4.0 migration guide

Changed

  • View-only DataProduct pipeline resolve uses space-aware matching for external DataProducts
  • External DataProduct local YAML overrides matched on both space and external_id
  • Operator docs aligned with 0.4.0 RuleSet-only runtime contract
  • Release bump skips burned PyPI slot 0.4.0 (accidental publish); beta line starts at 0.4.1b1, graduates to 0.4.1 stable

Fixed

  • Sync lease lock node initialization is create-if-absent; load errors no longer clobber lock state
  • Deploy skips DataProduct publish for external_dataproducts references
  • Version file guard prevents accidental PyPI publish when files are not at 0.0.0 placeholder
  • RAW-only deployments and related CI lint issues

0.3.6 — 2026-06-19

Improved

  • Align lifecycle heuristics across handlers and test_and_deploy scripts for stable semver ordering, payload-aware version reuse, and robust live queue assertions.
  • Update lifecycle skill guidance to codify handler/script parity requirements and queue-state validation behavior.

0.3.5 — 2026-06-17

Fixed

  • Retry release trigger with valid changelog format and 0.3.6 target.

Added

  • None.

0.3.4 — 2026-06-16

Fixed

  • RuleSet publish fails when shacl_rules.external_id includes a .ttl filename suffix — normalize to a valid RuleSet API ID before publishing (#235).
  • RuleSet/DataProduct create treats HTTP 400 duplicate responses as already-exists (API returns 400, not 409, when the external ID exists) (#235).

0.3.3 — 2026-06-16

Fixed

  • Prepare release branch for 0.3.3 and validate release checks via pre-commit.

Added

  • Add 0.3.3 release marker entry in release.md.

0.3.2 — 2026-06-16

Added

  • Incremental RuleEngineResult sync chaining for cross-run listener workflows (#228).
  • Data Product-driven time series datapoint quality rules with RuleSet API integration (#227).
  • Rule Engine Conditional Logic user guide (#226).

Fixed

  • Resolve timeseries RuleSet ID when instance and timeseries rules coexist in Data Products.
  • Multi-DataProduct state persistence for timeseries validation configs.
  • RuleEngineResult sync handler pagination, watermark, cursor lifecycle, and filter normalization edge cases.

Improved

  • Uniqueness documentation for grouped overflow heuristics, trigger heuristics, and 100-item limits (#222, #223).

0.3.1 — 2026-06-12

Fixed

  • Include final partition max boundary in partitioned validation so latest instances are validated.
  • Improve historic queue superseding cleanup to avoid stale orchestration state blocking newer runs.

Added

  • Add release 0.3.1 branch marker in release.md for CD release flow.

0.3.0 — 2026-06-12

Added

  • Explicit validation record typing in DataQualityValidationRecord using validationType (single_violation and group_violation) and grouped subtype handling via groupViolationType (global_overflow, value_overflow, pass).
  • Grouped PASS heartbeat behavior for all-green uniqueness runs.

Changed

  • Uniqueness overflow behavior to deterministic N+1 mode detection (aggregate and instances.list with 101 semantics), including global-overflow shortcut and per-value overflow grouping.
  • Documentation and records contract guidance to support deterministic consumer branching for grouped vs single violation flows.

Improved

  • Release packaging and communication alignment for the 0.3.0 contract change context from issue #217.

0.2.28 — 2026-06-10

Fixed

  • Add release marker for 0.2.29 in release.md to trigger the next release PR flow.

Added

  • None.

0.2.27 — 2026-06-09

Fixed

  • Use triggers.upsert() instead of non-existent triggers.create() when deploying uniqueness workflow triggers
  • Add cognite-sdk to mypy pre-commit hook dependencies so SDK attribute errors are caught at commit time
  • ContainerProperty → ContainerPropertyApply in write-path container definitions (cursor_state, timeseries_validation)
  • list[HasData] filter list widened to list[Filter] to allow Equals/Prefix appends
  • _fetch_filtered_rows return type corrected from Iterable[Row | RowList] to Iterable[Row]
  • DMS property dict values cast to str where used as dict keys (cursor_state, orchestration_status)

0.2.26 — 2026-06-09

Fixed

  • data_product_sync: pass function secrets to trigger credentials so uniqueness workflow triggers can be created when running inside a CDF Function (runtime client uses OIDC token, not ClientCredentials)

0.2.25 — 2026-06-09

Added

  • SHACL-native aggregate uniqueness validation: declare dqs:uniquenessConstraint or dqs:unique in TTL/RuleSet, run via dedicated scheduled dq-{view}-uniqueness workflow with aggregate-first duplicate detection
  • validation_type: shacl entrypoint discovers uniqueness constraints at runtime and compiles to the aggregate executor
  • HistoricJobQueue for sequential historic validation in DataProduct mode
  • AGENTS.md with Cursor Cloud development instructions

Fixed

  • Sync-cursor and instance validation no longer auto-run uniqueness after each sync completion; uniqueness is scheduled-only unless explicit extensions are passed
  • Uniqueness workflow deploy gated on SHACL constraint presence; RuleSet content fetched at deploy for DataProduct mode
  • data_product_sync diagnostics and defensive null guards; malformed ruleset references logged instead of silently skipped
  • Separated runtime credentials with fallback for workflow trigger deployment
  • Skip forward auto-loading when no sh:node constraints are declared
  • SDK 8.4+ alignment for instances.aggregate / instances.list; trigger API uses triggers.create in SDK v8

Improved

  • Instance-space scoping propagated into uniqueness aggregate and detail filters
  • Duplicate-value processing capped at 100 per run with explicit warnings; peer payload and URN encoding hardening

0.2.24 — 2026-06-04

Fixed

  • Add config_space to DeploymentSettings with effective_config_space property (falls back to records.space); use throughout deploy, containers, and invoke so state/settings containers can live in a separate DMS space from validation records
  • deploy_validation_pipeline Mode 4 and data_product_sync workflow now pass the correct config space instead of hardcoding records.space
  • load_views_from_data_product now fetches each RuleSet's content, extracts sh:targetClass URIs, and only assigns SHACL rules to views whose DMS class URI appears in the rules

0.2.23 — 2026-06-02

Fixed

  • Skip forward auto-loading when no sh:node constraints are declared (avoids redundant DMS fetches for views with only non-node constraints)
  • Only persist and apply non-empty instance_spaces in ViewConfigState: view YAMLs without an explicit instance_spaces list were saving {"spaces": []} to DMS, causing the DataProduct API's correct spaces to be overridden with an empty list and zero instances to be validated

0.2.22 — 2026-05-29

Fixed

  • Set allow_expired_cursors_and_accept_missed_deletes=True on sync queries — prevents monitoring blackouts when a cursor is older than 3 days (previously caused a 400 error that silently stopped validation)

0.2.21 — 2026-05-29

Fixed

  • Replace raw client.post list scans in cursor_state.py with instances.retrieve for direct state lookups — eliminates silent misses when stale nodes accumulate past the filter window

0.2.20 — 2026-05-29

Fixed

  • Historic orchestrator monitor schedule deleted prematurely when it fired in the same cron window as orchestrator startup, causing partitions to run only once without retry. Fixed by saving an initial orchestration state before creating the monitor schedule.

0.2.19 — 2026-05-28

Added

  • Function display name in CDF Console now includes the package version: Data Quality Validation (v0.2.19)

Removed

  • generate_template and generate_raw_table_template removed from top-level public API (cognite_data_quality); the underlying template_generator module is still present

0.2.18 — 2026-05-28

Added

  • Skip views without SHACL rules in DataProduct mode: guard added in deploy_instance_workflows, data_product_sync handler, and deploy_validation_pipeline Mode 4

0.2.17 — 2026-05-27

Fixed

  • Use trigger_rule= instead of schedule= in WorkflowTriggerUpsert for data_product_sync scheduled trigger (SDK breaking change)

Added

  • sync_data_product_workflows() helper to deploy data product sync workflow and trigger
  • SHACL-missing guards hardened in data product sync handler

Changed

  • deploy_validation_pipeline deploys and triggers data_product_sync in DataProduct mode
  • Rewrite release.md with correct gh pr merge command for setting auto-merge commit body

0.2.16 — 2026-05-26

Added

  • sync_data_product_workflows() helper to deploy data product sync workflow and trigger
  • SHACL-missing guards hardened in data product sync handler

Changed

  • deploy_validation_pipeline deploys and triggers data_product_sync in DataProduct mode

0.2.15 — 2026-05-21

Fixed

  • Guard shacl_rules access in invoke.py _build_orchestrator_data when shacl_rules is None
  • Bake DATA_QUALITY_SPACE env var into function at deploy time
  • Import ordering in data_product_sync.py (lint)

0.2.14 — 2026-05-20

Fixed

  • shacl_rules is now optional in DeploymentSettings — settings.yaml files using config_source: "dataproduct" with external_dataproducts no longer need a shacl_rules: section (rules come from the RuleSet API, not local TTL files). A validator still requires shacl_rules when config_source is "yaml" so existing deployments get a clear error if it is accidentally omitted.
  • Orchestrator: replace schedule.get_input_data() with client.functions.schedules.get_input_data(id=schedule.id) to avoid SDK v8 AsyncCogniteClient error when checking schedule ownership before deletion.

0.2.13 — 2026-05-18

Fixed

  • Sort imports in test_cdf_sdk_in_construct.py and test_view_config_external_dp.py (lint)
  • Make shacl_rules and instance_spaces optional in ViewConfig for external DataProduct mode
  • Guard shacl_rules attribute access in _workflow_deployer.py and _dataproduct_publisher.py when shacl_rules is None

Changed

  • ViewConfig defaults: use_sync_cursor_mode=True and max_concurrent_executions=1
  • Replace DMS DataProductSyncState node with Workflow API version tag for DataProduct sync state check
  • _merge_local_view_config preserves API instance_spaces when local YAML omits them

Added

  • Tests for time series SPARQL CONSTRUCT rules and RuleEngineResult pipeline
  • Tests for ViewConfig optional fields and external DataProduct merge logic

Improved

  • Full Rule Engine (SHACL-AF) documentation across README, ARCHITECTURE, API and usage guides

0.2.12 — 2026-05-12

Fixed

  • Skip instance workflow deployment when config_source: dataproduct — workflow deployment in this mode is owned by the data_product_sync scheduled function, which sets the correct rule_set_id from the DataProduct API

Added

  • Document exact markdown list format required in the `

0.2.11 — 2026-05-12

Fixed

  • Orchestrator create_sync_trigger: pass WorkflowVersionId(external_id, version="1") to client.workflows.versions.retrieve() instead of the now-removed version= keyword argument. Previously the step silently failed with SyncWorkflowVersionAPI.retrieve() got an unexpected keyword argument 'version', causing the sync trigger to run with a stale job_run_id.

Changed

  • README_PYPI.md: remove hardcoded version number to avoid staleness.

0.2.10 — 2026-05-11

Fixed

  • Container creation and updates no longer fail with Unexpected field - constraintState (400) on SDK 8.4+ — switched from ContainerProperty (read model) to ContainerPropertyApply (write model) when building ContainerApply payloads

0.2.9 — 2026-05-11

Fixed

  • ContainerPropertyApply (write-only class) is now used instead of ContainerProperty when applying container schemas, avoiding the constraintState field added by SDK 8.4.0 which the CDF API rejects with HTTP 400.

0.2.8 — 2026-05-11

Fixed

  • Stable job_run_id across sync cursor invocations in DataProduct mode — cursor nodes now use a stable key (view_space/view_external_id/instance_space) and store the run ID as a property, so all invocations within a session share the same run ID and the Records API run picker no longer shows a new entry on every trigger.
  • cognite_data_quality/_function_code/requirements.txt was still pinned to cognite-sdk<8, causing deployed Cognite Functions to fail at runtime with v8-only types (NodeResultSetExpressionSync). Updated to >=8.0.0,<9.0.0 to match pyproject.toml.

Changed

  • Upgraded cognite-sdk dependency from >=7.0.0,<8.0.0 to >=8.0.0,<9.0.0. All instances.sync() calls updated to use NodeResultSetExpressionSync.

0.2.7 — 2026-05-08

Fixed

  • ViewConfigState upsert now stores instance_spaces as {"spaces": [...]} to satisfy the DMS Json() property constraint (plain Python lists caused a 400 error).

0.2.6 — 2026-05-08

Fixed

  • instance_spaces was stored in the ViewConfigState DMS container but not merged back into the ViewConfig at sync time — local YAML instance space overrides were silently dropped in both merge_view_config_state() and _merge_local_view_config()

0.2.5 — 2026-05-08

Added

  • Support for externally-owned DataProducts via external_dataproducts in settings.yaml — DataProduct and RuleSet definitions owned by other teams can now be referenced directly, with ViewConfig objects built from the DataProduct API rather than local YAML files.

Fixed

  • Per-view rule_set_id is now taken from each view's own config in DataProduct mode, instead of a single shared value.
  • dry_run parameter is now forwarded correctly to deploy_instance_workflows.
  • External DataProduct deployment results are returned separately from standard deployment results.

0.2.4 — 2026-05-07

Fixed

  • producedAt was written as a Unix millisecond integer (1778152091557); the Records API requires an ISO 8601 datetime string — converted using datetime.now(tz=timezone.utc)

0.2.3 — 2026-05-07

Fixed

  • Add common/rule_engine_records.py and common/shacl_classifier.py to the hardcoded file_list in _get_embedded_function_files — both files existed in source but were omitted, causing ModuleNotFoundError: No module named 'common.rule_engine_records' at runtime in deployed functions
  • Add tests/test_function_code_imports.py to catch future mismatches between _function_code/ source files and the packaging list in deploy.py

0.2.2 — 2026-05-07

Fixed

  • Remove targetClass from DataQualityValidationRecord write paths — the property was added to the code in 81a09ba but the CDF container schema was never updated, causing all records with violations to fail with Unknown property: dataQuality.DataQualityValidationRecord.targetClass

0.2.1 — 2026-05-07

Fixed

  • KeyError: 'shacl_file_external_id' in the orchestrator when deploy_validation_pipeline runs in DataProduct/RuleSet mode. The partition base config now uses .get() and forwards ruleset_references to partitioned workers so SHACL is loaded from the RuleSet API correctly.

0.2.0 — 2026-05-06

Added

  • SHACL-AF rule engine: sh:rule / sh:SPARQLRule inference rules execute automatically alongside quality shapes in a single pyshacl.validate(advanced=True) pass. Rules CONSTRUCT dqs:RuleEngineResult subjects that are materialised into a new RuleEngineResult CDF Records container.
  • RuleEngineResult container with properties ruleSetId, ruleSetVersion, ruleId, runId, resultType, focusNode, focusNodeInstance (DirectRelation), resultValue, resultPayload (JSON), causedBy (DirectRelation list), dataDomainExternalId, producedAt.
  • Auto-detection of inference shapes: shacl_classifier.py inspects the TTL graph at runtime; no config changes required.
  • dqs:dependsOn chained inference rules: deploy-time validator blocks malformed chains (missing dependency, wrong sh:order, cross-shape, cycle, duplicate rule ID). Runtime lineage captured via causedBy DirectRelation list on each record.
  • RecordsSettings.rule_engine_container and rule_engine_stream_id optional settings.
  • All four validation handlers (instance, instance-sync-cursor, partitioned, timeseries) automatically post inferences when the SHACL graph contains sh:rule triples.

0.1.30 — 2026-04-30

Fixed

  • deploy_validation_infrastructure no longer deploys instance workflows when config_source is "dataproduct" — in that mode workflows are managed by the DataProduct sync pipeline

0.1.29 — 2026-04-30

Fixed

  • _safe_apply_container now detects when an existing DMS container is missing properties from the current definition and applies an update, fixing deployment failures when new container properties are added across releases

0.1.28 — 2026-04-30

Added

  • deploy_validation_pipeline Mode 4: DataProduct + DMS mode — loads settings from DMS, discovers view configs from DataProduct API, and fires all orchestrators with wait=False; view_external_id now accepts str | list[str] | None to filter specific views
  • build_deployment_settings_from_props() as canonical DMS-to-DeploymentSettings mapping in _containers.py

Fixed

  • data_product_sync.py missing from function zip bundle (caused ModuleNotFoundError in deployed function)
  • DMS read failures: use ViewId instead of container ID so Properties.load() parses properties correctly
  • DMS write failures: use ContainerId for NodeOrEdgeData source in _save_sync_state
  • Timestamps stored as milliseconds integers instead of ISO 8601 strings (caused CogniteAPIError)
  • trigger_external_id_prefix never stored in DMS container schema or upsert dict
  • TriggerSettings.external_id_prefix missing default value (caused ValidationError for nodes deployed before fix)
  • list_versions default limit exceeded CDF API max of 10
  • shacl_file_external_id required in orchestrator even when ruleset_references is set; ruleset_references not forwarded to partition calls
  • Redundant CDF Files uploads in DataProduct mode (settings and view config)
  • IndexError when list_versions returns empty list in deploy_validation_pipeline
  • datetime.utcnow() deprecated in Python 3.12+; replaced with datetime.now(timezone.utc)
  • Broad except Exception narrowed to CogniteAPIError in _load_settings_from_dms

0.1.27 — 2026-04-20

Changed

  • DataProduct API: remove dataModel wrapper — views are now top-level on the version payload, each carrying its own space and version fields, matching the updated spec

Fixed

  • Null-safe JSON field access in DataProduct loader: use or [] / or "" instead of .get(key, default) so explicit null values in API responses are handled correctly

0.1.26 — 2026-04-15

Fixed

  • Skip RuleSet version creation when the version already exists (API returns 400 on duplicate) — makes SHACL publish idempotent on repeated deploys
  • Skip DataProduct version creation when the version already exists for the same reason

0.1.25 — 2026-04-15

Fixed

  • Use ContainerId instead of a raw tuple for NodeOrEdgeData.source in _upsert_settings_to_dms — raw tuples cause a TypeError when the SDK serializes the node apply request
  • Pin cognite-sdk>=7.0.0,<8.0.0 to prevent ContainerProperty serialization breakage introduced in SDK 8.x (constraintState field rejected by DMS API)

0.1.24 — 2026-04-15

Added

  • config_source=dataproduct mode: load view configs directly from a live CDF DataProduct instead of local YAML files
  • data_product_sync handler to keep RuleSets and DataProduct versions in sync with deployed SHACL rules and view configs
  • Thin HTTP wrappers for the alpha DataProduct and RuleSet CDF APIs (_dataproduct_client.py, _dataproduct_publisher.py, _dataproduct_loader.py)
  • schema_space and datamodel_external_id fields on DataProductRef for explicit data model targeting

Fixed

  • Aligned DataProduct version API payload with live API schema: views nested under dataModel, space/version on the dataModel not individual view entries
  • create_version no longer includes views at creation time (API rejects it)
  • Corrected status enum values and instanceSpaces/schemaSpace fields in DataProduct version payload
  • RuleSet version GET endpoint and DMS sources lookup

Changed

  • Extracted deploy_instance_workflows into a standalone _workflow_deployer module for lighter imports

0.1.23 — 2026-04-08

Added

  • Log the effective record landing space once per posting call so operators can confirm in function logs whether records are going to the instance space, a custom override space, or the default dataQuality space

0.1.22 — 2026-04-08

Added

  • use_instance_space option to write validation records into the validated instance's space
  • records_space override in view YAML records config

Changed

  • Updated readthedocs: removed internal environment name refs, clarified local vs CDF Function execution

0.1.21 — 2026-03-30

Fixed

  • Rewind cursor to before failed chunk so records are not lost on retry
  • Revert verbose default back to False in sync cursor handler to match partitioned handler
  • Prevent false retries on stream-full 400 errors
  • Handle sync_limits_exhausted in orchestrator

0.1.20 — 2026-03-26

Fixed

  • Version logging: cognite-data-quality package version is now printed at the start of every function invocation, making it easy to verify which version is running from function logs.

0.1.19 — 2026-03-26

Fixed

  • Explicit GC after each validation chunk: rdflib Graph objects have internal circular references that Python's cyclic GC may not reclaim promptly during tight loops. Even with a fresh NeatSession per chunk, unreleased Graph objects from prior chunks accumulate in memory until OOM. Fixed by explicitly del neat, report_graph and calling gc.collect() after violations are extracted in both instance_sync_cursor_validation.py and partitioned_validation.py.

0.1.18 — 2026-03-26

Fixed

  • NeatSession recreated per chunk: NeatSession was instantiated once before the validation loop in both instance_sync_cursor_validation.py and partitioned_validation.py. Its internal RDF graph accumulated across chunks without being cleared, causing monotonic memory growth and OOM on multi-chunk validations. Fixed by moving NeatSession(...) inside the loop, right before each with_shacl() call.
  • Sync trigger restored to view-scoped filter: Commit 1851a26 removed HasData(views=[view_id]) to avoid a deployment-time timeout, causing the trigger to fire for all node types in the instance spaces (Tags, Operations, Documents, ...) instead of only view members. HasData is restored.
  • Sync trigger uses no_backfill mode: Replaces Range(lastUpdatedTime) in the trigger filter. no_backfill prevents the trigger from returning existing (historical) instances on creation, eliminating the need for the range filter entirely. The initial_sync_cutoff (max lastUpdatedTime from the historic validation run) is passed to the sync validation function so it picks up exactly where historic processing left off.

0.1.17 — 2026-03-25

Fixed

  • Remove HasData from sync trigger DMS filter to prevent query timeout on large views. Combining HasData with Range(lastUpdatedTime) caused the CDF query validator to time out (~780k+ instances). Using In(space) + Range(lastUpdatedTime) is sufficient since the sync validation function re-fetches and validates only view members.

0.1.16 — 2026-03-25

Fixed

  • Include chunk_size, sync_batch_size, partition_count, partition_field, and use_sync_cursor_mode in workflow config hash so changes to these fields trigger redeployment instead of being silently ignored
  • Prevent double-counting of partial partition progress in orchestration status display
  • Log reverse query collection only once per unique query type instead of once per instance

0.1.15 — 2026-03-25

Fixed

  • Skip reverse relation loading explosion for non-primary classes when SHACL guidance exists
  • Fix dev.py changelog parser to strip content after --- (ThematicBreak) preventing release workflow failures
  • Auto-bump _function_code/requirements.txt cognite-data-quality pin via dev.py bump command, eliminating manual version bump commits in release branches

0.1.14 — 2026-03-23

Fixed

  • Restore live_in_progress sum and use validated_live in progress output
  • Show live validated_instances by summing in-progress processedCount

Changed

  • Split config and deployment into separate repo

0.1.13 — 2026-03-20

Fixed

  • Add orchestration_status.py to _get_embedded_function_files() and FunctionSettings.include_files in deploy.py — file was missing from both the deployed function bundle and the PyPI package, causing ModuleNotFoundError at runtime
  • Show live validated_instances by summing processedCount from in-progress partition cursors — previously showed 0 while partitions were running since the orchestration state is only updated on partition completion

0.1.12 — 2026-03-20

Added

  • New orchestration_status handler for querying orchestration run progress

Fixed

  • Support view_config_external_id shorthand in orchestrator handler
  • Simplify settings loading in orchestrator

0.1.11 — 2026-03-19

Fixed

  • Fix OrchestrationState container and sync cursor handler to support instance_spaces list — save_orchestration_state now serializes instance_spaces to the instance_space string field via json.dumps() with backward-compatible JSON parsing on load; sync cursor handler accepts both instance_space (singular) and instance_spaces (list), derives a stable cursor-state key, and uses an In filter to support multiple spaces

0.1.10 — 2026-03-19

Fixed

  • Fix OrchestrationState container and sync cursor handler to support instance_spaces list from the multi-instance-spaces feature — three related bugs: (1) save_orchestration_state remaps instance_spaces list to the instance_space string field via json.dumps(), with backward-compatible JSON parsing on load; (2) sync cursor handler now accepts both instance_space (singular) and instance_spaces (list), derives a stable cursor-state key, and uses an In filter to support multiple spaces

0.1.9 — 2026-03-19

Fixed

  • Fix invoke.py to use instance_spaces (list) in _build_orchestrator_data — ViewConfig.instance_space was renamed to instance_spaces in the multi-instance-spaces feature (#69) but invoke.py was not updated, causing orchestrator invocation to break

0.1.8 — 2026-03-19

Added

  • Upload view configs to CDF Files during deploy; view configs can be referenced by external ID for fully remote pipeline invocation
  • Upload settings to CDF Files to allow fully remote pipeline invocation without local config
  • Support multiple instance spaces per view config (instance_spaces list)
  • Batched CDF SDK prefetch for time series SHACL validation

Fixed

  • Use exact view external_id (case-sensitive) for view config file external ID
  • Pass data_quality_space to RAW validation state functions
  • Accept instance_spaces list in validate_config.py

Changed

  • Standardize sync_batch_size to 500 and increase chunk_size from 100 to 500 for all views in abp-dev and abp-test
  • Add RAW validation rules for RMDM-feeding tables in abp-dev
  • Extract _inject_prefetch_errors helper to reduce duplication in time series validation
  • Document CDF Files upload and remote pipeline invocation
  • Add tests for data_quality_space propagation in RAW validation
  • Bump cognite-data-quality to 0.1.8 in function_code requirements

0.1.7 — 2026-03-16

Fixed

  • Pin cognite-sdk>=7.65.3,<8 to prevent breaking changes from a potential SDK v8 release causing function startup failures
  • Remove inline # pinned comment from cognite-data-quality==0.1.6 line in requirements.txt

0.1.6 — 2026-03-16

Fixed

  • FunctionSchedule.get_input_data() used instead of non-existent .data attribute in orchestrator monitor mode cleanup, which caused monitor mode to silently fail when checking whether to delete a stale schedule after orchestration completes
  • passedValidation=True when there are no Violation-severity failures (only Warning/Info)
  • Bumps cognite-data-quality to ==0.1.6 in function requirements

0.1.5 — 2026-03-13

Added

  • Propagate orchestration_id as job_run_id across all handlers for consistent run tracking
  • Add job_run_id placeholder in deploy.py sync-cursor task data, patched at runtime by orchestrator

Changed

  • Change orchestration_id generated prefix from orch_ to job_ for consistency
  • Unify job_run_id fallback pattern across all handlers using job_ prefix
  • timeseries handler now injects job_run_id from data into records_config

0.1.4 — 2026-03-12

Fixed

  • Remove invalid . entry from requirements.txt that caused CDF function deployment to fail

Changed

  • Bump cognite-data-quality requirement to >=0.1.4 in function requirements

0.1.3 — 2026-03-12

Fixed

  • Fix cycle detection in _auto_load_references to use instance_id/source_view key instead of declaring_view/source_view, so all child Tag instances independently load their TimeSeries at depth 2
  • Fix time series SPARQL to use forward traversal via reverse-loaded triple instead of reverse parent traversal
  • Fix WorkOrder/Notification/FailureNotification SPARQL to use view property names instead of container property names
  • Fix invoke.py to derive data model params from view space/id/version when datamodel field is absent from view config

Added

  • Add pump time series coverage SHACL rule validating that every Pump has related pressure, temperature, and flow instrument tags with linked time series

Changed

  • Remove redundant datamodel field from all view YAML configs across all environments; deploy logic now derives data model info from the view reference
  • Enable cog-ai environment in CI/CD deploy pipeline
  • Add sync_batch_size, use_sync_cursor_mode, max_concurrent_executions to all view configs
  • Reduce sync_batch_size to 500 for operation and tag views to avoid OOM during sync processing
  • Remove neat:reverseDirectRelation from compressor/heatexchanger/electricmotor SHACL rules

0.1.2 — 2026-03-10

Fixed

  • Add pandas as explicit dependency for inlined SHACL helpers

Improved

  • Remove thisisneat references, update docs to reflect inlined SHACL validation
  • Remove outdated "What's New" section from documentation index

0.1.1 — 2026-03-10

Fixed

  • Stop sync cursor loop on empty chunk to match partitioned validation behavior
  • Save cursor state before breaking on empty chunk
  • Wire force_views flag through to workflow redeployment

Added

  • Cursor-based instance sync validation handler
  • Pump completeness SHACL rules (docs, WOs, notifications, failure mechanism)
  • SHACL rules extended to Pump, HeatExchanger, Compressor views
  • maxConcurrentExecutions support for instance validation workflows

Improved

  • Updated thisisneat to 0.2.5 with SHACL-guided auto-loading

0.1.0 — 2026-03-02

Added

  • PyPI publishing via CI (release workflow)
  • dev.py CLI for version bump and changelog from merge commit
  • cognite_data_quality/_version.py for version tracking
  • CD environment support with PYPI_API_TOKEN

Changed

  • release.md: Added PyPI publishing documentation
  • pyproject.toml: Added dev dependencies (typer, marko, packaging, twine)