Lifecycle and CI/CD

Publish rules to the Data Product and RuleSet APIs, keep workflow definitions aligned with semver versions, and operate idempotent deploy/sync cycles in CI and production.

For deploying infrastructure and running validation pipelines, see Deploy infrastructure and pipeline. For rule-source policy, see Rule sources.

Data Product and RuleSet publishing

Set config_source: "dataproduct" in settings.yaml to publish SHACL rules to the CDF RuleSet API and view configs to the DataProduct API. This is the primary model for rule management and persistence. Each view config must declare which DataProduct(s) it belongs to via the dataproducts: list.

# settings.yaml
config_source: "dataproduct"

# views/my_view.yaml
shacl_rules:
  file: "my_view_shacl.ttl"
  external_id: "my_view_shacl"

dataproducts:
  - external_id: "my-data-product"
    version: "1.0.0"
    schema_space: "my_schema_space"

Schema space must be dedicated to the DataProduct

A DataProduct takes ownership of its schema_space, and CDF only lets a data-product-owned space contain views. Reusing a space that already holds data models, containers, or instances corrupts them (Could not upsert item '…' into space owned by data product …. Spaces owned by data products can only contain views.).

Deploy therefore fails if schema_space already exists and is not already owned by this DataProduct — the space must be created fresh for the DataProduct. To override this guard and reuse an existing space at your own risk, set allow_existing_schema_space: true on the DataProduct entry:

dataproducts:
  - external_id: "my-data-product"
    version: "1.0.0"
    schema_space: "my_schema_space"
    allow_existing_schema_space: true  # override the dedicated-space guard

cdf_* system spaces are exempt from the guard, and re-deploying a DataProduct that already owns the space is always allowed.

Brownfield: views in an operational space

When views already live in a shared operational space (for example sp_ent_ops), set schema_space to a dedicated DataProduct-owned space. Keep view.space as the operational space — validation queries instances there. Deploy publishes the view to the DataProduct catalog under schema_space (CDF requires catalog views[].space to equal schemaSpace or cdf_*), copies the view definition into that space when needed, and stores instanceSpaces.read from your YAML:

view:
  space: "sp_ent_ops"
  external_id: "COMCommissioningTag"
  version: "1.18.0"

instance_spaces:
  - "sp_ins_ygg"

dataproducts:
  - external_id: "dp_ent_commissioning"
    version: "1.0.0"
    schema_space: "sp_ent_dp_sif"

schema_space is required when view.space is not a cdf_* space. Omitting it no longer defaults to view.space (which would collide with the dedicated-space guard on brownfield projects).

In Data Product mode the view YAML uses the same fields as the standard config (partition settings, validation depth, sync cursor mode, etc.) — these are needed and applied during deploy. The records: block is the one exception: rule_set_id and rule_set_version are set automatically at deploy and should be omitted.

deploy_validation_infrastructure() stores each view's config as a ViewConfigState DMS node so runtime paths (data_product_sync, deploy_validation_pipeline(), historic queue rebuild, orchestrator create_sync_trigger()) use the same settings. YAML instance_space / instance_spaces is persisted there and applied at runtime — not only at publish. When no override is stored, DataProduct instanceSpaces.read remains the default.

Publish and version reuse

Publish behavior is payload-aware and idempotent:

• if the requested RuleSet/DataProduct version already exists, deploy skips version creation
• if the requested version is missing but payload is unchanged, deploy reuses the latest existing version instead of creating a new immutable version
• downstream references use the resolved version (the one that exists after publish/reuse), not just the requested version

config_source: yaml was removed in 0.4.0. See Migration to 0.4.0.

Time series rules in Data Product mode

Time series validation follows the same Data Product / RuleSet model as instance validation. Each timeseries YAML under timeseries/ can declare dataproducts: (one or more DataProducts) and a local TTL file; deploy publishes the TTL to the RuleSet API, adds a CogniteTimeSeries view entry to each DataProduct version, and stores operational settings (filter, schedule, backfill, rule_set_id) in one TimeseriesConfigState DMS node per DataProduct for sync-time replay. At sync time, only RuleSets whose SHACL sh:targetClass matches the timeseries datamodel are attached (so instance RuleSets on the same Data Product are ignored).

# timeseries/demo_sensors.yaml
name: demo-sensors
description: "Datapoint quality for demo sensors"

dataproducts:
  - external_id: "iot-data-product"
    version: "1.0.0"

shacl_rules:
  file: "demo_sensors_quality.ttl"
  external_id: "demo-sensors-shacl"

datamodel:
  space: "cdf_cdm"
  external_id: "CogniteTimeSeries"
  version: "v1"

filter:
  space: "sensor-space"
  external_id_prefix: "demo_"

schedule:
  cron: "0 * * * *"   # WorkflowScheduledTriggerRule — same pattern as other scheduled DQ workflows

records:
  data_domain_external_id: "iot-data-product"   # optional; defaults to the owning DataProduct per state node
  # Omit rule_set_id and rule_set_version — set at deploy from primary focus RuleSet externalId (fallback: view externalId) + RuleSet reference version

In Data Product mode:

• Workflow task payloads use ruleset_references (not shacl_rules_file_external_id).
• Instance validation records use the primary focus RuleSet externalId as rule_set_id — the RuleSet whose focus target classes match the view (exact focus preferred over multi-class / quality.rules order; neighbour *ExistenceShape / structural stubs ignored). Falls back to view externalId when no match. data_domain_external_id is the DataProduct external_id.
• Timeseries validation records use the RuleSet externalId as rule_set_id and the DataProduct external_id as data_domain_external_id.
• rule_set_version on validation records uses the RuleSet reference version from ruleset_references for that primary id (for example 1.3.0), not the DataProduct semver when they differ.
• For timeseries datapoint validation, rule_set_id is the RuleSet externalId; rule_set_version follows the same RuleSet reference version rule.
• Timeseries workflows carry [dp:{version}] tags; data_product_sync redeploys them when the DP version changes.

RuleEngineResult listener workflows

You can define scheduled listener workflows that consume changed RuleEngineResult records incrementally (for rule chaining pipelines).

Listener configs are deploy-side YAML (not stored on the Data Product API). Filter on record fields that upstream producers already write.

• Add YAML configs under rule_engine_result_sync/ (or set rule_engine_result_sync.config_dir in settings.yaml).
• deploy_validation_infrastructure() loads that directory automatically when the section exists (or pass rule_engine_result_sync_dir=... explicitly).
• Each config deploys one scheduled workflow and trigger that calls validation_type: "rule_engine_result_sync".
• Deployed workflows enforce max_concurrent_executions=1 (checkpoint safety).

# settings.yaml (optional — enables automatic listener deploy from config dir)
rule_engine_result_sync:
  config_dir: "rule_engine_result_sync"

Example listener config for an instance-validation Data Product:

name: maintenance_order_rule_a_listener
description: "Consume new RuleA outputs for maintenance orders"
listener_id: mo-rule-a-listener
source_mode: filter
stream_id: dq_validation_stream
records_space: dataQuality
records_container: RuleEngineResult
initial_watermark_ms: 1710000000000
initialize_cursor: "7d-ago"   # sync mode first-run bootstrap when no saved cursor
page_limit: 1000
limit: 1000
schedule:
  cron: "*/5 * * * *"
filter:
  rule_ids: ["RuleA"]
  data_domain_external_ids: ["equipment-data-product"]
  rule_set_ids: ["my-view-shacl-rules"]   # RuleSet externalId (or view externalId when no RuleSet match)
  rule_set_versions: ["1.2.0"]

For timeseries upstream producers, use the RuleSet externalId in rule_set_ids instead of the view external ID.

For an end-to-end chaining walkthrough, see Chained conditional logic.

Operational notes:

• source_mode: sync remains the default listener mode. Set initialize_cursor explicitly for first-run bootstrap (deploy default: 7d-ago).
• Scheduled workflows omit full record payloads by default (include_items: false); use record_external_ids in the handler response.
• In environments with strict filter interval limits, prefer source_mode: filter with bounded initial_watermark_ms for first-run bootstrap.
• Listener workflows are deployed with single concurrency to protect checkpoint correctness.

data_product_sync and historic queue

To deploy the data_product_sync workflow:

deploy_validation_infrastructure(
    client=client,
    settings_path=settings_path,
    views_dir=views_dir,
    deploy_data_product_sync=True,
    data_product_sync_cron="0 * * * *",
)

When deploy_data_product_sync=True, deployment also provisions a second scheduled workflow:

• dq-historic-queue-manager (default cron: */5 * * * *)
• Processes HistoricJobQueue sequentially (one running orchestration at a time)
• Starts the next queued historic job after the current one completes/fails
• When data_product_sync enqueues a historic job for a new ruleset/DataProduct version, any queued or running jobs for the same view and older version are marked superseded, the monitor schedule is removed, and orchestration state is cleaned up so the new version can start on the next queue-manager cycle
• Each historic orchestration relies on a per-view function schedule named Monitor {view} validation (every 2 minutes) to retrigger the orchestrator in monitor mode. If that schedule is missing while OrchestrationState is still in_progress/monitoring, the queue manager marks the job failed as orphaned so the queue does not block forever

You can override its schedule:

deploy_validation_infrastructure(
    client=client,
    settings_path=settings_path,
    views_dir=views_dir,
    deploy_data_product_sync=True,
    data_product_sync_cron="0 * * * *",
    historic_queue_manager_cron="*/10 * * * *",
)

Per-view control of automatic historic re-processing is available via:

historic_reprocess_on_update: false

Set this in a view YAML to prevent data_product_sync from auto-enqueueing historic jobs for that view on DataProduct/ruleset version updates.

data_product_sync enqueues historic jobs when it deploys workflows for a new DataProduct version. If workflows already carry the correct [dp:{version}] tag, sync skips deploy and does not enqueue — use enqueue_historic_validation() after infrastructure deploy in that case. See Deploy → Historic queue enqueue mode and Migration 0.4.4.

Externally-owned DataProducts

When another team owns the DataProducts and RuleSets in CDF, add external_dataproducts to settings.yaml. No local TTL is required — the deploy function reads views and rules from the DataProduct API and deploys workflows immediately.

Optional local view YAML files can supply operational overrides (for example instance_spaces, partition settings, sync cursor mode). These are matched to API-derived views by view.space and view.external_id — the same key used for ViewConfigState DMS nodes. Do not add a dataproducts: block on view YAML for externally-owned DataProducts; that field is for DataProducts this deploy publishes into.

# settings.yaml
config_source: "dataproduct"  # only supported value since 0.4.0 (default if omitted)

external_dataproducts:
  - external_id: "equipment-product"
    version: "latest"   # auto-picks the highest published semver
  - external_id: "iot-sensors-product"
    version: "1.2.0"    # pin to a specific version

# views/equipment.yaml — optional override for an externally-owned DataProduct
view:
  space: "my_space"
  external_id: "Equipment"
  version: "v1"
instance_spaces: ["my_instances"]
partition_count: 20
# no dataproducts: block — membership comes from the DataProduct API

deploy_validation_infrastructure(
    client=client,
    settings_path=settings_path,
    # views_dir is optional — external_dataproducts doesn't require local view YAMLs
)

deploy_validation_infrastructure() fetches each DataProduct from CDF via DataProductClient, builds ViewConfig objects from the DataProduct's views and quality.rules, merges any matching local YAML overrides, and deploys instance validation workflows directly. The rule_set_id on validation records is the primary focus RuleSet externalId from quality.rules (exact focus preferred; neighbour stubs skipped; falls back to view externalId when no match). See Building a quality app — Engine focus binding. data_domain_external_id tracks the DataProduct.

Operational view space: when a DataProduct catalogs views under a dedicated schemaSpace (for example sp_ent_dp_commissioning) while instances and RuleSet sh:targetClass URIs live in an operational space (sp_ent_ops), external-DataProduct deploy passes the same resolve_operational_view_space callback used by data_product_sync / historic enqueue. Loaded view.space becomes the operational space so SHACL rules match, workflows deploy with rules, and ViewConfigState upserts {ops}__{view} with view_space=ops. Catalog alias nodes {schemaSpace}__{view} keep that same operational view_space — they must not be overwritten with the schema space.

The deploy return value includes external_dataproduct_workflows[].view_deployments with per-view created, updated, or skipped status so pipelines can run first-time historic validation when a view is new to DQ, without relying on missing per-view workflows. See Deploy API → deploy_validation_infrastructure().

YAML instance_spaces (and other operational overrides) are persisted to ViewConfigState in DMS and reapplied at runtime by data_product_sync, deploy_validation_pipeline() (DataProduct mode), historic queue payload rebuild, and the orchestrator sync-trigger filter. DataProduct instanceSpaces.read is used only when no override is stored. Do not rely on publish-time merge alone — a later data_product_sync or historic run reloads views from the DataProduct API and needs the stored override.

Version changes in external_dataproducts are part of workflow change detection. Updating a pinned version in settings.yaml should trigger workflow redeployment even when SHACL content is unchanged.

For time series datapoint validation owned by an external DataProduct, operational settings must still be present as TimeseriesConfigState nodes (written during a prior deploy from local timeseries YAML). The sync handler then rebuilds TimeseriesConfig objects via load_timeseries_configs_from_data_product().

CI/CD patterns

Typical promotion flow:

1. Publish — deploy_validation_infrastructure() publishes local TTL to RuleSet + DataProduct APIs (or references external_dataproducts).
2. Sync — data_product_sync detects new [dp:{version}] tags and deploys consolidated workflows when the semver changes.
3. Historic — on version change, sync enqueues historic jobs; when workflows are already tagged, call enqueue_historic_validation() after deploy (see Migration 0.4.4).
4. Verify — spot-check workflow triggers, orchestrator completion, and Records jobRunId / ruleSetVersion.

Goal	Approach
Idempotent CI deploy	Rely on payload-aware publish reuse; bump DataProduct semver only when rules or deploy config change
Sequential historic after every deploy	deploy_validation_pipeline(..., historic_mode="enqueue", trigger_queue_manager=True) or enqueue_historic_validation()
Re-run historic without a version bump	enqueue_historic_validation(..., force=True, trigger_queue_manager=True) — all ruled views, including opted-out; does not redeploy workflows or load unpublished TTL. See Confirm that rules work
Refresh workflows without semver bump	Run data_product_sync with force: true — historic enqueue stays idempotent
Skip auto-historic on version bump	historic_reprocess_on_update: false on the view YAML

Related guides

• Confirm that rules work
• Deploy infrastructure and pipeline
• Invoke Cognite Functions
• Migration 0.4.0
• Migration 0.4.4

Previous section

• Deploy infrastructure and pipeline

Next section

• Migration 0.4.0