Skip to content

Confirm that rules work

There are two tracks, depending on whether you are iterating on rules or re-validating the estate.

Historic validation always reads what is published in the live DataProduct and RuleSet APIs. Publishing new TTL locally (or deploying workflows) without a new published DataProduct + RuleSet version does not change the rules historic uses.

Choose a track

Goal What you do
Iterate on new rules (fast feedback) run_validation() locally — more setup, much faster feedback. Does not use the historic queue or Records unless you opt in.
New rules across the whole estate Publish a new DataProduct + RuleSet version, then enqueue_historic_validation() without force.
New full scan with the same published rules enqueue_historic_validation(..., force=True)

Use A when you are tweaking SHACL (including SPARQL / consistency). Use B when you need a full historic run against published rules in CDF.

A. Fast iteration on rules: run_validation()

run_validation() runs SHACL in your local Python process against a sample of DMS instances. It does not start workflows, the historic queue, or Records (unless you set post_to_records=True). You see hits and auto-load logs immediately (print_output=True, verbose=True).

It needs a local environment: Python, CDF credentials, and a TTL file or view YAML.

from cognite_data_quality import run_validation

result = run_validation(
    client=client,
    rules_path="views/my_view.yaml",
    rules_format="yaml",
    limit=20,
    print_output=True,
    post_to_records=False,
)

This path validates the rules you pass in (rules_path), not necessarily the latest published RuleSet in CDF.

Limits: per-instance pyshacl on a limited sample. Global uniqueness (dqs:uniquenessConstraint) and estate-wide edge existence need the deployed executors — see Uniqueness and Edge existence.

Full parameters: Run validation locally.

Alternative without a local runner: Function test type

If the validation Function is already deployed, you can exercise a sample of instances in CDF without writing Records. Invoke with validation_type: "test" (see Invoke Cognite Functions). This is still a sample check, not an estate historic scan.

B. Full estate scan: enqueue_historic_validation()

Historic always validates the RuleSets attached to the published DataProduct version in CDF. It does not pick up unpublished local TTL.

If historic is skipped because a job is already completed for the same DataProduct version, that is expected: enqueue is idempotent per (view, data_product_version).

from cognite_data_quality import enqueue_historic_validation

enqueue_historic_validation(
    client,
    data_product_external_id="my-data-product",
    data_quality_space="sp_data_quality",
    trigger_queue_manager=True,
)

Omit force when you have just published a new DataProduct + RuleSet version and want the first historic for that version.

Equivalent pipeline helper:

from cognite_data_quality import deploy_validation_pipeline

deploy_validation_pipeline(
    client,
    data_product_external_id="my-data-product",
    data_quality_space="sp_data_quality",
    historic_mode="enqueue",
    trigger_queue_manager=True,
)

Infrastructure (deploy_validation_infrastructure(), historic queue manager) must already be in place. Details: Deploy → Historic queue enqueue mode.

Can you re-run historic without bumping the DataProduct version?

Yes, for a new full scan of the same published rules. Pass force=True:

from cognite_data_quality import enqueue_historic_validation

enqueue_historic_validation(
    client,
    data_product_external_id="my-data-product",
    data_quality_space="sp_data_quality",
    force=True,
    trigger_queue_manager=True,
)

The same flag on the pipeline helper: deploy_validation_pipeline(..., historic_mode="enqueue", force=True).

What force=True does and does not do
Does Re-queue a completed historic job for the current published DataProduct / RuleSet version (including views with historic_reprocess_on_update: false).
Does not Load unpublished rules. The run still uses the DataProduct and RuleSet versions that already exist in CDF.
Still skips Jobs that are already queued or running (a view never gets two active historic orchestrations).

trigger_queue_manager=True starts dq-historic-queue-manager immediately. Without it, the job stays queued until the next cron.

To put new rules on the whole estate: publish a new DataProduct + RuleSet version (for example with Cognite Toolkit), then enqueue without force. force=True is only for another full scan with the same published rules.

Do not confuse enqueue force=True with data_product_sync payload force: true (that redeploys workflows and does not re-queue a completed historic job). See Deploy.

Previous section

Next section