Skip to content

Troubleshooting

View function logs

from cognite.client import CogniteClient

client = CogniteClient()

calls = client.functions.calls.list(
    function_external_id="data-quality-validation",
    limit=10,
)

logs = client.functions.calls.get_logs(call_id=calls[0].id)
print(logs)

Check workflow execution status

executions = client.workflows.executions.list(
    workflow_external_id="dq-shacl-equipment",
    limit=10,
)

for execution in executions:
    print(f"Status: {execution.status}, Started: {execution.started_at}")

Workflow not triggering on data changes

Symptoms: Instance changes in DMS don't start the validation workflow.

Check:

  1. Retrieve the trigger and verify it is active: 
    trigger = client.workflows.triggers.retrieve(external_id="dq-shacl-trigger-<view>")
print(trigger.trigger_type, trigger.is_active)
  2. Verify the instance space in the trigger filter matches the space where data changes occur.
  3. Confirm the workflow version in the trigger matches the deployed workflow version.
  4. If you deployed with --skip-trigger, redeploy without that flag.

Uniqueness workflow not deployed

Symptoms: View has dqs:unique in SHACL but no dq-{view}-uniqueness workflow in CDF.

Check:

  1. Confirm the constraint's sh:targetClass matches the view's data model class (deploy scans SHACL for the view's external_id).
  2. For DataProduct/RuleSet mode, verify RuleSetClient.get_version() returns TTL with the constraint (deploy fetches RuleSet content at deploy time).
  3. Check deploy logs for Skipping uniqueness workflow ... no SHACL uniqueness constraints found or SHACL parse warnings.
  4. Ensure uniqueness_cron is not set to null in view YAML (that disables the scheduled workflow).

Uniqueness not running after sync

Expected behaviour. Sync-cursor (instance_sync_cursor) runs incremental pyshacl only. Global uniqueness runs on the scheduled dq-{view}-uniqueness workflow, not after each sync completion.

To run on demand: call_validate_shacl() with validation_type: shacl or trigger the uniqueness workflow manually.

Uniqueness workflow runs but no records appear

Symptoms: The scheduled uniqueness workflow succeeds, SHACL discovery finds dqs:uniquenessConstraint, and the investigation script reports duplicates, but the Records stream has no uniqueness rows.

Check:

  1. Confirm function logs include job_run_id on the AGGREGATE UNIQUENESS VALIDATION banner. Scheduled uniqueness mints job_{view}_uniqueness_{epoch} when the cron trigger omits job_run_id. MISSING_JOB_RUN_ID now means the low-level aggregate_uniqueness executor was invoked without an id (the function call should be Failed).
  2. Confirm logs include [aggregate_uniqueness] outcome ... recordsPosted=.... Absence of Posting batch means posting was never attempted.
  3. Read the function return JSON for globalRuleResults[].result (recordsPosted, violatingBuckets, errorCode). Do not rely on the short log banner alone, especially when verbose: false.
  4. Filter Records by ruleSetId / validationType (single_violation or group_violation). Uniqueness records set dimensions to ["Uniqueness"] (or the dq:dimension on the NodeShape). dq:dimension is not required for uniqueness to run. After this release, uniqueness rows no longer use nested dimension: Consistency — update dashboard filters that keyed uniqueness findings on Consistency.
  5. Confirm the uniqueness workflow is the scheduled dq-{view}-uniqueness path (validation_type: shacl). Direct aggregate_uniqueness invokes still require an explicit job_run_id.

Uniqueness duplicates still exist but few/no new failure records

Often expected behaviour. Uniqueness now suppresses repeated failure writes for unchanged instances.

Suppression rule:

  • If latest existing failed record for a focusNode has record.lastUpdatedTime >= instance.lastUpdatedTime, runtime suppresses writing a new failure record.

Check:

  1. Inspect uniqueness run output fields:
  2. violatingInstanceCount
  3. recordsPosted
  4. recordsSuppressed
  5. If recordsSuppressed is high and recordsPosted is low, dedupe is likely working as intended.
  6. Update an affected instance and rerun uniqueness — a new failure record should appear if the violation remains.

From validation records:

  1. Check grouped uniqueness records for typed overflow signals:
  2. groupViolationType = "global_overflow" (run-level overflow)
  3. groupViolationType = "value_overflow" (single-value overflow)
  4. If either overflow type is present, this run hit a guardrail and more duplicate groups may still remain.
  5. Fix the reported duplicates and rerun uniqueness until overflow records disappear.

Note:

  • If no new records are written (for example due to dedupe suppression), users may not see a new overflow record for that run.
  • In that case, ask an operator to confirm run-level warnings from workflow/function output.

Grouped uniqueness records are explicitly marked with:

  • validationType = "group_violation"
  • groupViolationType = "global_overflow" | "value_overflow" | "pass"

"Failed to load SHACL rules"

Symptoms: Function logs show an error loading rules.

Instance, time series, or uniqueness validation

Check:

  1. Verify ruleset_references is present on the workflow trigger input or function payload.
  2. Confirm each referenced RuleSet version exists and contains Turtle rules.
  3. After upgrading to 0.4.0, redeploy workflows — handlers no longer load SHACL from shacl_rules_file_external_id.
  4. Validate Turtle syntax locally: 
    python scripts/validate_config.py

RAW table validation

Check:

  1. Verify the SHACL file exists in CDF Files with the expected external ID.
  2. Confirm the external ID in the RAW table config (shacl_rules.external_id) matches the file.
  3. Verify the file is in the correct CDF dataset (shacl_rules.dataset_external_id in settings.yaml).

"No instances to validate"

Symptoms: Validation completes with zero instances processed.

Check:

  1. Confirm the DMS filter in the view config or time series config is correct.
  2. Verify instances exist in the configured space.
  3. Ensure the view external ID and version match the actual view in CDF.

Zero violations when you expect some

Symptoms: Validation runs but produces no violations for a rule you expect to fire.

Check:

  1. The sh:targetClass in the SHACL rule targets a concrete view type, not a CDF interface.
    Interfaces (e.g. MMSINumber) are not used as rdf:type in the RDF graph — concrete view types (e.g. NavigationAid) are. A shape targeting an interface silently matches nothing.
  2. Run with print_output=True and verbose=True to see per-instance output and zero-instance class warnings.
  3. Check auto_load_depth — if the rule uses properties from referenced instances, depth 0 won't load them.

See SHACL rules and CDF data models for details.

Function dependency errors

"ModuleNotFoundError: No module named 'cognite_data_quality'"

The function was deployed before the package was published to PyPI, or is pinned to an old version.

  1. Publish the latest package: python -m build && twine upload dist/*
  2. Force redeploy: python scripts/deploy_infrastructure.py --env <env> --force

RAW table validation issues

"RAW table not found"

  • Verify the RAW database and table names in the config match what exists in CDF.
  • Confirm the service principal has read access to the RAW table.

"No rows updated since last run"

Normal for incremental validation when no changes occurred. Check RawValidationState in the dataQuality DMS space for last_processed_timestamp to confirm.

Cursor state not saving

  • Ensure the dataQuality space exists in CDF.
  • Verify the service principal has write access to the dataQuality space.
  • Run deploy_validation_infrastructure() to ensure state containers (FunctionValidationState, RawValidationState, etc.) exist.

Records API returns no results

Symptoms: Filter queries return empty items.

Check:

  1. Confirm client.config.headers["cdf-version"] = "alpha" is set before the request.
  2. Verify the stream ID in the query matches records.stream_id in settings.yaml.
  3. Check the time range — Records API filters use milliseconds since epoch. Widen lastUpdatedTime to confirm records exist.

Sync-cursor validation issues

Sync lock / data_quality_space errors

Symptoms: Sync-cursor fails immediately with sync lock errors, for example space must not be null.

Check:

  1. Omit data_quality_space from trigger/workflow input to use the default dataQuality, or pass a non-empty space string. The sync-cursor handler normalizes null/empty to dataQuality via resolve_data_quality_space() — explicit null in upstream workflow refs can still break other handlers if not stripped.
  2. Confirm the dataQuality space exists and the function principal can read/write DMS state there.
  3. See Sync-cursor and historic orchestration.

Full backlog replay after historic

Symptoms: Sync-cursor processes tens of thousands of instances after historic completed; steady "Sync lock held" logs.

Cause: Stale sync cursor preserved across historic handoff (fixed in 0.4.1).

Fix: Upgrade to 0.4.1+, re-run historic orchestration for the view, or manually reset sync cursor state after confirming historic completed.

No sync trigger after deploy

Expected: Deploy does not create DMS instance sync triggers. The historic orchestrator creates them in Step 3.

Fix: Run deploy_validation_pipeline() or wait for version-change historic enqueue via data_product_sync.

Note: Sync trigger creation can fail inside the orchestrator (timeout, secrets). Historic partitions may still complete; check orchestrator logs for Sync trigger: not created and sync_trigger_error.

job_run_id and historic runs

Symptoms: Missing job_run_id error from sync-cursor or the low-level aggregate uniqueness executor.

Fix: For instance sync / historic, pass explicit job_run_id in workflow trigger input (format: job_{view}_{epoch_seconds}). Scheduled uniqueness mints job_{view}_uniqueness_{epoch} when omitted. Direct aggregate_uniqueness calls still need an explicit id.

Symptoms: Dashboard shows no records for a historic run you triggered.

Check:

  1. Filter on jobRunId using the orchestrator id (job_*), not legacy hq_* queue ids.
  2. Confirm orchestrator and partitions used the same job_run_id from deploy_validation_pipeline().

See Invoke → job_run_id.

Historic validation processes fewer instances than expected

Symptoms: Auto-enqueued historic run validates far fewer instances than expected, or validates instances outside the intended site space.

Check:

  1. HistoricJobQueue rebuilds orchestrator input from the live DataProduct API at job start (instanceSpaces.read), then overlays ViewConfigState (including YAML instance_spaces when stored).
  2. Compare orchestrator payload instance_spaces in logs with both the DataProduct view definition and the ViewConfigState node.
  3. If YAML instance_spaces is narrower than DataProduct instanceSpaces.read (for example a single site space on a multi-site DataProduct), the YAML override should win after deploy_validation_infrastructure(). Redeploy infrastructure if the override was added after the last ViewConfigState upsert.
  4. If no override is stored, the live DataProduct instanceSpaces.read list is used.

Edge existence issues

Symptoms: Edge rules deploy but no baseline coverage after version change.

Check:

  1. Confirm historic orchestrator triggered Step 3b (edge_existence_baseline call ids in orchestrator logs / OrchestrationState.edge_baseline_call_ids).
  2. Baseline runs asynchronously — orchestrator completion does not mean baseline finished. Check baseline Function calls and records.
  3. On baseline timeout, re-invoke edge_existence_baseline manually with the same job_run_id (no orchestrator auto-resume).
  4. For unchanged DP version needing workflow/trigger refresh: bump DataProduct version or run data_product_sync with force: true.
  5. For unchanged version needing full re-scan without historic: invoke baseline manually (force_rerun).

Symptoms: Incremental edge validation never fires.

Check:

  1. Verify two modeling triggers per rule (node + edge) on {dp_wf}-edge-existence.
  2. Confirm trigger sync_kind routing — node changes evaluate focus nodes; edge changes resolve focus from edge endpoints.

Symptoms: False failures using sh:minCount on MultiEdgeConnection property paths.

Fix: Use dqs:edgeExistenceConstraint instead — see Edge existence.

Empty dimensions on validation records

Symptoms: Rules include dq:dimension but dimensions is empty on records.

Check:

  1. @prefix dq: must be http://purl.org/cognite/cdf_dq/DataQuality/ (not http://example.org/dataquality/).
  2. Redeploy CDF Function with cognite-data-quality ≥ 0.4.3.
  3. Function logs: Dimension lookup: N named shapes, M property paths — if both zero, namespace mismatch.

See Dimensions in SHACL Rules.