Skip to content

Edge existence validation

Procedure and reference for typed-edge cardinality (dqs:edgeExistenceConstraint). For the mode overview and industrial example, see Edge existence mode.

Prerequisites

  • dqs:edgeExistenceConstraint in RuleSet SHACL with {space}/{externalId} dqs:edgeType — see Rule sources.
  • Data Product + RuleSet deploy path (config_source: dataproduct); edge workflows are created by deploy / data_product_sync — see Deploy → Edge existence workflows.
  • Records stream if you query DataQualityValidationRecord output — see Records output reference.

Why not sh:minCount on the relation path?

Views that store relationships as typed edge instances often materialize a null relation property on the node. pySHACL on that dump reports missing edges incorrectly. Edge existence ignores the node dump for cardinality and queries the live edge graph instead.

Deploy model (DataProduct only)

Production deploy is DataProduct + RuleSet only (config_source: dataproduct). There is no YAML per-view edge path and no CDF Files TTL fallback for this feature.

When SHACL in the RuleSet contains one or more dqs:edgeExistenceConstraint blocks for a view:

Resource Pattern
Workflow {instance_workflow_id}-edge-existence
Task validation_type: edge_existence_event
Triggers 2 per rule: DMS node sync + DMS edge sync (no_backfill)
Trigger IDs {wf}-{view}-edge-{rule_id}-node / ...-edge

Instance sync triggers remain orchestrator-only. Edge existence triggers are created at deploy / data_product_sync time because rule fields (edge_type, counts, ruleset_references) are known then and steady-state evaluation needs no historic timestamp handoff.

Incremental evaluation (sync_kind)

sync_kind Source Behavior
node Node modeling trigger Evaluate each changed focus node (skip deleted)
edge Edge modeling trigger Resolve focus from the edge (start or end by dqs:direction), including deleted edges; re-count live edges for that focus

Edge existence does not use the per-view sync lease or initial_sync_cutoff handoff used by instance sync-cursor — each invocation performs a live edge count for the affected focus nodes.

Estate-wide baseline

Day-one / version-change coverage is triggered by the historic orchestrator:

  1. Historic orchestrator Step 3b fires one async edge_existence_baseline Function call per rule (wait=False) — fire-and-forget. The orchestrator does not monitor completion or auto-retrigger on timeout.
  2. The baseline handler paginates in-scope focus nodes via instances.sync() with sync_mode="two_phase" (not instances.list) and runs the same evaluator as steady-state.
  3. On Function timeout, pagination checkpoint in FunctionValidationState (key edge_baseline_{job_run_id}_{rule_id}) saves progress — manually re-invoke baseline with the same job_run_id to resume.

Historic orchestration success ≠ baseline complete. Check baseline Function calls and records separately.

data_product_sync skips workflow redeploy when workflows already carry [dp:{version}] — bump the DataProduct version or run sync with force: true to refresh -edge-existence workflows/triggers without a version change.

For an unchanged version where you still need a full re-scan without historic orchestration, invoke edge_existence_baseline manually with force_rerun: true.

What is out of scope (v1)

  • Cron / dqs:schedule on edge existence (use modeling triggers)
  • dqs:edgePropertyConstraint / dqs:edgeEndpointConstraint (not supported — use normal SHACL on an edge view in the Data Product; historic and sync-cursor query the edges channel — see Sync-cursor and historic orchestration)
  • Per-rule workflows (one consolidated -edge-existence workflow per DataProduct)

SHACL declaration

@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix dqs: <http://purl.org/cognite/dqs#> .
@prefix sp:  <http://purl.org/cognite/my_space/WorkOrder/> .

sp:WorkOrderShape a sh:NodeShape ;
    sh:targetClass sp:WorkOrder ;
    dqs:edgeExistenceConstraint [
        dqs:connectionProperty "workItems" ;
        dqs:edgeType "edge_sp/WorkItemEdge" ;
        dqs:direction "outwards" ;
        dqs:minCount 1 ;
        dqs:maxCount 50
    ] .
Field Required Notes
dqs:connectionProperty Yes Relation property name; rule_id becomes edge_exists_{property}
dqs:edgeType Yes Must be {space}/{externalId} (CDF space component required). Missing space component → constraint skipped at deploy
dqs:direction No outwards (default, focus = start) or inwards (focus = end)
dqs:minCount / dqs:maxCount At least one Inclusive; sh:minCount / sh:maxCount also accepted

Attach the constraint to a shape whose sh:targetClass matches the view/datamodel external ID used in the DataProduct.

Deploy and verify

  1. Publish SHACL to the RuleSet and bump the DataProduct version (or point external_dataproducts at the version that already has the rules).
  2. Let data_product_sync (or deploy_validation_infrastructure with external DPs) upsert {dp_wf}-edge-existence and the dual triggers.
  3. On version change, confirm historic orchestration includes Step 3b baseline jobs for each edge rule.
  4. After baseline, node/edge changes should keep Records current via modeling triggers.

Records contract

Edge existence writes the same DataQualityValidationRecord container as other validators:

Field Value
validationType Always "single_violation"
focusNode / focusNodeInstance Always the focus node
failedConstraints ["dqs:edge_existence::{relationProperty}"] on fail; [] on pass
Report detail sourceConstraintComponent: "dqs:edge_existence"
Stable externalId dq_{ruleSetId}_edge_{relationProperty}_{instanceSpace}_{externalId}

PASS and FAIL are both emitted. The _edge_{relationProperty}_ segment in the key keeps multiple relations on the same node from colliding on a mutable stream upsert.

See Records output for stream mutability and query examples.

Trigger budget

CDF limits triggers per project. Edge existence adds two triggers per rule:

triggers ≈ views × 2 (instance sync + uniqueness cron, when present)
         + 2 × edge_rules

Node change events can hit both the instance sync workflow and the edge-existence node trigger — plan capacity accordingly.

Best practices

  • Prefer dqs:edgeExistenceConstraint over node-dump sh:minCount for MultiEdgeConnection / typed edges.
  • Always use {space}/{externalId} in dqs:edgeType (both CDF space component and type external ID).
  • Treat PASS volume seriously on mutable streams — every focus node evaluation upserts a record.
  • Do not expect uniqueness-style cron; edge existence is modeling-trigger driven plus historic baseline.

Troubleshooting

Symptom Check
No -edge-existence workflow RuleSet SHACL for the view must contain dqs:edgeExistenceConstraint; sync must see [dp:{version}] on the workflow description
Constraint ignored at deploy dqs:edgeType missing space component ({space}/{externalId} required)
False fails with only sh:minCount Switch to edge existence — node dump may not materialize the relation
Stale FAIL after deletes Edge trigger with sync_kind: edge should re-evaluate focus when edges are deleted
Baseline incomplete / timeout Re-invoke edge_existence_baseline with same job_run_id; orchestrator does not auto-resume
Edge count capped at 10,000 Incremental evaluator uses instances.query with limit=10_000 per focus node — contact Cognite support if a single focus node can exceed this
Baseline never triggered Version-change historic path / Step 3b; check orchestrator warnings; for unchanged version use manual baseline

Previous section

Next section