Building an app on the data quality engine

This page is the contract for an application that publishes quality rules or shows validation results. The engine writes the records. The app decides which rules to publish and which of those records to count.

Three layers

Layer	What it is	What the screen uses it for
Instances and views	Operational data in CDF	Counts, deep links, schema
Data Product and RuleSets	Published rules in quality.rules[]	Which rules are listed, Publish, which ruleset a view uses
Validation Records	Output of a validation job, inside a time window	Pass, fail, warning, and rule rows

An empty score means no records in the selected window. It does not mean the view, the RuleSet, or the instances are gone.

Publishing rules

One Data Product binds many views. Each view has its own entry in quality.rules: { externalId, version }.

Do

1. List Data Product versions and RuleSet versions with a limit of 100 or more.
2. Take the current Data Product version as the highest semantic version among published items.
3. Refuse to publish when quality.rules is empty. Do not invent a binding, and do not store the edit as a CDF File.
4. If the product already has 10 versions, or this RuleSet already has 50, stop before any write and ask the author which versions to delete. CDF rejects another version at those caps. Cancel deletes nothing and creates nothing. The current published Data Product version cannot be deleted. A RuleSet version that a surviving Data Product still names cannot be deleted.
5. Choose the next RuleSet version as one patch above the highest existing version, including versions nothing currently points at.
6. POST that RuleSet version, then POST a new Data Product version whose quality.rules replaces only the edited external id and keeps every sibling. Refuse a shorter list.
7. Reload the RuleSet version the new entry names.
8. Expect one historic validation run, and one jobRunId, for that publish. Do not start a second historic run because a status lookup was empty while the first run is still going.

Test, in-session edits, and an agent draft do not publish. The button label is Publish. sh:targetClass uses the operational view space, not the Data Product schema space.

Do not

• PATCH an existing RuleSet or Data Product version.
• Publish a list that keeps only the ruleset you just edited when other views had bindings.
• When the named RuleSet version is missing, open a different ruleset or the highest version that happens to remain. Show RuleSet binding broken for {externalId}@{version}. The only allowed replacement is the same external id at a different version on the latest published quality.rules.

Finding the rules for a view

Keep one row per stream and view. Do not collapse rows that share a rule_set_id. Keep a card when the trigger is paused.

With one binding, use it. With several, prefer an external id that matches the view name. If none match, keep the row's own rule_set_id when that id is in quality.rules. Otherwise leave the row as discovered. Do not attach a sibling ruleset.

App discovery (name match / row rule_set_id) is for UI cards and editors. After deploy / data_product_sync, the engine binds each view's ruleset_references and Records ruleSetId by focus target class, not by quality.rules order alone. Prefer querying Records with the engine-stamped ruleSetId (or the view card after rebind from a healthy sync), not blindly with quality.rules[0].

Engine focus binding (one RuleSet per view)

On Data Product load (load_views_from_data_product / sync), each view should end up with the RuleSet(s) whose focus classes match that view — ideally one. Neighbour stubs in another view's Turtle must not pull that RuleSet onto the neighbour view.

Skipped as neighbour stubs (do not contribute focus classes):

• Shape local names ending in ExistenceShape
• sh:node objects without focus constraints (sh:property, dqs:uniquenessConstraint / unique / edgeExistenceConstraint, sh:rule, or sh:and / or / not / xone)
• Bare sibling sh:targetClass NodeShapes when another shape in the graph already carries constraints
• SPARQL alone on an sh:node object (does not make that stub a focus contributor)

Kept as focus: property / dqs / rule / logical shapes; SPARQL-only focus shapes that are not sh:node stubs; sh:target / sh:SPARQLTarget selects on those focus shapes.

Records stamp: when several RuleSets still match, exact focus (focus URIs equal {view}) wins over multi-class matches; quality.rules order breaks remaining ties. Requires cognite-data-quality 0.5.0b6 or later for the full contract.

Querying immutable records

Compliance history lives on an immutable stream (BasicArchive). Send cdf-version: alpha on Records calls.

Every filter and every aggregate on that stream includes lastUpdatedTime (milliseconds). Omitting it fails. A range the API calls too wide fails with an interval error — split the lookback into smaller ranges and combine the results. Do not drop the time range to make the call succeed.

Use the same lastUpdatedTime range on every cursor page. Follow nextCursor until it is absent. A limit of 1000 is one page.

On an immutable stream one externalId matches one row per run, not one row forever. Select a run with jobRunId.

Property paths are [space, container, propertyName], for example ["dataQuality", "DataQualityValidationRecord", "ruleSetId"].

Filter

POST /streams/{stream_id}/records/filter

{
  "lastUpdatedTime": {"gt": 1710000000000},
  "limit": 1000,
  "filter": {
    "and": [
      {"equals": {"property": ["dataQuality", "DataQualityValidationRecord", "ruleSetId"], "value": "EquipmentSHACLv1"}},
      {"equals": {"property": ["dataQuality", "DataQualityValidationRecord", "jobRunId"], "value": "job_equipment_1710000000"}},
      {"not": {"equals": {"property": ["dataQuality", "DataQualityValidationRecord", "validationType"], "value": "group_violation"}}}
    ]
  }
}

Instance screens exclude grouped uniqueness (group_violation). When the ruleset id is known, also leave edge-existence rows out of the instance card. Uniqueness summaries are a separate query.

Aggregate

POST /streams/{stream_id}/records/aggregate

Count the window with one call that has no listed-rule filter:

{
  "lastUpdatedTime": {"gt": 1710000000000},
  "filter": {"equals": {"property": ["dataQuality", "DataQualityValidationRecord", "ruleSetId"], "value": "EquipmentSHACLv1"}},
  "aggregates": {
    "totalCount": {"count": {}},
    "jobRuns": {"uniqueValues": {"property": ["dataQuality", "DataQualityValidationRecord", "jobRunId"]}}
  }
}

Read aggregates.totalCount.count and aggregates.jobRuns.uniqueValueBuckets.

Count listed hits with a second call. Same time range. Add a hit filter:

• names → containsAny on failedConstraintNames
• constraint ids → containsAny on failedConstraints
• both → or of those two clauses, on the request filter

Do not put that or inside the aggregate on the totals call. The API rejects it, and the screen shows zeros while the job picker still has the run.

passedValidation is false when any stored check failed, including schema checks and rules that are not on screen. Do not use it as the card total. Passed = total − listed violation hits. Warnings are a third call with the warning names and ids.

If the rule list failed to load, show not validated.

Default lookback is 6 days. Offer 1, 6, 14, 30, and 90. Remember the choice. Empty text names the day count and says the data was not validated recently. It does not say "No data". Widening the window does not start a job.

A product's numbers are the sums of its view cards. Fleet numbers are the sums of the product cards. A record that fails two listed rules counts once on the card and once on each rule row. The caption uses the unique record count.

Example: 18,416 records and no listed violation hit means 18,416 passed and 0 failed. The same records with 86 listed violation hits means 18,330 passed and 86 failed.

The job picker lists distinct jobRunId values in the window. It is not filtered by the listed rules. Hide uniqueness ids (_uniqueness_), time-series retention ids (ts_…), edge-retention ids (edge_…), and ids that are only job_ plus a timestamp. Default to the newest remaining run that has records.

SPARQL rules match by sh:name only. Do not match the shared SPARQL constraint component URI. Skip informational rules. Keep one record per instance.

Navigation

Open the instance from focusNodeInstance (space and externalId). focusNode is often the related node, and its id may already be percent-encoded. Linking from it opens the wrong object or encodes a colon twice.

Fusion path for the view on screen:

/search/{viewSpace}/{viewVersion}/{viewExternalId}/{instanceSpace}/{id}

Do not insert /search/asset/ and do not substitute a default Tag view.

To go from an instance back to its records, query the Records API on focusNodeInstance. An in filter accepts at most 100 {space, externalId} values. Grouped uniqueness, RAW rows, and edge-view findings omit focusNodeInstance — filter those on the stream instead. See Reverse navigation.

Failing text stays dark on a light page. "N pass this rule" still says when that record failed a different rule.

Audit an app

Review the RuleSet API and the Records queries separately.

RuleSet API — do not ship if any check fails

• Publish creates a new RuleSet version and a new Data Product version, and never patches.
• Sibling bindings survive, and a shorter list is refused.
• An empty quality.rules stops the publish.
• Current version is the highest published semantic version, and lists use a high limit.
• The next version accounts for orphan versions.
• At the 10 / 50 caps, nothing is written until the author confirms. Cancel writes nothing. The current Data Product version and any RuleSet a survivor still names cannot be deleted.
• A missing RuleSet version stays visible.
• Test and draft actions do not publish.
• sh:targetClass uses the operational view space.
• One publish has one historic jobRunId.

Records queries — unsafe if any check fails

• Empty windows do not say "No data".
• Changing the window does not start a job.
• Views are not deduplicated by rule_set_id.
• A rule list that failed to load shows not validated.
• Pass, fail, and warning use the listed rules. Passed is total minus failed.
• The record total and the hit counts are separate requests, and a combined name/URI filter is not placed inside the totals aggregate.
• SPARQL rules match by name, not by the shared SPARQL component URI.
• Product and fleet numbers are sums of the child cards.
• Every immutable filter and aggregate sends lastUpdatedTime, and cursor pages reuse that range.
• Instance cards exclude grouped uniqueness, and the client follows the cursor until it ends.
• Links use focusNodeInstance and the view path above. Reverse lookup sends at most 100 ids.
• The job picker omits uniqueness and retention runs.

Rating	When
Do not ship	Any RuleSet API check fails
Unsafe	Those checks pass or do not apply, and any Records check fails
Risky	Every check above passes, but the job picker, pass samples, failing text, or a SPARQL name still misleads
Safe to ship	Every check above passes, and the remaining gaps do not mislead
Contract complete	The checks above pass, and so do lookback presets, the post-publish reload, and the unique-record caption

Prove a write change on a Data Product that already has two quality.rules bindings. Prove a read change on an immutable stream: the totals call must include lastUpdatedTime and must stay non-zero when the listed-rule filter is empty.

See also

• Editing Data Products and RuleSets in Flows apps
• Records output
• Records streams
• Reverse navigation from instance views
• Rule sources
• Atlas rule authoring — ExistenceShape stubs