Skip to content

Records streams — mutable vs immutable

What this is

Validation results are stored in CDF Records API streams. A stream's mutability is fixed at creation and cannot be changed later. Operators choose a stream template when the stream is first created; runtime code auto-detects mutability and picks the correct write endpoint.

When to use each template

Template (records.stream_type) Mutability Best for
BasicArchive (default) Immutable Audit history, per-run triage, time-window analytics
BasicLiveData Mutable Current-state KPIs, de-duplicated pass/fail counts
ImmutableTestStream Immutable Non-production / test streams

Set in settings.yaml:

records:
  stream_id: dq_validation_stream
  stream_type: BasicArchive   # or BasicLiveData

records.stream_type only affects stream creation (persisted in DMS as records_stream_type). At runtime the function calls GET /streams/{id} and dispatches to ingest or upsert.

Creating the stream

deploy_validation_infrastructure() ensures the Records container and state containers — it does not create the Records API stream. Operators must create the stream separately with the intended template before validation runs:

  • Production: create the stream in CDF (Console / API) with template BasicArchive or BasicLiveData, or call ensure_records_infrastructure() from a one-off script using the same stream_id and records.stream_type from settings.yaml.
  • Local validation: run_validation(..., post_to_records=True) calls ensure_records_infrastructure(), which creates the stream if missing using records.stream_type.

Mutability is fixed at creation. Changing records.stream_type in settings after the stream exists does not alter an existing stream.

Write behavior

Stream type Endpoint Behavior
Immutable POST /streams/{id}/records (ingest) Every validation run creates a record; full history
Mutable POST /streams/{id}/records/upsert One record per stable externalId key — latest state wins (key shape varies by validation type; see Records output)

Records use a stable externalId (no jobRunId in the key), for example:

dq_{ruleSetId}_{instanceSpace}_{instanceExternalId}

jobRunId remains a queryable property on the record.

Immutable streams — multi-run semantics

After the stable-key migration, a single externalId can match multiple records on an immutable stream (one per run). Consumers must filter or sort by:

  • jobRunId property
  • lastUpdatedTime / createdTime

Do not assume externalId uniqueness per run on immutable streams.

Mutable streams — current state

Re-validating the same focus upserts its record for that stable-key dimension (instance+ruleset, edge relation, etc.) — not strictly “one row per instance + ruleset” for every validation type.

Requires streamsAcl:READ on the validation function principal so runtime can detect stream type. Without it, the code fails safe to immutable ingest — on a mutable stream duplicate ingest may be rejected (409/422).

Choosing a stream for your estate

Need Recommendation
Compliance / audit trail BasicArchive
Operations dashboard (latest state) BasicLiveData after consumers support upsert semantics
High ruleset churn Mutable reduces record volume (per stable-key dimension)
Streamlit validation dashboard Stay on immutable (BasicArchive) — the dashboard queries by lastUpdatedTime windows, not mutable upsert keys

Migration notes

  • Mixed externalId schemes: older run-scoped ids (dq_{ruleSetId}_{externalId}_{jobRunId}) may coexist with stable ids until retention removes them.
  • Legacy mutable / immutable strings in settings are mapped to CDF template names when creating a new stream via ensure_records_infrastructure() (not when loading DMS settings at runtime).
  • RAW on mutable streams: stable keys omit db/table identity (dq_{ruleSetId}_{rowKey}). Sharing one mutable stream across RAW tables with overlapping row keys risks upsert collisions — prefer BasicArchive for RAW or isolate streams per table.

Full schema and query examples: Records output.