Skip to content

Example notebook

A full end-to-end notebook is available per environment (check your environment's configuration directory).

The notebook typically covers:

  1. Setup – Installation from PyPI, credentials (TOML or manual).
  2. Verify – Import and embedded function files (_get_embedded_function_files()).
  3. Client – Load client from TOML and set timeout; prepare function_secrets for deployment.
  4. Optional cleanup – Delete container, functions, triggers/workflows (for a clean redeploy).
  5. Run validation – run_validation() with TTL rules, DataModelConfig, instance_space, RecordsConfig, limit, print_output, post_to_records.
  6. Records API – List streams; filter records (e.g. failed validations, Violation/Warning in last hour).
  7. Deploy infrastructure – deploy_validation_infrastructure(settings_path, views_dir, function_secrets).
  8. Deploy validation pipeline – deploy_validation_pipeline(client, settings_path, view_external_id, wait=True) and display orchestration ID, partitions, sync trigger, monitor schedule.

Minimal script example

from cognite_data_quality import load_cognite_client_from_toml, run_validation, DataModelConfig, RecordsConfig

client = load_cognite_client_from_toml("config.toml")

result = run_validation(
    client=client,
    rules_path="shacl_rules/my_view_shacl.ttl",
    rules_format="ttl",
    datamodel=DataModelConfig(space="my_space", external_id="MyDataModel", version="v1"),
    instance_space="my_instances",
    records_config=RecordsConfig(
        stream_id="dq_validation_stream",
        rule_set_id="MyViewSHACLv1",
        rule_set_version="1.0",
    ),
    limit=10,
    post_to_records=False,
)
print(result.conforms, result.instance_count, len(result.violations))

For YAML-based config (datamodel and instance space from the file):

result = run_validation(
    client=client,
    rules_path="views/my_view.yaml",
    rules_format="yaml",
)