Skip to content

Runtime options (Python, Notebook, Function)

What this is

This package runs in three common contexts: Python scripts/REPL, notebooks, and Cognite Functions. All local contexts use the same API; the difference is execution model and dependency management.

When to use each runtime

  • Python script / REPL for repeatable local or CI execution.
  • Notebook for iterative investigation and rule tuning.
  • Cognite Function for scheduled or event-driven production execution.

User mental model

  • Keep validation API surface consistent across runtimes.
  • Switch runtime based on orchestration and operational requirements, not rule syntax.

Minimal happy path

Usage contexts

Context Install Entry points Typical use
Python script / REPL pip install cognite-data-quality in a venv run_validation, deploy_validation_infrastructure, deploy_validation_pipeline, call_* invoke helpers Headless runs, CI, scheduled jobs, invoking deployed functions
Notebook %pip install cognite-data-quality or venv Same as above Interactive exploration, inspecting result.violations, iterating over records
Cognite Function requirements.txt or [requirements] tags Handler imports from package Event-driven or scheduled validation in CDF

Minimal examples

Python script

from cognite_data_quality import load_cognite_client_from_toml, run_validation

client = load_cognite_client_from_toml("config.toml")
result = run_validation(
    client=client,
    rules_path="shacl_rules/my_view_shacl.ttl",
    rules_format="ttl",
    instance_space="my_instances",
)
print(result.conforms, len(result.violations))

Notebook

%pip install cognite-data-quality

from cognite_data_quality import load_cognite_client_from_toml, run_validation

client = load_cognite_client_from_toml("config.toml")
result = run_validation(
    client=client,
    rules_path="shacl_rules/my_view_shacl.ttl",
    rules_format="ttl",
    instance_space="my_instances",
)
result.violations[:5]

Invoking deployed functions

For scheduled global uniqueness, the dq-{view}-uniqueness workflow invokes validation_type: shacl, which posts RuleEngineResult records. See Uniqueness.

from cognite_data_quality import load_cognite_client_from_toml, call_validate_instances_shacl

client = load_cognite_client_from_toml("config.toml")
data = {
    "instances": {"items": {"n": [...]}},
    "shacl_rules_file_external_id": "my_shacl_rules",
    "datamodel_space": "space", "datamodel_external_id": "model", "datamodel_version": "v1",
    "records_config": {"stream_id": "dq_stream", "rule_set_id": "MyRuleSet", "rule_set_version": "1.0"},
}
result = call_validate_instances_shacl(client, data)

Cognite Function

def handle(client, data):
    """
    [requirements]
    cognite-data-quality
    [/requirements]
    """
    from cognite_data_quality import run_validation

    return run_validation(
        client=client,
        rules_path="rules/equipment_shacl_rules.ttl",
        rules_format="ttl",
        instance_space="my_instances",
    ).model_dump()

Runtime behavior

Options per entry point

  • run_validation: rules_path, rules_format, datamodel, instance_space, verbose, records_config, post_to_records, namespace_base
  • load_cognite_client_from_toml: config_path (default config.toml)
  • deploy_validation_infrastructure: settings_path, views_dir, function_secrets, dry_run, force
  • deploy_validation_pipeline: settings_path, view_external_id, wait
  • Invoke helpers (call_validate_instances_shacl, etc.): client, data (payload dict), wait, external_id

Best practices

  • Keep local and deployed payload shapes aligned by reusing YAML config and shared rule artifacts.
  • Use notebooks for exploration, then codify stable flows in scripts/pipelines.
  • Use function runtime for any scheduled production workload.

Troubleshooting

  • Works locally but fails in function: verify secrets, runtime dependencies, and payload fields.
  • Different results between notebook and CI: verify same rule files, datamodel, and instance scope.
  • Invocation issues: confirm infrastructure was deployed before using invoke helpers.

Previous section

Next section