Skip to content
AvraAvra
Esc
↑↓navigate↵open⌘Jpreview
On this page

Data contracts and checks

How your pipeline definition declares what your data must look like, and how the quality gate enforces it before anything is published.

Your data contract is the pipeline definition you agree with your Avra team. It declares the sources the pipeline reads, with the schema, identity key, and time columns of each; the records they describe and how they link; the checks every delivery must pass; and where the output goes.

The definition is validated as a whole before anything runs. A missing field, an unknown option, or a misspelled name is reported up front, instead of surfacing halfway through a run. Access to each source can be tested on its own, by reading one row without landing anything.

The quality gate

Every run passes through the quality gate before anything is published. One check always runs: a declared column missing from a delivery, or a row that can’t be read with its declared types, blocks the run, and a column the contract doesn’t declare raises an alert.

The other checks are declared in the contract, each on a column, a table, or a record:

Check What it verifies
Not null A feature computed over a time window has no missing values
Unique Each key identifies one row
Observed after event No row was recorded before it happened: its event time isn’t later than its snapshot time
Relationship resolves Every key points at a record that exists
Matches format Values match a declared pattern
In range Values fall within declared bounds
Accepted values Values come from a declared set
Valid document CPF and CNPJ numbers, numeric or alphanumeric, have valid check digits
Row count A delivery’s volume stays in line with recent runs
Cardinality A column’s number of distinct values stays in line with recent runs
Drift A column’s distribution stays in line with recent runs

Alert or block

Each check can alert, block, or both, at thresholds set in the contract. An alert reports a violation and lets the delivery through, and a block rejects it. Unless the contract says otherwise, a key that doesn’t resolve or a row recorded before it happened blocks the run, and every other check only alerts.

Checks that compare a delivery with recent runs — row count, cardinality, and drift — need a baseline: the window of past runs to compare against, such as the last 90 days. Not null and unique can use one too, to flag a rate that’s unusual for your data on top of a fixed threshold.

When a check fails

When a blocking check fails, the run is rejected: nothing invalid is published, the previous version stays current, and the run’s report names the check that failed. A failed check is a verdict about the data. A check that couldn’t run is reported as an error instead, so a broken source isn’t mistaken for bad data.

A backfill never blocks on a comparison with recent runs. Its comparative checks only alert, and its measurements stay out of the baselines.

Changing the contract

A change in what you send — a new table, a new column, a new check — means a change to the contract, agreed with Avra. See Preparing your data for what each table declares, and Time and history for how its time columns are read.

Was this page helpful?