Skip to content
AvraAvra
Esc
↑↓navigate↵open⌘Jpreview
On this page

Data ingestion

How the data you send becomes validated, versioned inputs for Avra's models, in Avra's cloud or in your own environment.

Data ingestion is how your data reaches Avra’s models. Every table you send goes through Avra’s ingestion pipeline, which checks it against the data contract agreed with your Avra team and publishes a new version only when it passes. Labels take the same path, and nothing reaches a model any other way.

The records you send for scoring are the exception. A request to the API, or the file you submit to a batch, goes straight to the model.

What a published version guarantees

Every version the pipeline publishes has passed the blocking checks in your contract: declared columns present with their types, keys that resolve to the records they point at, timestamps that agree with each other, and whatever else the contract requires. Readers only ever see a whole version, never one still being written.

Every row also keeps when it was true and when your systems recorded it, so your data can be read as it stood on any past date. That’s what lets a backtest see only what was known at the time. See Time and history.

How it works

Choose where it runs

In Avra’s cloud, inside your isolated workspace, or in your own environment, where the pipeline runs as part of the data layer next to your data. There, the Avra agent connects out to Avra, so your network needs no inbound port.

Bring the data

Point the pipeline at the bucket where your files already land, or upload them in the dashboard and your Avra team takes them from there. In Avra’s cloud, Avra can list the tables and columns a location holds and profile them, so the contract is written from what’s actually there. See Preparing your data.

Agree on the contract

The pipeline definition declares each table’s schema, keys, and time columns, how the tables link, the checks every delivery must pass, and where the output goes. See Data contracts and checks.

Run it

Each run takes new data through five stages and publishes a new version only if no blocking check fails.

The five stages

Your files, in a bucket the pipeline readsIngestionNormalizationFeatureengineeringQuality gatePublicationA check fails: run rejectedprevious version stays currentv12v13new, complete version
Every run takes new data through five stages. The quality gate decides: if every blocking check passes, publication writes a new, complete version; if one fails, the run is rejected and the previous version stays current.
Stage What it does
Ingestion Reads each declared source and lands what’s new, tagging every row with the run that brought it
Normalization Casts each source to its declared schema and drops repeated deliveries of the same row. No business rules
Feature engineering Organizes the sources into records, the keys that identify them, and the relationships between them, and computes any features the contract declares
Quality gate Runs the contract’s checks, and rejects the run if a blocking check fails
Publication Writes a new version of the output. Readers only ever see a complete version

Runs

Run What it covers
One-off Everything that arrived since the last run
Backfill A window of history, read without moving each source’s regular checkpoint. A backfill doesn’t change the baselines the quality gate compares new deliveries with
Continuous New data as it arrives, one increment at a time, until stopped

Each source keeps a checkpoint, so a run reads only what arrived since the last one, and Avra can reset a source’s checkpoint to read it again from the start. A run can be cancelled while it’s in progress. When a blocking check fails, a one-off run ends rejected, while a continuous run records the rejected increment and moves on to the next one.

Isolation

The pipeline keeps your data and everything derived from it inside your workspace, in the environment you chose. In your environment, its tables stay in your bucket and its run state in your database. What returns to Avra is execution metadata, such as run status, and operational telemetry labeled with your cluster name, which covers logs from Avra’s pods, the metrics and traces of Avra’s components, and resource metrics for the collector’s node and the pods on it. See Install the data layer.

Processing your data doesn’t make it part of any shared model’s training. See Data privacy & compliance.

Was this page helpful?