Data ingestion
How the data you send becomes validated, versioned inputs for Avra's models, in Avra's cloud or in your own environment.
Data ingestion is how your data reaches Avra’s models. Every table you send goes through Avra’s ingestion pipeline, which checks it against the data contract agreed with your Avra team and publishes a new version only when it passes. Labels take the same path, and nothing reaches a model any other way.
The records you send for scoring are the exception. A request to the API, or the file you submit to a batch, goes straight to the model.
What a published version guarantees
Every version the pipeline publishes has passed the blocking checks in your contract: declared columns present with their types, keys that resolve to the records they point at, timestamps that agree with each other, and whatever else the contract requires. Readers only ever see a whole version, never one still being written.
Every row also keeps when it was true and when your systems recorded it, so your data can be read as it stood on any past date. That’s what lets a backtest see only what was known at the time. See Time and history.
How it works
Choose where it runs
In Avra’s cloud, inside your isolated workspace, or in your own environment, where the pipeline runs as part of the data layer next to your data. There, the Avra agent connects out to Avra, so your network needs no inbound port.
Bring the data
Point the pipeline at the bucket where your files already land, or upload them in the dashboard and your Avra team takes them from there. In Avra’s cloud, Avra can list the tables and columns a location holds and profile them, so the contract is written from what’s actually there. See Preparing your data.
Agree on the contract
The pipeline definition declares each table’s schema, keys, and time columns, how the tables link, the checks every delivery must pass, and where the output goes. See Data contracts and checks.
Run it
Each run takes new data through five stages and publishes a new version only if no blocking check fails.
The five stages
| Stage | What it does |
|---|---|
| Ingestion | Reads each declared source and lands what’s new, tagging every row with the run that brought it |
| Normalization | Casts each source to its declared schema and drops repeated deliveries of the same row. No business rules |
| Feature engineering | Organizes the sources into records, the keys that identify them, and the relationships between them, and computes any features the contract declares |
| Quality gate | Runs the contract’s checks, and rejects the run if a blocking check fails |
| Publication | Writes a new version of the output. Readers only ever see a complete version |
Runs
| Run | What it covers |
|---|---|
| One-off | Everything that arrived since the last run |
| Backfill | A window of history, read without moving each source’s regular checkpoint. A backfill doesn’t change the baselines the quality gate compares new deliveries with |
| Continuous | New data as it arrives, one increment at a time, until stopped |
Each source keeps a checkpoint, so a run reads only what arrived since the last one, and Avra can reset a source’s checkpoint to read it again from the start. A run can be cancelled while it’s in progress. When a blocking check fails, a one-off run ends rejected, while a continuous run records the rejected increment and moves on to the next one.
Isolation
The pipeline keeps your data and everything derived from it inside your workspace, in the environment you chose. In your environment, its tables stay in your bucket and its run state in your database. What returns to Avra is execution metadata, such as run status, and operational telemetry labeled with your cluster name, which covers logs from Avra’s pods, the metrics and traces of Avra’s components, and resource metrics for the collector’s node and the pods on it. See Install the data layer.
Processing your data doesn’t make it part of any shared model’s training. See Data privacy & compliance.