---
title: Data ingestion
description: How the data you send becomes validated, versioned inputs for Avra's models, in Avra's cloud or in your own environment.
sidebar:
  label: Overview
---

Data ingestion is how your data reaches Avra's models. Every table you send goes through Avra's ingestion pipeline, which checks it against the data contract agreed with your Avra team and publishes a new version only when it passes. Labels take the same path, and nothing reaches a model any other way.

The records you send for scoring are the exception. A request to the API, or the file you submit to a batch, goes straight to the model.

## What a published version guarantees

Every version the pipeline publishes has passed the blocking checks in your contract: declared columns present with their types, keys that resolve to the records they point at, timestamps that agree with each other, and whatever else the contract requires. Readers only ever see a whole version, never one still being written.

Every row also keeps when it was true and when your systems recorded it, so your data can be read as it stood on any past date. That's what lets a backtest see only what was known at the time. See [Time and history](/pt-BR/data-ingestion/time).

## How it works

1. **Choose where it runs**

    In Avra's cloud, inside your isolated workspace, or in your own environment, where the pipeline runs as part of the [data layer](/pt-BR/deployment/data-layer) next to your data. There, the Avra agent connects out to Avra, so your network needs no inbound port.

2. **Bring the data**

    Point the pipeline at the bucket where your files already land, or upload them in the dashboard and your Avra team takes them from there. In Avra's cloud, Avra can list the tables and columns a location holds and profile them, so the contract is written from what's actually there. See [Preparing your data](/pt-BR/data-ingestion/preparing-data).

3. **Agree on the contract**

    The pipeline definition declares each table's schema, keys, and time columns, how the tables link, the checks every delivery must pass, and where the output goes. See [Data contracts and checks](/pt-BR/data-ingestion/data-contracts).

4. **Run it**

    Each run takes new data through five stages and publishes a new version only if no blocking check fails.

## The five stages

<svg class="fig" viewBox="0 0 640 222" role="img" aria-label="Your files, read from a bucket, flow through ingestion, normalization, feature engineering, the quality gate, and publication. From the quality gate, a failed check rejects the run and keeps the previous version. Publication writes version 13 after version 12.">
  <defs>
    <marker id="fig-pipeline-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0,0 L10,5 L0,10 z" class="arrow" />
    </marker>
    <marker id="fig-pipeline-arrow-accent" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0,0 L10,5 L0,10 z" class="arrow-accent" />
    </marker>
  </defs>
  <rect class="card" x="16" y="10" width="292" height="32" rx="6" />
  <text class="small" x="30" y="31">Your files, in a bucket the pipeline reads</text>
  <line class="edge" x1="72" y1="42" x2="72" y2="66" marker-end="url(#fig-pipeline-arrow)" />
  <rect class="card" x="16" y="70" width="112" height="42" rx="6" />
  <text class="small mid" x="72" y="95">Ingestion</text>
  <line class="edge" x1="128" y1="91" x2="138" y2="91" marker-end="url(#fig-pipeline-arrow)" />
  <rect class="card" x="140" y="70" width="112" height="42" rx="6" />
  <text class="small mid" x="196" y="95">Normalization</text>
  <line class="edge" x1="252" y1="91" x2="262" y2="91" marker-end="url(#fig-pipeline-arrow)" />
  <rect class="card" x="264" y="70" width="112" height="42" rx="6" />
  <text class="small mid" x="320" y="88">Feature</text>
  <text class="small mid" x="320" y="103">engineering</text>
  <line class="edge" x1="376" y1="91" x2="386" y2="91" marker-end="url(#fig-pipeline-arrow)" />
  <rect class="solid" x="388" y="70" width="112" height="42" rx="6" />
  <text class="small mid on-solid title" x="444" y="95">Quality gate</text>
  <line class="edge-accent" x1="500" y1="91" x2="510" y2="91" marker-end="url(#fig-pipeline-arrow-accent)" />
  <rect class="soft" x="512" y="70" width="112" height="42" rx="6" />
  <text class="small mid" x="568" y="95">Publication</text>
  <line class="edge dash" x1="444" y1="112" x2="444" y2="150" marker-end="url(#fig-pipeline-arrow)" />
  <rect class="node dash" x="316" y="154" width="200" height="46" rx="6" />
  <text class="small mid" x="416" y="173">A check fails: run rejected</text>
  <text class="small mid label" x="416" y="190">previous version stays current</text>
  <line class="edge-accent" x1="568" y1="112" x2="568" y2="150" marker-end="url(#fig-pipeline-arrow-accent)" />
  <rect class="node faint" x="526" y="154" width="40" height="26" rx="5" />
  <text class="mono small mid faint" x="546" y="171">v12</text>
  <rect class="solid" x="574" y="154" width="46" height="26" rx="5" />
  <text class="mono small mid on-solid" x="597" y="171">v13</text>
  <text class="small label end" x="622" y="214">new, complete version</text>
</svg>

Every run takes new data through five stages. The quality gate decides: if every blocking check passes, publication writes a new, complete version; if one fails, the run is rejected and the previous version stays current.

| Stage | What it does |
|-------|--------------|
| **Ingestion** | Reads each declared source and lands what's new, tagging every row with the run that brought it |
| **Normalization** | Casts each source to its declared schema and drops repeated deliveries of the same row. No business rules |
| **Feature engineering** | Organizes the sources into records, the keys that identify them, and the relationships between them, and computes any features the contract declares |
| **Quality gate** | Runs the contract's checks, and rejects the run if a blocking check fails |
| **Publication** | Writes a new version of the output. Readers only ever see a complete version |

## Runs

| Run | What it covers |
|-----|----------------|
| **One-off** | Everything that arrived since the last run |
| **Backfill** | A window of history, read without moving each source's regular checkpoint. A backfill doesn't change the baselines the quality gate compares new deliveries with |
| **Continuous** | New data as it arrives, one increment at a time, until stopped |

Each source keeps a checkpoint, so a run reads only what arrived since the last one, and Avra can reset a source's checkpoint to read it again from the start. A run can be cancelled while it's in progress. When a blocking check fails, a one-off run ends rejected, while a continuous run records the rejected increment and moves on to the next one.

## Isolation

The pipeline keeps your data and everything derived from it inside your workspace, in the environment you chose. In your environment, its tables stay in your bucket and its run state in your database. What returns to Avra is execution metadata, such as run status, and operational telemetry labeled with your cluster name, which covers logs from Avra's pods, the metrics and traces of Avra's components, and resource metrics for the collector's node and the pods on it. See [Install the data layer](/pt-BR/deployment/data-layer#security).

Processing your data doesn't make it part of any shared model's training. See [Data privacy & compliance](/pt-BR/security/data-privacy-and-compliance).

**[Preparing your data](/pt-BR/data-ingestion/preparing-data)**

What to send, in which formats, and how to deliver it

**[Time and history](/pt-BR/data-ingestion/time)**

Event time, snapshot time, and reading your data as of a date

**[Data contracts and checks](/pt-BR/data-ingestion/data-contracts)**

What the quality gate checks, and what happens when a check fails
