Skip to content
AvraAvra
Esc
↑↓navigate↵open⌘Jpreview
On this page

Preparing your data

What to send, in which formats, and how to deliver it, so the ingestion pipeline can validate it and models learn from it without seeing the future.

Your models learn from the tables you already keep: customers, accounts, invoices, payments, collections, support tickets, and the keys that link them. Every delivery goes through the ingestion pipeline, which checks it against your data contract before any model sees it.

Send the tables as they are. There’s no need to flatten them into one wide row or build features first.

What to send

Tables and keys

The records you want to score and everything related to them, linked by consistent identifiers: legal documents or your own keys.

Time

For every row, when it was true at the source and when your systems recorded it. History comes from these columns.

Outcomes

The labels that define your target: defaults, conversions, renewals, churn, confirmed fraud, each with its date.

Enough history

Enough past outcomes to evaluate a model on periods it didn’t see.

Send the tables that say how your records connect, not only what they are: account hierarchies, subsidiaries, and partnerships; payments, invoices, and transfers, with the counterparties on each side; and sequences of events such as customer journeys, interactions, and lifecycle events. Alongside them go the attributes your operational systems keep, from profile and account details to purchases, usage, and billing events, and outcomes such as defaults, renewals, upgrades, and cancellations.

What each table declares

The contract declares four things for every table you send. Its schema lists the columns and their types: strings, booleans, integers, floating-point numbers, decimals with a declared precision, dates, timestamps, and binary values. Its identity key is the column or columns that identify one row, such as payment_id in a payments table or account_id in an accounts table. Its two time columns say when each row was true and when your systems recorded it. And its links to other tables are the columns that carry their keys, such as the customer_id in an accounts table.

Identifiers can be legal documents, such as CPF or CNPJ, or your own keys, as long as they’re consistent across tables.

Here’s what a delivery for a lending portfolio might look like. Each table has its identity key, the keys that link it to other tables, and its two time columns, and the outcome, a default, arrives as a table of its own:

File formats

Send files as Parquet, CSV, or JSON. Parquet fits best: it’s typed and compressed, so values keep their precision and large deliveries move faster. CSV files need a header row, and the pipeline matches their columns to the schema by position, not by name, so they must follow the schema’s column order.

How to deliver it

Keep writing files to an Amazon S3 or Google Cloud Storage bucket you already use, and the pipeline reads what’s new from there, in Avra’s cloud or in your environment.

You can also upload files in the dashboard, under File Transfer → Sent to Avra, once Avra enables it for your organization. Uploads land in Avra’s cloud, and your Avra team takes them from there, so when the pipeline runs in your environment, deliver to your bucket instead. Files Avra prepares for you appear under File Transfer → Received from Avra.

Discovery and profiling

In Avra’s cloud, Avra can describe your data before the contract is written. It lists the tables and columns a location holds, then profiles them — null rates, distinct counts, frequent values — so the contract starts from what’s actually there. In either environment, access to each source can be tested by reading one row without landing anything.

Time and leakage

Leakage happens when information from after a decision sneaks into the data used to predict it. A model that sees the future looks excellent in a backtest and fails in production.

Every table carries two times to prevent it: when each row was true, in a column such as created_at, effective_at, or occurred_at, and when your systems recorded it. A payment recorded three days late was unknown for those three days, and the pipeline keeps that difference, so predictions and backtests see only what was known on their date. See Time and history.

Working with your Avra team

Your Avra team designs the pipeline with you: which sources it reads, how they link, and which checks each delivery must pass. If a delivery breaks the contract, the run is rejected with the reason, and the last valid version stays in use. A change in what you send means a change to the contract, agreed with Avra.

Contact your Avra representative to start the data discovery process.

Was this page helpful?