Skip to content
AvraAvra
Esc
↑↓navigate↵open⌘Jpreview
On this page

Time and history

How the ingestion pipeline keeps when each row was true and when it was recorded, so every prediction and backtest sees your data as it stood on its date.

Data rarely arrives in order. A payment made on Friday reaches your warehouse on Monday. An address is corrected weeks after it changed. A limit increase is approved with effect from a date that has already passed.

A system that keeps only the latest version of each row loses all of that. It can say what’s true now, but not what was known on a given day, and a model trained on its latest values learns from information it wouldn’t have had at the time. That model looks excellent in a backtest and disappoints in production.

The ingestion pipeline keeps two times for every row, so it can tell the two apart.

Two times for every row

Every table you send declares two time columns.

Event time is when the row was true at the source: when the payment happened, when the address took effect, when the limit changed. You choose the column that carries it, such as paid_at, effective_at, or occurred_at.

Snapshot time is when your systems recorded the row: the capture of the source it came from. A payment made on Friday and loaded on Monday has a Friday event time and a Monday snapshot time. When your systems don’t record it, use the date of the extract that delivered the row.

Data that tracks both is called bitemporal. Event time puts facts in the order they happened, and snapshot time says when each one became known. A row can’t be recorded before it happened, so the contract can declare a check that no row’s event time is later than its snapshot time, and that check blocks the run by default when one is.

Every version is kept

Rows aren’t overwritten. Each table also declares an identity key, the columns that identify one row, and the pipeline keeps a version of the row for every snapshot it appears in. When your systems send a row again with a later snapshot time, the new version joins the earlier ones instead of replacing them. Sending the same snapshot twice doesn’t create a duplicate: the later delivery replaces the earlier copy.

History comes from these columns, not from when files happened to land, so the pipeline can reproduce your data as it stood at any past moment.

Reading data as of a date

To read your data as of a date, the pipeline takes, for each row, the latest version recorded on or before that date. Anything recorded later — a late payment, a correction, a new row — is out of view.

That’s what makes predictions and backtests honest. A prediction for March 10 sees what your systems had recorded by March 10, and a backtest at a past date sees exactly what a model running that day would have seen.

An example

Take one account’s credit limit. On January 10, the account opens with a limit of 5,000. On March 1, the limit is raised to 8,000, and your system records the change the same day. On March 20, an analyst finds that the raise had been approved with effect from February 15, and corrects the record.

The pipeline holds three versions of the account’s row:

Version Limit Event time Snapshot time
1 5,000 January 10 January 10
2 8,000 March 1 March 1
3 8,000 February 15 March 20
recorded before it happened: blocked by defaultrecorded as it happenedJanFebMarAprFebMarAprevent time: when it was truesnapshot time: when it was recordedas of Feb 20: sees v1as of Mar 10: sees v2as of Mar 25: sees v3v1 · 5,000v2 · 8,000v3 · 8,000
The three versions on both time axes. A row can’t be recorded before it happened, so every version sits on or above the diagonal. A read as of a date sees the versions on or below its line and takes the latest one.

Read as of February 20, the limit is 5,000. The raise had in fact taken effect by then, but on February 20 nothing in your systems said so. Read as of March 10, the limit is 8,000, effective March 1. Read as of March 25, it’s 8,000, effective February 15: the corrected history.

A model scoring the account on February 20 sees the 5,000 limit, and so does a backtest at that date. The correction recorded on March 20 can’t leak into a prediction made before it.

Two questions about the past

Keeping both times gives two different questions about the past an answer. What was known on that date? reads by snapshot time. It’s the question every prediction and backtest asks, so a model is only ever judged on information it could have had. What’s known now about that date? reads by event time, with every later correction applied: the history as you understand it today.

Late and corrected data

Late data and corrections need no special handling. Send a late row with its real event time and the snapshot it arrived in, or a corrected row with a new snapshot time, and the pipeline adds it as a new version. Reads as of that snapshot or later see it, and reads as of earlier dates don’t.

Backfills

To load history, such as a year of past snapshots, your Avra team runs a backfill: a run over a window of snapshot times. It reads under its own checkpoint, so the sources’ regular checkpoints don’t move, and its measurements stay out of the baselines the quality gate compares new deliveries with.

Outcomes

Outcomes are data like any other, and each one carries the date it was measured. A prediction never learns from an outcome whose window hadn’t closed by its date. See How predictions work.

Was this page helpful?