---
title: Time and history
description: How the ingestion pipeline keeps when each row was true and when it was recorded, so every prediction and backtest sees your data as it stood on its date.
---

Data rarely arrives in order. A payment made on Friday reaches your warehouse on Monday. An address is corrected weeks after it changed. A limit increase is approved with effect from a date that has already passed.

A system that keeps only the latest version of each row loses all of that. It can say what's true now, but not what was known on a given day, and a model trained on its latest values learns from information it wouldn't have had at the time. That model looks excellent in a backtest and disappoints in production.

The ingestion pipeline keeps two times for every row, so it can tell the two apart.

## Two times for every row

Every table you send declares two time columns.

**Event time** is when the row was true at the source: when the payment happened, when the address took effect, when the limit changed. You choose the column that carries it, such as `paid_at`, `effective_at`, or `occurred_at`.

**Snapshot time** is when your systems recorded the row: the capture of the source it came from. A payment made on Friday and loaded on Monday has a Friday event time and a Monday snapshot time. When your systems don't record it, use the date of the extract that delivered the row.

Data that tracks both is called bitemporal. Event time puts facts in the order they happened, and snapshot time says when each one became known. A row can't be recorded before it happened, so the contract can declare a check that no row's event time is later than its snapshot time, and that check blocks the run by default when one is.

## Every version is kept

Rows aren't overwritten. Each table also declares an identity key, the columns that identify one row, and the pipeline keeps a version of the row for every snapshot it appears in. When your systems send a row again with a later snapshot time, the new version joins the earlier ones instead of replacing them. Sending the same snapshot twice doesn't create a duplicate: the later delivery replaces the earlier copy.

History comes from these columns, not from when files happened to land, so the pipeline can reproduce your data as it stood at any past moment.

## Reading data as of a date

To read your data as of a date, the pipeline takes, for each row, the latest version recorded on or before that date. Anything recorded later — a late payment, a correction, a new row — is out of view.

That's what makes predictions and backtests honest. A prediction for March 10 sees what your systems had recorded by March 10, and a backtest at a past date sees exactly what a model running that day would have seen.

## An example

Take one account's credit limit. On January 10, the account opens with a limit of 5,000. On March 1, the limit is raised to 8,000, and your system records the change the same day. On March 20, an analyst finds that the raise had been approved with effect from February 15, and corrects the record.

The pipeline holds three versions of the account's row:

| Version | Limit | Event time | Snapshot time |
|---------|-------|------------|---------------|
| 1 | 5,000 | January 10 | January 10 |
| 2 | 8,000 | March 1 | March 1 |
| 3 | 8,000 | February 15 | March 20 |

<svg class="fig" viewBox="0 0 640 362" role="img" aria-label="A chart with event time on the horizontal axis and snapshot time on the vertical axis, from January to April. Version 1 sits at January 10 on both axes, version 2 at March 1 on both, and version 3 at February 15 event time and March 20 snapshot time. Below the diagonal, rows recorded before they happened are blocked by default. Horizontal lines mark reads as of February 20, March 10, and March 25, which see versions 1, 2, and 3.">
  <defs>
    <marker id="fig-bitemporal-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0,0 L10,5 L0,10 z" class="arrow" />
    </marker>
  </defs>
  <polygon class="head" points="80,310 600,50 600,310" />
  <text class="small label end" x="592" y="296">recorded before it happened: blocked by default</text>
  <line class="edge dash" x1="80" y1="310" x2="600" y2="50" />
  <text class="small label" x="508" y="80" transform="rotate(-26.57 508 80)">recorded as it happened</text>
  <line class="edge" x1="80" y1="310" x2="612" y2="310" marker-end="url(#fig-bitemporal-arrow)" />
  <line class="edge" x1="80" y1="310" x2="80" y2="38" marker-end="url(#fig-bitemporal-arrow)" />
  <text class="small label mid" x="80" y="328">Jan</text>
  <text class="small label mid" x="235" y="328">Feb</text>
  <text class="small label mid" x="375" y="328">Mar</text>
  <text class="small label mid" x="530" y="328">Apr</text>
  <text class="small label end" x="72" y="236">Feb</text>
  <text class="small label end" x="72" y="166">Mar</text>
  <text class="small label end" x="72" y="89">Apr</text>
  <text class="small mid" x="340" y="352">event time: when it was true</text>
  <text class="small mid" x="26" y="176" transform="rotate(-90 26 176)">snapshot time: when it was recorded</text>
  <line class="edge-accent dash" x1="80" y1="185" x2="330" y2="185" />
  <text class="small accent" x="88" y="179">as of Feb 20: sees v1</text>
  <line class="edge-accent dash" x1="80" y1="140" x2="420" y2="140" />
  <text class="small accent" x="88" y="134">as of Mar 10: sees v2</text>
  <line class="edge-accent dash" x1="80" y1="102.5" x2="495" y2="102.5" />
  <text class="small accent" x="88" y="96.5">as of Mar 25: sees v3</text>
  <circle class="solid" cx="125" cy="287.5" r="6" />
  <text class="small mono" x="137" y="292">v1 · 5,000</text>
  <circle class="solid" cx="375" cy="162.5" r="6" />
  <text class="small mono" x="387" y="167">v2 · 8,000</text>
  <circle class="solid" cx="305" cy="115" r="6" />
  <text class="small mono" x="317" y="128">v3 · 8,000</text>
</svg>

The three versions on both time axes. A row can't be recorded before it happened, so every version sits on or above the diagonal. A read as of a date sees the versions on or below its line and takes the latest one.

Read as of February 20, the limit is 5,000. The raise had in fact taken effect by then, but on February 20 nothing in your systems said so. Read as of March 10, the limit is 8,000, effective March 1. Read as of March 25, it's 8,000, effective February 15: the corrected history.

A model scoring the account on February 20 sees the 5,000 limit, and so does a backtest at that date. The correction recorded on March 20 can't leak into a prediction made before it.

## Two questions about the past

Keeping both times gives two different questions about the past an answer. *What was known on that date?* reads by snapshot time. It's the question every prediction and backtest asks, so a model is only ever judged on information it could have had. *What's known now about that date?* reads by event time, with every later correction applied: the history as you understand it today.

## Late and corrected data

Late data and corrections need no special handling. Send a late row with its real event time and the snapshot it arrived in, or a corrected row with a new snapshot time, and the pipeline adds it as a new version. Reads as of that snapshot or later see it, and reads as of earlier dates don't.

## Backfills

To load history, such as a year of past snapshots, your Avra team runs a backfill: a run over a window of snapshot times. It reads under its own checkpoint, so the sources' regular checkpoints don't move, and its measurements stay out of the baselines the quality gate compares new deliveries with.

## Outcomes

Outcomes are data like any other, and each one carries the date it was measured. A prediction never learns from an outcome whose window hadn't closed by its date. See [How predictions work](/pt-BR/rfm/predictions).
