---
title: Evaluation
description: How every model proves itself on your history and on live traffic before it decides anything, and how it's watched afterward.
---

No model decides anything on a promise. Every version, whether in context, adapted, or fine-tuned, is measured on your data against the model you run today, in three stages.

<svg class="fig" viewBox="0 0 640 228" role="img" aria-label="Three panels. Backtest: predictions at past dates, each scored on what happened in the window after it. Shadow: the current model decides while the new version's answers are only logged. Promote: the production alias moves from version 3 to version 4, and moving it back rolls back.">
  <defs>
    <marker id="fig-eval-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0,0 L10,5 L0,10 z" class="arrow" />
    </marker>
    <marker id="fig-eval-arrow-accent" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0,0 L10,5 L0,10 z" class="arrow-accent" />
    </marker>
  </defs>
  <rect class="card" x="16" y="16" width="196" height="196" rx="10" />
  <text class="title" x="32" y="42">1 · Backtest</text>
  <circle class="solid" cx="40" cy="74" r="4" />
  <rect class="soft" x="40" y="68" width="62" height="12" rx="3" />
  <circle class="solid" cx="76" cy="98" r="4" />
  <rect class="soft" x="76" y="92" width="62" height="12" rx="3" />
  <circle class="solid" cx="112" cy="122" r="4" />
  <rect class="soft" x="112" y="116" width="62" height="12" rx="3" />
  <line class="edge" x1="30" y1="146" x2="196" y2="146" marker-end="url(#fig-eval-arrow)" />
  <text class="small label" x="32" y="172">Predict at past dates,</text>
  <text class="small label" x="32" y="188">score on what followed</text>
  <rect class="card" x="222" y="16" width="196" height="196" rx="10" />
  <text class="title" x="238" y="42">2 · Shadow</text>
  <rect class="node" x="238" y="62" width="76" height="30" rx="6" />
  <text class="small mid" x="276" y="81">current</text>
  <line class="edge-accent" x1="314" y1="77" x2="334" y2="77" marker-end="url(#fig-eval-arrow-accent)" />
  <rect class="solid" x="338" y="62" width="66" height="30" rx="6" />
  <text class="small mid on-solid" x="371" y="81">decides</text>
  <rect class="soft" x="238" y="106" width="76" height="30" rx="6" />
  <text class="small mid" x="276" y="125">new</text>
  <line class="edge dash" x1="314" y1="121" x2="334" y2="121" marker-end="url(#fig-eval-arrow)" />
  <rect class="node dash" x="338" y="106" width="66" height="30" rx="6" />
  <text class="small mid label" x="371" y="125">logged</text>
  <text class="small label" x="238" y="172">Same live inputs, two</text>
  <text class="small label" x="238" y="188">answers, one decision</text>
  <rect class="card" x="428" y="16" width="196" height="196" rx="10" />
  <text class="title" x="444" y="42">3 · Promote</text>
  <rect class="soft" x="488" y="56" width="80" height="24" rx="12" />
  <text class="mono small mid" x="528" y="72">production</text>
  <path class="edge dash faint" d="M508,80 L478,108" />
  <path class="edge-accent" d="M548,80 L572,106" marker-end="url(#fig-eval-arrow-accent)" />
  <circle class="node faint" cx="474" cy="124" r="16" />
  <text class="mono mid faint" x="474" y="128">v3</text>
  <circle class="solid" cx="580" cy="124" r="16" />
  <text class="mono mid on-solid" x="580" y="128">v4</text>
  <text class="small label" x="444" y="172">Move the alias. Roll back</text>
  <text class="small label" x="444" y="188">by moving it again</text>
</svg>

The three stages every version goes through. A backtest scores predictions made at past dates, a shadow run compares answers on live traffic without affecting decisions, and promotion moves the production alias.

## Backtest

The model predicts at past dates whose outcomes are known, on your data and your labels, and it's scored on what happened next. You review the results before production is on the table.

A backtest is out of time: predictions are made at past dates and scored on what happened after them, never on data the model could have seen. The same rules that keep a live prediction honest apply: each prediction sees only what was known on its date, and learns only from outcomes whose window had closed by then. See [Time and leakage](/rfm/predictions#time-and-leakage).

Each model is compared with a simple baseline, such as each record's last known outcome, and with the model you run today, and it has to beat both to matter. Results can be broken down by product, region, and portfolio slice, not only reported overall, and measured with the metric that fits the task:

| Task | Metrics |
|------|---------|
| Yes-or-no outcomes: default, churn, fraud, conversion | ROC-AUC; PR-AUC when the outcome is rare; KS for credit |
| Ranking for a team with limited capacity | Lift and precision at the top of the ranking |
| Numeric outcomes | Error against the realized value |

## Shadow

The model runs beside your current one on live traffic. Same inputs, two answers, and no effect on the decision until the comparison is settled. A shadow run shows what a backtest can't: how the model behaves on today's data, through your real integration, at your real volumes.

## Promote

Reassign the `production` alias to the version that earned it. Every version is immutable, and rolling back is reassigning the alias again. See [Model versioning](/core/model-versioning).

## Governed releases

Every version Avra serves is a governed release: authorized by Avra's research team and approved by someone who didn't train it. Versions are immutable, and aliases move between them.

## After promotion

Keep measuring after promotion. Rescore the whole book on a schedule with [batch inference](/core/batch-inference), compare the score distribution with the previous run's, and check realized outcomes against your bands as they come in, so drift shows up before it reaches decisions.

Results are shared with you, on your data. Avra doesn't publish benchmark claims.

**[Model versioning](/core/model-versioning)**

Versions, aliases, promotion, and rollback

**[Risk bands](/use-cases/credit-intelligence/homogeneous-groups)**

Turn scores into bands whose outcomes stay distinct
