Evaluation
How every model proves itself on your history and on live traffic before it decides anything, and how it's watched afterward.
No model decides anything on a promise. Every version, whether in context, adapted, or fine-tuned, is measured on your data against the model you run today, in three stages.
Backtest
The model predicts at past dates whose outcomes are known, on your data and your labels, and it’s scored on what happened next. You review the results before production is on the table.
A backtest is out of time: predictions are made at past dates and scored on what happened after them, never on data the model could have seen. The same rules that keep a live prediction honest apply: each prediction sees only what was known on its date, and learns only from outcomes whose window had closed by then. See Time and leakage.
Each model is compared with a simple baseline, such as each record’s last known outcome, and with the model you run today, and it has to beat both to matter. Results can be broken down by product, region, and portfolio slice, not only reported overall, and measured with the metric that fits the task:
| Task | Metrics |
|---|---|
| Yes-or-no outcomes: default, churn, fraud, conversion | ROC-AUC; PR-AUC when the outcome is rare; KS for credit |
| Ranking for a team with limited capacity | Lift and precision at the top of the ranking |
| Numeric outcomes | Error against the realized value |
Shadow
The model runs beside your current one on live traffic. Same inputs, two answers, and no effect on the decision until the comparison is settled. A shadow run shows what a backtest can’t: how the model behaves on today’s data, through your real integration, at your real volumes.
Promote
Reassign the production alias to the version that earned it. Every version is immutable, and rolling back is reassigning the alias again. See Model versioning.
Governed releases
Every version Avra serves is a governed release: authorized by Avra’s research team and approved by someone who didn’t train it. Versions are immutable, and aliases move between them.
After promotion
Keep measuring after promotion. Rescore the whole book on a schedule with batch inference, compare the score distribution with the previous run’s, and check realized outcomes against your bands as they come in, so drift shows up before it reaches decisions.
Results are shared with you, on your data. Avra doesn’t publish benchmark claims.