Pular para o conteúdo
AvraAvra
Esc
↑↓navegar↵abrir⌘Jpré-visualizar
Nesta página

Evaluation

How every model proves itself on your history and on live traffic before it decides anything, and how it's watched afterward.

No model decides anything on a promise. Every version, whether in context, adapted, or fine-tuned, is measured on your data against the model you run today, in three stages.

1 · BacktestPredict at past dates,score on what followed2 · ShadowcurrentdecidesnewloggedSame live inputs, twoanswers, one decision3 · Promoteproductionv3v4Move the alias. Roll backby moving it again
The three stages every version goes through. A backtest scores predictions made at past dates, a shadow run compares answers on live traffic without affecting decisions, and promotion moves the production alias.

Backtest

The model predicts at past dates whose outcomes are known, on your data and your labels, and it’s scored on what happened next. You review the results before production is on the table.

A backtest is out of time: predictions are made at past dates and scored on what happened after them, never on data the model could have seen. The same rules that keep a live prediction honest apply: each prediction sees only what was known on its date, and learns only from outcomes whose window had closed by then. See Time and leakage.

Each model is compared with a simple baseline, such as each record’s last known outcome, and with the model you run today, and it has to beat both to matter. Results can be broken down by product, region, and portfolio slice, not only reported overall, and measured with the metric that fits the task:

Task Metrics
Yes-or-no outcomes: default, churn, fraud, conversion ROC-AUC; PR-AUC when the outcome is rare; KS for credit
Ranking for a team with limited capacity Lift and precision at the top of the ranking
Numeric outcomes Error against the realized value

Shadow

The model runs beside your current one on live traffic. Same inputs, two answers, and no effect on the decision until the comparison is settled. A shadow run shows what a backtest can’t: how the model behaves on today’s data, through your real integration, at your real volumes.

Promote

Reassign the production alias to the version that earned it. Every version is immutable, and rolling back is reassigning the alias again. See Model versioning.

Governed releases

Every version Avra serves is a governed release: authorized by Avra’s research team and approved by someone who didn’t train it. Versions are immutable, and aliases move between them.

After promotion

Keep measuring after promotion. Rescore the whole book on a schedule with batch inference, compare the score distribution with the previous run’s, and check realized outcomes against your bands as they come in, so drift shows up before it reaches decisions.

Results are shared with you, on your data. Avra doesn’t publish benchmark claims.

Esta página foi útil?