Skip to content

Evaluating and Revalidating Models

Once a predictor is trained, there are two ways to check how well it's actually performing:

  • evaluate() — score the predictor against labeled data you supply (a holdout set, new labeled data, a re-upload of your original test split). This is the primary way to re-test a model.
  • coverage() — check "answer only when you know" selective-prediction metrics. Depending on training state, this may return the metrics captured during training rather than a fresh live re-run — see the caveat below before relying on it as a re-test.

Re-testing with evaluate()

evaluate() runs your records through the trained model and scores the predictions against your ground-truth labels — all on the server, using the same metric code the predictor itself reports from. You don't reimplement scoring client-side, so the positive class, thresholds, and metric conventions always match what the model reports elsewhere (training metrics, predict() results, etc).

from featrixsphere.api import FeatrixSphere

featrix = FeatrixSphere()
fm = featrix.foundational_model("your-session-id")
predictor = fm.list_predictors()[0]

# From a labeled holdout DataFrame
metrics = predictor.evaluate(holdout_df, labels="churned")
print(metrics["accuracy"], metrics["auc"], metrics["f1"])

Or pass records and labels explicitly:

metrics = predictor.evaluate(
    records=[{"age": 35, "income": 50000, "city": "NYC"}, ...],
    labels=["yes", "no", ...],
)

What you get back:

Task type Metrics
Classification accuracy, f1, auc (plus f1_error / auc_error if the positive class can't be resolved)
Regression r2, rmse, mae

When to use this: any time you want to know "how does this model actually do on labeled data" — a re-uploaded original test split, a fresh batch of labeled records, or data collected since deployment. This is the call to reach for when you want a genuine, on-demand re-validation.

Selective-prediction coverage with coverage()

coverage() reports "answer only when you know" metrics — how much of your data the model can answer confidently at various operating points, and the accuracy/precision/recall lift you get by abstaining on the rest.

# All four strategy views at once
cov = predictor.coverage()

if not cov.get('intent_feasible', True):
    print(f"Warning: {cov['intent_feasibility_reason']}")
    # Don't auto-deploy without acknowledging this

for key, view in cov['strategies'].items():
    print(f"{view['label']}: coverage={view['coverage']:.0%}, AUC={view['covered_auc']:.3f}")

# A single strategy
pos = predictor.coverage(strategy="only_on_strong_positives")
print(f"Caught {pos['true_positives_caught']}/{pos['true_positives_total']} positives")

Strategies:

Strategy What it optimizes
everything Act on every row, no abstention (baseline)
only_when_sure Max AUC on the answered set; demurred rows feed a human-in-the-loop queue
only_on_strong_positives Max precision when predicting positive
only_on_strong_negatives Max NPV when predicting negative

Each view returns coverage, covered_auc, full_auc, auc_lift, confidence_threshold, n_covered, n_total, and label. Always check intent_feasible — if False, the framework couldn't honor the intent contract you trained with (e.g. a precision floor), and the returned operating point is a max-AUC fallback, not what you asked for.

Caveat: this may return cached training-time numbers, not a live re-run

Internally, coverage() recomputes metrics by pulling the model's own stored validation set and re-running predictions against it — a genuine re-validation, not a client-side estimate. However, if the predictor already has selective-prediction metrics captured from training, the endpoint returns those cached numbers by default rather than recomputing. There is currently no SDK parameter to force a fresh live re-run — coverage() will transparently use whichever result is available.

In practice: treat coverage() as "the selective-prediction picture for this model" rather than an on-demand re-test tool. If you need a guaranteed fresh score against specific data, use evaluate().

Which one should I use?

Goal Call
"Does this model still perform well on labeled data I have?" evaluate()
"What's my model's answer-only-when-confident tradeoff?" coverage()
"I need a guaranteed fresh recomputation, not cached numbers" evaluate()

Next Steps