Model Cards, Publishing, and Monitoring¶

This guide covers the model lifecycle: understanding what your model learned (model cards), deploying to production (publishing), and keeping it healthy (monitoring).

Model Cards¶

A model card is a complete record of everything Featrix decided about your data and how the model was trained. It's essential for:

Auditing and compliance
Understanding model behavior
Debugging prediction issues
Reproducing results

See the Model Card Specification for the complete JSON schema and field definitions.

Accessing the Model Card¶

fm = featrix.foundational_model("session-id")
model_card = fm.get_model_card()

print(model_card)

Model Card Contents¶

Section	What It Contains
`columns`	Detected type, encoding strategy, statistics for each column
`excluded_columns`	Columns auto-excluded and why
`training_config`	Epochs, batch size, architecture decisions
`quality_metrics`	Loss history, convergence metrics
`calibration`	Probability calibration method and results
`data_summary`	Row count, column count, null distribution
`warnings`	Any issues detected during training

Example: Checking Column Decisions¶

model_card = fm.get_model_card()

for col_name, col_info in model_card.get('columns', {}).items():
    print(f"\n{col_name}:")
    print(f"  Type: {col_info.get('detected_type')}")
    print(f"  Encoding: {col_info.get('encoding_strategy')}")
    print(f"  Null rate: {col_info.get('null_rate', 0):.1%}")

Publishing Models¶

Publishing moves a model to a protected production directory where it won't be garbage collected.

Publish a Foundational Model¶

fm.publish(
    org_id="my_org",
    name="customer_model_v1"
)

This:

Copies the model to a permanent location
Protects it from automatic cleanup
Makes it findable by the published name

Publish a Training Checkpoint¶

If training is still running or you want to snapshot a specific epoch:

# Publish checkpoint from epoch 50
checkpoint_fm = fm.publish_checkpoint(
    name="customer_model_checkpoint_50",
    org_id="my_org",
    checkpoint_epoch=50
)

# The returned FoundationalModel is ready to use
predictor = checkpoint_fm.create_binary_classifier(target_column="churned")

Unpublish¶

Remove a model from the published directory:

fm.unpublish()

Delete¶

Mark a model for garbage collection:

fm.delete()

API Endpoints¶

Create named, versioned API endpoints for production prediction serving.

Create an Endpoint¶

predictor = fm.list_predictors()[0]

endpoint = predictor.create_api_endpoint(
    name="churn_api_v1",
    description="Production churn prediction endpoint"
)

print(f"Endpoint ID: {endpoint.id}")
print(f"API Key: {endpoint.api_key}")
print(f"URL: {endpoint.url}")

Make Predictions via Endpoint¶

# Using the endpoint object
result = endpoint.predict(
    {"age": 35, "income": 50000},
    api_key=endpoint.api_key
)

# Or via HTTP
import requests

response = requests.post(
    f"{endpoint.url}/predict",
    headers={"X-API-Key": endpoint.api_key},
    json={"age": 35, "income": 50000}
)
print(response.json())

API Key Management¶

# Regenerate the API key
new_key = endpoint.regenerate_api_key()

# Revoke the key (make endpoint public or disabled)
endpoint.revoke_api_key()

Usage Statistics¶

stats = endpoint.get_usage_stats()
print(f"Total predictions: {stats['usage_count']}")
print(f"Last used: {stats['last_used_at']}")

Delete an Endpoint¶

endpoint.delete()

Prediction Feedback¶

Send ground truth labels back to Featrix to track model performance and enable retraining.

From a Prediction Result¶

result = predictor.predict(record)

# Later, when you know the actual outcome:
if result.predicted_class != actual_label:
    feedback = result.send_feedback(ground_truth=actual_label)
    feedback.send()

Direct Feedback (Using Stored UUID)¶

# If you stored the prediction_uuid
featrix.prediction_feedback(
    prediction_uuid="stored-uuid-from-prediction",
    ground_truth="correct_label"
)

Feedback for Regression¶

featrix.prediction_feedback(
    prediction_uuid="stored-uuid",
    ground_truth=actual_numeric_value  # float
)

Webhooks¶

Configure webhooks to get notified about model and prediction events.

Configure Webhooks on a Predictor¶

predictor.configure_webhooks(
    # Training events
    training_finished="https://your-server.com/webhooks/training-done",
    training_started="https://your-server.com/webhooks/training-started",

    # Monitoring alerts
    alert_drift="https://your-server.com/webhooks/drift-detected",
    alert_performance_degradation="https://your-server.com/webhooks/perf-degradation",
    alert_error_rate="https://your-server.com/webhooks/error-rate",
    alert_quota_threshold="https://your-server.com/webhooks/quota-warning",

    # Prediction events
    prediction_error="https://your-server.com/webhooks/prediction-error",
    batch_job_completed="https://your-server.com/webhooks/batch-done",

    # Security
    webhook_secret="your-secret-for-signature-verification"
)

Available Webhook Events¶

Event	Trigger
`training_started`	Predictor training begins
`training_finished`	Predictor training completes
`alert_drift`	Data drift detected in predictions
`alert_performance_degradation`	Model performance dropping
`alert_error_rate`	Prediction error rate increasing
`alert_quota_threshold`	Approaching usage quota
`prediction_error`	Individual prediction fails
`batch_job_completed`	Batch prediction job finishes

Check Current Webhooks¶

webhooks = predictor.get_webhooks()
print(webhooks)

Disable a Webhook¶

predictor.disable_webhook("alert_drift")

Model Deprecation¶

When replacing a model, deprecate the old version with a warning:

from datetime import datetime, timedelta

# Set expiration 90 days from now
expiration = (datetime.now() + timedelta(days=90)).isoformat() + "Z"

fm.deprecate(
    warning_message="This model is being replaced by v2. Please migrate by the expiration date.",
    expiration_date=expiration
)

After deprecation:

The model still works
API responses include a deprecation warning
Webhooks can alert consumers
After expiration, behavior depends on your configuration

Model Cloning¶

Copy a model to a different compute cluster (for redundancy or regional deployment):

cloned_fm = fm.clone(
    target_compute_cluster="us-east-node",
    new_name="customer_model_us_east"
)

Complete Lifecycle Example¶

from featrixsphere.api import FeatrixSphere
from datetime import datetime, timedelta

featrix = FeatrixSphere()

# 1. Train the model
fm = featrix.create_foundational_model(
    name="customer_churn_v1",
    data_file="customers.csv"
)
fm.wait_for_training()

# 2. Train predictor
predictor = fm.create_binary_classifier(
    target_column="churned",
    rare_label_value="yes",
    cost_false_negative=500
)
predictor.wait_for_training()

# 3. Verify quality
print(f"AUC: {predictor.auc:.4f}")
assert predictor.auc >= 0.75, "Model quality too low"

# 4. Publish to production
fm.publish(org_id="acme_corp", name="churn_model_v1")

# 5. Create API endpoint
endpoint = predictor.create_api_endpoint(
    name="churn_api_v1",
    description="Production churn prediction"
)

# 6. Configure monitoring
predictor.configure_webhooks(
    alert_drift="https://slack-webhook.acme.com/drift",
    alert_performance_degradation="https://slack-webhook.acme.com/perf"
)

# 7. Document the deployment
print(f"""
Deployment Complete:
- Model: {fm.id}
- Predictor: {predictor.id}
- Endpoint: {endpoint.url}
- API Key: {endpoint.api_key}
- AUC: {predictor.auc:.4f}
""")

# Later: Deprecate when v2 is ready
# fm.deprecate(
#     warning_message="Migrating to v2",
#     expiration_date=(datetime.now() + timedelta(days=90)).isoformat() + "Z"
# )

Best Practices¶

1. Always Check the Model Card Before Publishing¶

model_card = fm.get_model_card()
print("Excluded columns:", model_card.get('excluded_columns'))
print("Warnings:", model_card.get('warnings'))

2. Set Up Webhooks Before Going to Production¶

predictor.configure_webhooks(
    alert_drift="...",
    alert_performance_degradation="..."
)

3. Store Prediction UUIDs for Feedback¶

# In your application
prediction_log.append({
    "input": record,
    "prediction": result.predicted_class,
    "uuid": result.prediction_uuid,
    "timestamp": datetime.now()
})

4. Send Feedback When You Get Ground Truth¶

# When you learn the actual outcome
for log_entry in prediction_log:
    if log_entry['actual'] is not None:
        featrix.prediction_feedback(
            prediction_uuid=log_entry['uuid'],
            ground_truth=log_entry['actual']
        )

5. Version Your Models¶

Use clear naming:

fm.publish(org_id="acme", name="churn_model_v1_2024_01")

6. Deprecate, Don't Delete Immediately¶

Give consumers time to migrate:

fm.deprecate(
    warning_message="Use churn_model_v2 instead",
    expiration_date="2024-04-01T00:00:00Z"
)

Summary¶

Task	Method
View model decisions	`fm.get_model_card()`
Publish to production	`fm.publish(org_id, name)`
Create API endpoint	`predictor.create_api_endpoint()`
Send feedback	`result.send_feedback(ground_truth)`
Set up alerts	`predictor.configure_webhooks()`
Deprecate old model	`fm.deprecate(message, expiration)`
Clone for redundancy	`fm.clone(target_cluster)`