Module Review: Model Evaluation
1. Key Takeaways
- Bias-Variance Tradeoff: The fundamental tension in ML. Simple models have high bias (underfit), complex models have high variance (overfit).
- Error Decomposition: Error = Bias2 + Variance + Irreducible Error. You cannot reduce irreducible error.
- Classification Metrics: Accuracy is dangerous for imbalanced data. Use Precision (low FP), Recall (low FN), and F1-Score (balance).
- ROC/AUC: AUC measures the probability that a positive sample ranks higher than a negative one. 0.5 is random, 1.0 is perfect.
- Regression Metrics: MSE penalizes large errors heavily. MAE is more robust to outliers. R2 measures explained variance.
- Cross-Validation: K-Fold gives a more reliable performance estimate than a single Train/Test split. Use Stratified K-Fold for classification.
2. Cheat Sheet
| Metric | Type | Formula (Simplified) | Use Case |
|---|---|---|---|
| Accuracy | Class | (TP+TN) / Total | Balanced datasets only. |
| Precision | Class | TP / (TP+FP) | Minimize False Positives (e.g., Email Spam). |
| Recall | Class | TP / (TP+FN) | Minimize False Negatives (e.g., Cancer Detection). |
| F1 Score | Class | 2 * (P * R) / (P + R) | Imbalanced datasets, balance P & R. |
| MSE | Reg | Mean((y - y_pred)2) | Standard regression, penalize outliers. |
| MAE | Reg | Mean(|y - y_pred|) | Regression with outliers you want to ignore. |
| R2 | Reg | 1 - (SS_res / SS_tot) | Explainability (0 to 1). |
3. Flashcards
What is the symptom of High Bias?
High Training Error AND High Validation Error (Underfitting).
What is the symptom of High Variance?
Low Training Error BUT High Validation Error (Overfitting).
When should you use F1 Score over Accuracy?
When the class distribution is imbalanced (e.g., 99% Negative, 1% Positive).
What does AUC = 0.5 indicate?
The model performs no better than random guessing.
Why use Stratified K-Fold?
To ensure each fold has the same percentage of samples for each target class as the complete set.
What is "Data Leakage" in Cross-Validation?
When information from the test/validation set accidentally "leaks" into the training process (e.g., normalizing before splitting).
4. Next Steps
Now that you can evaluate models effectively, you are ready to move on to Feature Engineering, where we will learn how to create better inputs to improve these scores.