Look Beyond Accuracy
Only 10 of 1,000 emails are spam. A model calling every message normal achieves 99% accuracy while finding no spam. A high number may not mean useful performance.
Give datasets distinct roles
- Training: Learn values such as means and model weights.
- Validation: Choose settings such as neighbor counts or decision thresholds.
- Test: Evaluate the finished selection.
Repeated records from the same person may require splitting by person. Forecasting should respect time order. Evaluation should resemble the intended use.
Which errors matter?
Treating spam as positive, TP means correctly found spam, FP means normal mail incorrectly flagged, and FN means missed spam.
true_positive, false_positive, false_negative = 6, 2, 4
precision = true_positive / (true_positive + false_positive)
recall = true_positive / (true_positive + false_negative)
print(precision)
print(recall)
Precision is 0.75; recall is 0.6. Precision asks how many flagged messages were actually spam. Recall asks how much actual spam was found. A zero denominator requires an explicit reporting convention.
Blocking normal mail and missing spam have different costs. Compare both errors as you vary the decision threshold. Strong training performance with weak validation performance can indicate overfitting.
Check your understanding
Can you fill missing values with the mean of the entire dataset before splitting training and test data?
Show explanation
That lets test information enter training. Split first, learn the mean from training data only, and apply that same value elsewhere. This is an example of data leakage; see the preprocessing and leakage guide.