Does the scaler actually matter?

Feature scaling is routinely applied before distance-based and probabilistic classifiers, but the choice of which scaler to use is usually made by convention rather than evidence, and papers rarely report whether the resulting differences are statistically distinguishable from noise or merely an artifact of a single train/test split.

• RQ1 — How does feature scaling affect KNN performance on the UCI Dry Bean dataset?
• RQ2 — How does feature scaling affect Gaussian Naive Bayes performance?
• RQ3 — Which scaling strategy is best for KNN?
• RQ4 — Which scaling strategy is best for Gaussian Naive Bayes?
• RQ5 — Which scaling strategy is most stable for each classifier?
• RQ6 — Are the differences between scaling strategies statistically significant?

UCI Dry Bean Dataset ↗ — 13,611 samples, 16 geometric features, 7 bean varieties. 0 missing values; 68 duplicate rows, retained.

Class distribution: DERMASON 3546, SIRA 2636, SEKER 2027, HOROZ 1928, CALI 1630, BARBUNYA 1322, BOMBAY 522

• 5 scaling strategies × 2 classifiers = 10 configurations.
• RepeatedStratifiedKFold with 5 folds × 3 repeats = 15 evaluations per configuration; 150 model evaluations in total.
• Splits are generated once and reused unchanged across all 10 configurations, so every later test is genuinely paired.
• KNN fixed at n_neighbors=5. GaussianNB at scikit-learn defaults. random_state=42. No hyperparameter tuning, by design.
• Scaling is fitted only on the training fold inside a scikit-learn Pipeline; the full dataset is never scaled before cross-validation.
• Wilcoxon signed-rank tests (paired, non-parametric) with Holm–Bonferroni step-down correction at alpha = 0.05. Significance is reported from adjusted p-values only.

• KNN: unscaled mean weighted F1 = 0.7237; best scaled = 0.9226 (StandardScaler). 8 of 10 pairwise comparisons among KNN scaling strategies were significant after correction.
• GaussianNB: unscaled mean weighted F1 = 0.7615; best scaled = 0.8969. Only 4 of 10 pairwise comparisons were significant — and all four are the comparisons against No Scaling.
• For Gaussian Naive Bayes, StandardScaler, MinMaxScaler, RobustScaler and MaxAbsScaler produced identical mean weighted F1 to four decimal places (0.8969, SD 0.0064). The study reports them as tied.
• KNN StandardScaler vs MinMaxScaler was significant (adjusted p = 0.02509), but StandardScaler vs RobustScaler was not (adjusted p = 0.0511) — a difference that would have looked real on a single split.
• 24 of 40 comparisons were significant across both metrics and both classifiers after Holm correction.

Every figure above is copied from results/research_summary.txt and results/tables/final_comparison_table.csv, which are generated by research.py rather than written by hand. The report states: “This report is generated entirely from the results produced in this run. No values below are manually written or assumed.”

Interpretation, not output: the four scalers tie for Gaussian Naive Bayes because each is a per-feature affine transform and the classifier fits a per-feature mean and variance, so the fitted model absorbs the transform. Unscaled data is the only condition that changes what the classifier can represent well. The experiment was not designed to test that explanation; it reports the tie.

Statistical significance indicates an observed difference is unlikely to be due to chance alone under this repeated-CV design; it does not by itself mean the difference is large enough to matter in practice. These results describe this dataset and this evaluation protocol only.

• No hyperparameter tuning was performed by design; results reflect a fixed KNN (k=5) and default GaussianNB, not the best achievable performance of either classifier.
• Repeated CV folds are not fully independent samples, so the reported 95% confidence intervals and p-values are approximate, standard-practice heuristics rather than exact.
• Only KNN and Gaussian Naive Bayes were studied; conclusions do not extend to other classifiers.
• The confusion matrices use a single held-out split and illustrate error patterns; they are not an independent performance estimate and were not used for model selection.
• Duplicate rows (68) were retained; results were not recomputed with duplicates removed as a sensitivity check.

This list is produced automatically by the analysis, not curated afterwards. One of its entries — the retained duplicate rows — is an omission the software flagged rather than something I remembered.

The result I expected was a ranking. The result I got was that half the ranking is noise — and that the honest output of the study is a smaller claim than the one I set out to make.