The Problem of Cross-Validation: Averaging and Bias, Repetition and Significance

摘要: Cross-Validation (CV) is the primary mechanism used in Machine Learning to control generalization error absence of sufficiently large quantities marked up (tagged or labelled) data undertake independent testing, training and validation (including early stopping, feature selection, parameter tuning, boosting and/or fusion). Repeated (RCV) try further improve accuracy our performance estimates, including compensating for outliers. Typically a researcher will compare new target algorithm against wide range competing algorithms on standard datasets. The combination many folds, CV repetitions, parameterizations, sets, adds very number points compare, massive multiple testing problem quadratic individual test combinations. Research sometimes involves basic significance provides confidence intervals, but seldom addresses whereby assumption p<.05 means that we expect spurious "significant" result 1 20 pairs. This paper defines explores protocol reduces scale repeated whilst providing principled way erosion due testing.