Best practices for measuring and avoiding overfitting?

External question — Cross Validated Stack Exchange Author: B Seven Original post: https://stats.stackexchange.com/questions/15585 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I started with training/validation sets, but the validation set always gets tainted. Also, the time series data is always changing because the market is always changing. How do you measure this and determine the likelihood of consistent results on unseen data? Thanks.

External answer — Cross Validated Stack Exchange Author: Dikran Marsupial Original post: https://stats.stackexchange.com/a/15590 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. For over-fitting in feature selection, then a paper worth reading is C. Ambroise and G. J. McLachlan, "Selection bias in gene extraction on the basis of microarray gene-expression data", PNAS, vol. 99 no. 10 6562-6566, May 2002. http://dx.doi.org/10.1073/pnas.102102699 For a discussion of the same sort of problem that arises in model selection, see G. C. Cawley, N. L. C. Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", Journal of Machine Learning Research, 11(Jul):2079−2107, 2010. http://jmlr.csail.mit.edu/papers/v11/cawley10a.html The way to solve the problem of the validation set becoming tainted is to use nested cross-validation, so the method used to make choices about the model is performed independently in each fold of the cross-validation used for performance estimation. Essentially the performance estimation must estimate the performance of the whole model fitting procedure (fitting the model, feature selection, model selection, everything). The other approach is to be a Bayesian. The risk of over-fitting is introduced whenever you optimise a criterion based on a finite sample of data, so if you marginalise (integrate out) rather than optimise then classical over-fitting is impossible. You do however have the problem of specifying the priors.