Comparing RMSEs of multiple test sets having different sizes

External question — Data Science Stack Exchange Author: Aditya Kulkarni Original post: https://datascience.stackexchange.com/questions/100018 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. The data I have is a time series data (stock returns), and I am training a Random Forest Regressor on it. Total observations = 2499 To better evaluate the performance, I have implemented rolling windows testing with training window sizes = 500, 700, 900,..., 2100. Though instinctively it would seem obvious to choose a window size which produced lowest RMSE, how can I be sure that the comparison is fair? I mean with increasing window size, the test set size decreases. With window size 500, test set size is 1999. With window size 700, test set size is 1799. I think the same question applies to Expanding Window So is it sensible to compare RMSEs when test samples are decreasing in size? If not, then how should one choose the best training window?

External answer — Data Science Stack Exchange Author: Jayaram Iyer Original post: https://datascience.stackexchange.com/a/100066 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. The "mean" in RMSE ensures that value of this metric are comparable and of the same scale irrespective of the size of the test window. Have you looked at k-fold cross validation? That allows you to use a single train test split ratio of say 70:30, yet create k different datasets to compute the RMSE on. K-fold cross validation is widely adopted and well researched. It's probably better known strategy than what you describe.