Tradeoff between Prediction Interval Accuracy & Mean Squared Error
Tradeoff between Prediction Interval Accuracy & Mean Squared Error
Loading saved threads...
hayfreed · External communityPost link
External question — Cross Validated Stack Exchange
Author: hayfreed
Original post: https://stats.stackexchange.com/questions/651367
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
My goal is to quantify the prediction uncertainty in a model regressing climate covariates against GDP. I start with a model with temperature as a third degree polynomial, country fixed effects (
$\alpha_i$
}, year fixed effects (
$\theta_t$
), and incremental time trend by country (
$\gamma_i$
).
$$
GDP_{it} = \beta_1 * Temp_{it} + \beta_2 * Temp^2_{it} + \beta_3 * Temp^3_{it} + \alpha_i + \theta_t + \gamma_i
$$
I use some withheld data to gather out-of-sample Mean Squared Error for the above model. I also use the standard error of the out-of-sample predictions to construct 95% prediction intervals and then check the actual percentage of real Y (GDP) values that fall within those intervals as prediction interval accuracy.
Out-of-sample MSE: 0.017
Out-of-sample Prediction interval accuracy: 0.577
In an effort to get the prediction interval accuracy closer to the 95% target, I try a different model with higher-degree polynomial time trends, shown below:
$$
GDP_{it} = \beta_1 * Temp_{it} + \beta_2 * Temp^2_{it} + \beta_3 * Temp^3_{it} + \alpha_i + \theta_t + \gamma_i + \gamma^2_i + \gamma^3_i
$$
Out-of-sample MSE: 0.018
Out-of-sample Prediction interval accuracy: 0.722
The prediction interval accuracy is much improved as a result of the prediction intervals being wider, presumably because the model is incorporating more of the variance in the training data. However, probably due to overfitting from the additional model complexity, the MSE is higher in the second model than the first.
My question has less to do with this specific example than this phenomenon I have observed in general. I am wondering:
Is there a single underlying phenomenon that explains why increasing model complexity causes both the out-of-sample prediction intervals to move closer to the 95% target while at the same time increases the out-of-sample MSE of the model, or are these essentially independent observations?
If my goal is to quantify model uncertainty the best that I can, what is the right way to think about trading off higher quality (in this case meaning wider) out-of-sample prediction intervals for a subsequent increase in out-of-sample MSE?
Quote
Report
Stephan Kolassa · External communityPost link
External answer — Cross Validated Stack Exchange
Author: Stephan Kolassa
Original post: https://stats.stackexchange.com/a/651505
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
The only mechanism I could think of that would increase both the MSE and the PI converage would be straightforward overfitting. You overfit in the training sample, then get larger MAEs in the holdout set, and that of course leads to wider PIs, which have a larger coverage. In the meantime, you also get a larger MSE, again because of the overfitting.
Third degree polynomials can and will explode for very large and very small predictor values, and the effect is exacerbated by the fact that you very likely have few observations at these predictor values.
I would recommend you use natural or other splines. These are flexible across most of your predictor range, but linear at the extremes.
In addition, it looks like your MAE approach to PIs is not working very well. You are not saying how exactly you turn your MAE into PIs, but it presumably uses some distributional assumption. It may be that this assumption is not working out well. Alternatives would be a direct quantile regression for two quantiles using a pinball loss, or possibly conformal prediction. (Also, I do wonder how you get year fixed effects for forecasting. Are you forecasting these themselves?)
I would argue (
in this paper
and also
here
) that it is important to first understand the context. What are you forecasting for? Which subsequent decisions will depend on the forecast? What you mean by a "best way to quantify uncertainty" can only be understood in this context.
It seems like you want both an expectation forecast and a prediction interval. One way to proceed would be to aim for full predictive densities, from which you can extract point forecasts like the expectation and quantiles. Predictive densities can be assessed using proper scoring rules,
the tag wiki has more information
.
This thread gives some general resources about forecasting:
Resources/books for project on forecasting models
.
Quote
Report
Post Reply
Quoted from Forex.com.bd-Editorial External question — Cross Validated Stack Exchange Author: hayfreed Source score (net votes, not local likes): 2 Original post: https://stats.stackexchange.com/questions/651367 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. My goal is to quantify the prediction uncertainty in a model regressing climate covariates against GDP. I start with a model with temperature as a third degree polynomial, country fixed effects ( $\alpha_i$ }, year fixed effects ( $\theta_t$ ), and incremental time trend by country ( $\gamma_i$ ). $$ GDP_{it} = \beta_1 * Temp_{it} + \beta_2 * Temp^2_{it} + \beta_3 * Temp^3_{it} + \alpha_i + \theta_t + \gamma_i $$ I use some withheld data to gather out-of-sample Mean Squared Error for the above model. I also use the standard error of the out-of-sample predictions to construct 95% prediction intervals and then check the actual percentage of real Y (GDP) values that fall within those intervals as prediction interval accuracy. Out-of-sample MSE: 0.017 Out-of-sample Prediction interval accuracy: 0.577 In an effort to get the prediction interval accuracy closer to the 95% target, I try a different model with higher-degree polynomial time trends, shown below: $$ GDP_{it} = \beta_1 * Temp_{it} + \beta_2 * Temp^2_{it} + \beta_3 * Temp^3_{it} + \alpha_i + \theta_t + \gamma_i + \gamma^2_i + \gamma^3_i $$ Out-of-sample MSE: 0.018 Out-of-sample Prediction interval accuracy: 0.722 The prediction interval accuracy is much improved as a result of the prediction intervals being wider, presumably because the model is incorporating more of the variance in the training data. However, probably due to overfitting from the additional model complexity, the MSE is higher in the second model than the first. My question has less to do with this specific example than this phenomenon I have observed in general. I am wondering: Is there a single underlying phenomenon that explains why increasing model complexity causes both the out-of-sample prediction intervals to move closer to the 95% target while at the same time increases the out-of-sample MSE of the model, or are these essentially independent observations? If my goal is to quantify model uncertainty the best that I can, what is the right way to think about trading off higher quality (in this case meaning wider) out-of-sample prediction intervals for a subsequent increase in out-of-sample MSE?
Checking account access…