LSTM for Stock Return Prediction

External question — Data Science Stack Exchange Author: InvestingScientist Original post: https://datascience.stackexchange.com/questions/93570 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I am writing my masters thesis and am using LSTMs for daily stock return prediction. So far I am only predicting numerical values but will soon explore a classification style problem and predict whether it will go up or down each day. I have explored several scenarios A single LSTM using as input only the past 50 days return data A stacked (2 layers) using as input only the past 50 days return data The results are not great for either (and I didn't expect them to be). So I tried some feature engineering using 3 day MA, 5 day MA, 10 day MA, 25 day MA, 50 day MA of the daily returns as well as the actual daily return, meaning I have 6 input features. All other variables are kept constant yet the model now overfits (see the training and test loss plots below). Does anyone have any ideas why this may be? Test Loss in orange and Train in blue

External answer — Data Science Stack Exchange Author: hH1sG0n3 Original post: https://datascience.stackexchange.com/a/93571 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I am not sure this type of model is a good use case for the particular task. More specifically, citing Chollet from Deep Learning with Python book, Always remember that when it comes to markets, past performance is not a good predictor of future returns—looking in the rear-view mirror is a bad way to drive. Machine learning, on the other hand, is applicable to datasets where the past is a good predictor of the future. — Deep learning with python, Francois Chollet What is essentially being argued here is that, stock historical data is not a phenomenon that repeats itself based on its own underlying distribution.

External answer — Data Science Stack Exchange Author: leviva Original post: https://datascience.stackexchange.com/a/112947 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. First of all, adding features to your model can easily cause overfitting. Even adding a column of random values can cause overfitting. Take for example your series with exactly one feature - the price. The price can be the same value X several times - first when the series is rising, and then when it's decreasing. if you only use the price, you are forced to try to learn the average of the future value at that price. Now add a moving average. Now when the price is X and the series is rising, the MA will be less than X, while when the series is decreasing the MA will be more than X. your model can now on one hand learn to predict better (if the MA is actually a good feature), but it can also hack the data batter because it is now better able to memorize the training set. Now add another 50 moving averages to the features. you may now have a unique value of features for every output, so your model can fit the data extremely well, but obviously, that noise will not predict anything on the test set. How many features of (basically) the same thing to use is a good question, but you need to remember you're not actually adding a lot of information with those features. And frankly, LSTMs can calculate moving averages pretty well themselves.