Fitting a LSTM for stock price prediction using industry sector data

External question — Cross Validated Stack Exchange Author: Roger Gough Original post: https://stats.stackexchange.com/questions/544317 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I am quite new to the theory of RNNs so please excuse me if the question is trivial. I am trying to fit a multivariate LSTM neural network to predict stock prices from a firm in the S&P 500 list. I have seen many applications where the opening price is forecasted using the stock's own closing price, trading volume, highest and lowest trading prices. However, I was wondering whether it would be sensible to try and forecast the firm's opening price using other firms' opening prices (specifically, I was thinking of other firms in the same sector in the S&P list, e.g. healthcare or IT sectors depending on the chosen firm). Since prices in the same sector are likely to be highly correlated I thought this could be a good approach. However, this would create a dataset with many features (much more than in the examples I have seen so far) and I am thus worried that it would lead to overfitting. Could this be mitigated by increasing the size of the time series?

External answer — Cross Validated Stack Exchange Author: Adam Kells Original post: https://stats.stackexchange.com/a/544502 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. There a whole bunch of ways you could go about this. Overfitting is definitely a danger if you have too many features. A few suggestions: Rather than using all of the other stocks in the sector and potentially exploding the number of features in your dataset, try constructing some indicator features (e.g. average historic returns in sector, average volatility in sector, average historic returns of all stocks, fraction of stocks with positive returns). Hand crafting these will probably be your best return on investment in terms of performance. Clustering approaches (kmeans etc.) are good too, they can help you find clusters of stocks which are positively (or negatively) correlated with the stock you want to predict.