Time series model hardly fitting well

External question — Data Science Stack Exchange Author: GusC Original post: https://datascience.stackexchange.com/questions/111930 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I'm trying to forecast Google's stock prices. I've made two models one with LSTM and another one that's Bidirectional LSTM, but the forecasted values don't converge quite well with the test values. I've tried different parameters, but I've hardly got any improvement. First I had to install these libraries: !pip install yfinance !pip install yahoofinancials Then I import the needed libraries: import matplotlib as mpl import matplotlib.pyplot as plt import numpy as np import os import pandas as pd import yfinance as yf from yahoofinancials import YahooFinancials import datetime from datetime import date from datetime import timedelta import tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Input, Dense, GRU, LSTM, Embedding, SimpleRNN, Activation, Dropout from tensorflow.keras.optimizers import RMSprop from tensorflow import keras from sklearn.metrics import r2_score Next I set the parameters for the data I'm about to download: start_date = date(2004, 8, 1) end_date = date.today() - datetime.timedelta(days=1) I download the data: In [5]: df = yf.download('GOOG', start = start_date, end = end_date, interval = '1d') Out[5]: [*********************100%***********************] 1 of 1 completed I visualize the df: In [6]: df.head() Out[6]: Open High Low Close Adj Close Volume Date 2004-08-19 49.813290 51.835709 47.800831 49.982655 49.982655 44871361 2004-08-20 50.316402 54.336334 50.062355 53.952770 53.952770 22942874 2004-08-23 55.168217 56.528118 54.321388 54.495735 54.495735 18342897 2004-08-24 55.412300 55.591629 51.591621 52.239197 52.239197 15319808 2004-08-25 52.284027 53.798351 51.746044 52.802086 52.802086 9232276 Then I start to transform the data to use it: In [7]: uni_data = np.array(df['Adj Close'])[1:].reshape(1,-1) In [7]: uni_data Out[7]: array([[ 53.95277023, 54.49573517, 52.23919678, ..., 2143.87988281, 2207.81005859, 2132.7199707 ]]) In [7]: data_transpose = uni_data.T I proceed to add regularisation to the data: In [8]: tset = (uni_data.T-np.mean(uni_data.T))/np.std(uni_data.T) In [9]: data_mean = np.mean(uni_data) In [9]: data_std = np.std(uni_data) In [10]: tset = tset.T I generate the batch process: In [11]: n_steps = 15 In [11]: predict_ahead = 1 In [12]: def generate_batches(uni_data , n_steps, predict_ahead): _ , lenght = uni_data.shape batchs = lenght - horizonte - predict_ahead + 1 X = np.zeros((batchs, n_steps + predict_ahead )) print('Initial shape of the data : ' , uni_data.shape) print('Final shape of the data : ' , X.shape) for el in range(batchs): data_slice = uni_data[0, el : el + n_steps + predict_ahead] X[el , :] = data_slice return X , batchs In [13]: X , batchs = generate_batches(tset , n_steps, predict_ahead) Out[13]: Initial shape of the data : (1, 4488) Out[13]: Final shape of the data : (4473, 16) The Train/test split is made: In [14]: tf.random.set_seed(42) In [14]: porcentaje_train = 0.8 In [14]: train_range = int(round(len(df)) * porcentaje_train) In [14]: train = range(train_range) In [14]: valid = set(range(batchs)) - set(train) In [14]: valid = np.array(list(valid)) In [15]: X_train = X[train , :- predict_ahead ] In [15]: X_valid = X[valid , :- predict_ahead ] In [15]: Y_train = X[train , - predict_ahead :] In [15]: Y_valid = X[valid , - predict_ahead :] In [16]: print('X Train shape : ' , X_train.shape) In [16]: print('Y Train shape : ' , Y_train.shape) In [16]: print('X Valid shape : ' , X_valid.shape) In [16]: print('Y Valid shape : ' , Y_valid.shape) Out [16]: Train shape : (3591, 15) Out [16]: Y Train shape : (3591, 1) Out [16]: X Valid shape : (882, 15) Out [16]: Y Valid shape : (882, 1) Then I make the LSTM model In [17]: epochs = 25 In [18]: def lstm_fit(X_train, Y_train , X_valid , Y_valid, predict_ahead, num_hidden_layers , activ, num_units, loss_type): _ , _ , num_vars = X_train.shape model1 = lstm(predict_ahead, num_hidden_layers , activ, num_units, loss_type, num_vars) hist = model1.fit(x = X_train , y = Y_train, epochs = 20 ,validation_data=(X_valid , Y_valid) , verbose=0) return model1 In [19]: tf.random.set_seed(42) def lstm(predict_ahead, num_hidden_layers , activ, num_units, loss_type, num_vars): model1 = keras.models.Sequential() model1.add(keras.layers.InputLayer(input_shape = (n_steps,num_vars))) for el in range(num_hidden_layers): model1.add(keras.layers.LSTM(activation = activ, units = num_units , return_sequences = True)) model1.add(keras.layers.LSTM(units = num_units)) model1.add(keras.layers.Dense(predict_ahead)) model1.compile(optimizer='adam', loss=loss_type, metrics=['mae' , 'mse', 'mape']) return model1 In [20]: num_hidden_layers = 3 activ = keras.activations.relu loss_type = keras.losses.mean_squared_error num_units = 64 dropout_rate = 0.3 Then I make the Bidirectional LSTM model: First, I made a callback function since the model seemed to be overfitting. In [21]: class myCallback(tf.keras.callbacks.Callback): def on_epoch_end(self, epoch, logs={}): if(logs.get('loss')) <0.4: print("\nReached below 0.4 loss so cancelling training!") self.model.stop_training = True In [21]: callbacks = myCallback() In [22]: def bidir_fit(X_train, Y_train , X_valid , Y_valid, predict_ahead, num_hidden_layers , activ, num_units, loss_type, callbacks): _ , _ , num_vars = X_train.shape model2 = bidir(predict_ahead, num_hidden_layers , activ, num_units, loss_type, num_vars) hist = model2.fit(x = X_train , y = Y_train, epochs = epochs ,validation_data=(X_valid , Y_valid) , verbose=0 , callbacks=[callbacks]) return model2 In [23]: tf.random.set_seed(42) def bidir(predict_ahead, num_hidden_layers , activ, num_units_bidir, loss_type, num_vars): model2 = tf.keras.models.Sequential() model2.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32) , input_shape=X_train.shape[-2:] )) model2.add(tf.keras.layers.Dense(32, activation='relu')) model2.add(Dropout(0.2)) model2.add(tf.keras.layers.Dense(6, activation='relu')) model2.add(tf.keras.layers.Dense(1, activation='relu')) model2.compile(optimizer = tf.keras.optimizers.Adam() , loss='mse',metrics=['mae' , 'mse', 'mape']) return model2 Then I began training the models. In [24]: X_train = np.expand_dims(X_train ,2) In [24]: X_valid = np.expand_dims(X_valid ,2) In [24]: Y_train = np.expand_dims(Y_train ,2) In [24]: Y_valid = np.expand_dims(Y_valid ,2) In [25]: X_train.shape Out [25]: (3591, 15, 1) In [26]: lstm = lstm_fit(X_train, Y_train , X_valid , Y_valid , predict_ahead, num_hidden_layers , activ, num_units, loss_type) In [26]: bidir = bidir_fit(X_train, Y_train , X_valid , Y_valid , predict_ahead, num_hidden_layers , activ, num_units, loss_type, callbacks) Out [26]: Reached below 0.4 loss so cancelling training! Then I began evaluating and plotting the results. In [27]: print(lstm.evaluate(X_train, Y_train)) In [27]: print(bidir.evaluate(X_train, Y_train)) Out [27]: 13/113 [==============================] - 1s 8ms/step - loss: 0.0011 - mae: 0.0215 - mse: 0.0011 - mape: 23.0664 [0.0011325260857120156, 0.02146766521036625, 0.0011325260857120156, 23.0664119720459] Out [27]: 113/113 [==============================] - 0s 2ms/step - loss: 0.3158 - mae: 0.4743 - mse: 0.3158 - mape: 85.1597 [0.31578168272972107, 0.47427594661712646, 0.31578168272972107, 85.15969848632812] The LSTM returned a MAPE of 23.0664 whereas the Bidirectional LSTM returned a MAPE of 85.1597, which is quite high. This is a problem I found, because no matter how many changes I made to that model, its MAPE was always much higher than I wanted. In [28]: plt.figure(figsize=(10,10)) In [28]: plt.plot(data_mean + data_std * Y_valid.reshape(-1,1), label = 'Real') In [28]: plt.plot(data_mean + data_std * lstm(X_valid), label = 'LSTM') In [28]: plt.plot(data_mean + data_std * bidir(X_valid), label = 'Bidireccional') In [28]: plt.title('Valores Reales y Previstos') #This means Real vs Forecasted In [28]: plt.grid() In [28]: plt.legend() In [28]: plt.show() The plot returned this: As you can see, the models don't fit quite well, at least not during the first half of it, and I've already made a few changes in both models, but the first half never quite fits well. In [29]: historylstm = lstm.fit(X_train, Y_train, validation_data = (X_valid, Y_valid), verbose = 0) In [29]: historybidir = bidir.fit(X_train, Y_train, validation_data = (X_valid, Y_valid), verbose = 0) In [30]: lstm_error = pd.DataFrame.from_dict(historylstm.history).iloc[0:1, 7:8].to_string(index=False, header=False) In [30]: bidir_error = pd.DataFrame.from_dict(historybidir.history).iloc[0:1, 7:8].to_string(index=False, header=False) In [31]: print(f"LSTM Model - Val MAPE:{lstm_error}") In [31]: print(f"Bidirectional Model - Val MAPE:{bidir_error }") Out [31]: LSTM Model - Val MAPE:23.216141 Out [31]: Bidirectional Model - Val MAPE:14.745646 And with this last bit I'm in my current conundrum. For the LSTM model, the training MAPE was 23.0664, and the MAPE for validation is 23.21641 which seems quite a good fit, but for the Bidirectional model the training MAPE was 85.1597 whereas the validation on was 14.745646 which isn't a good fit, neither does it mean it's overfittiing or underfitting it's just quite odd. You would think the low validation results were because of the callback function and the dropout, but the validation MAPE was even lower before I applied those! Long story short, I have two main issues: 1. The LSTM model would seem a perfect fit when looking at its MAPE, but when its plotted against the real values the fit doesn't seem particularly good. In fact, going by the plot, the Bidirectionl model would seem like the better fit, and still not as good as it should be. 2. The Bidirectional model has an odd issue in which its training MAPE is much higher than the validation one, and yet, when plotted it seems as if doesn't fit that badly. Basically, I'm at a loss, and I would highly appreciate any help. Thank in advance!

External answer — Data Science Stack Exchange Author: Nicolas Martin Original post: https://datascience.stackexchange.com/a/111935 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. LSTM has limits and having the raw data as input might not work in some cases, because it can't learn easily values from ranges that are too far between each other (in your case 3000 is far from 1000). That's why you can do 2 things: Normalize values (ex: minmax), so that LSTMs only learn a range between 0 and 1. Transform your data to relative values (i.e. +X points if the value increase, 0 if stable, -X otherwise). You should also take into account that LSTMs learn between 250 and 500 values and that it is quite sensitive to noise. There is a lot of noise in stock markets, so you can have even better predictions if you smooth your data.