How can I make my neural network learn faster?
How can I make my neural network learn faster?
Loading saved threads...
postnubilaphoebus · External communityPost link
External question — Data Science Stack Exchange
Author: postnubilaphoebus
Original post: https://datascience.stackexchange.com/questions/117858
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
I would like to train an LSTM-based variational autoencoder on a large dataset (37 million sentences). However, I have calculated that my training speed as of now is too slow (on Google Colab). I am using a GPU provided by Google called A100-SXM4-40GB, and my framework of choice is Pytorch. I am already using automatic mixed precision, which sped up my code by about x2 (
https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html
). As a reminder, here is a stock image of a variational autoencoder:
With my current training speed, I get through about
130 million training examples/sentences in 24 hours
. My vocabulary size is about 85,000, the number of parameters in the VAE is 17.1 million (mostly high because of the embedding layer), my encoder has 100 and my decoder 600 hidden neurons. My batch size is 64, and I am using the Adam optimizer. I have plotted the time usage of different parts of my code after 3000 batches:
.
Following, you can see a more detailed breakdown of my model times:
What advice can you give me to speed up my model? For instance, I also have access to a TPU, but I have never seen a clear breakdown on GPU vs TPU performance (and what role batch size plays). Can I use parallel computing, and is this possible on Colab? I am training for 5 epochs, leading to a total estimated training time of about 34 hours.
With regards to the detailed model training breakdown, please note:
I am already using pack_padded_sequence for the encoder
The decoder is an LSTM cell (less optimised). I need to pass the last encoder hidden state (after reparametrization) to the decoder at each time step, together with the previous output of the cell state for the LSTM. I believe this is following
4
EDIT: I updated my total run time (made a miscalculation)
Quote
Report
Post Reply
Quoted from Forex.com.bd-Editorial External question — Data Science Stack Exchange Author: postnubilaphoebus Source score (net votes, not local likes): 2 Original post: https://datascience.stackexchange.com/questions/117858 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I would like to train an LSTM-based variational autoencoder on a large dataset (37 million sentences). However, I have calculated that my training speed as of now is too slow (on Google Colab). I am using a GPU provided by Google called A100-SXM4-40GB, and my framework of choice is Pytorch. I am already using automatic mixed precision, which sped up my code by about x2 ( https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html ). As a reminder, here is a stock image of a variational autoencoder: With my current training speed, I get through about 130 million training examples/sentences in 24 hours . My vocabulary size is about 85,000, the number of parameters in the VAE is 17.1 million (mostly high because of the embedding layer), my encoder has 100 and my decoder 600 hidden neurons. My batch size is 64, and I am using the Adam optimizer. I have plotted the time usage of different parts of my code after 3000 batches: . Following, you can see a more detailed breakdown of my model times: What advice can you give me to speed up my model? For instance, I also have access to a TPU, but I have never seen a clear breakdown on GPU vs TPU performance (and what role batch size plays). Can I use parallel computing, and is this possible on Colab? I am training for 5 epochs, leading to a total estimated training time of about 34 hours. With regards to the detailed model training breakdown, please note: I am already using pack_padded_sequence for the encoder The decoder is an LSTM cell (less optimised). I need to pass the last encoder hidden state (after reparametrization) to the decoder at each time step, together with the previous output of the cell state for the LSTM. I believe this is following 4 EDIT: I updated my total run time (made a miscalculation)
Checking account access…