Questions on reproducibility of TimeGAN results

External question — Data Science Stack Exchange Author: TripleH Original post: https://datascience.stackexchange.com/questions/121506 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I am playing the timeGAN model, using the example code from ydata-synthetic repo. To train the model, we used synth.train(stock_data, train_steps=50000) to generate data, we used synth_data = synth.sample(len(stock_data)) One way to evaluate the synthetic data is to measure discriminative score . Ideally, high-quality synthetic data should be similar to real data. So if we build a post-hoc real/fake data classifier, the classifier should have difficulty to tell data is real or generated, thus accuracy ~ 0.5 and discriminative score ~ 0 . I have two questions: Does synth_data have the same time order as stock_data ? For example, synth_data[123] somehow is synthesized from stock_data[123] ? I doubted it since synth.sample(xxx) where xxx could be any number > real data size. Does anyone (using the timeGAN on the stock data) reproduce the discriminative score ~ 0.1 from the paper Time-series Generative Adversarial Networks ? I trained a model with 50000 iterations myself and confirmed all the results shown in the example code, except discriminative score. I used GRU or LSTM to build real data/fake data classifier, which can show high accuracy (> 0.9) such that the score > 0.4.

External answer — Data Science Stack Exchange Author: Robin van Hoorn Original post: https://datascience.stackexchange.com/a/121531 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. For my master thesis AI, I've studied the TimeGAN model in-depth, including the architecture and the used metrics. 1 : Not necessarily. Vanilla TimeGAN uses (variations of) RNNs. These nets share the weights over all different timesteps. Hence, the generation is just a learned abstraction of the time-series data. synth_data[123] does not necessarily have to be based on stock_data[123] . However, if all 123rd timesteps of your data have some specific correlation, your model might pick up on this. 2 : Apologies for inserting my own opinion, but you should not put too much focus on trying to reproduce the discriminative scores. It is possible to reproduce the discriminative scores, I did so for my master thesis. However, please note that the results of the discriminative scores are VERY unstable. For the stocks data, there is an (unspecified kind of) instability of $.02$ on a metric score of $.1$ for TimeGAN. That's a lot, no matter how you turn it. Also, note that the discriminative score is not a well-established metric in GAN research. Preferably, a metric agrees with human judgement, is consistent, and is easy to calculate. The discriminative score is non of these. You train a non-optimized neural network for 2000 epochs, with a random learning rate, with a hidden neural network size half that of the time-series features (though you have to check their non-official implementation for these details). If you train with a different learning rate, a different network size, longer (or shorter) epochs, you could get a discriminative score of $0.00$ or $0.5$ on every single dataset. You can get any model to score $0.00$ by simply putting the learning rate or number of epochs on $0$ .