Document Type : Original Article
Authors
1 Iran University of Science and Technology (IUST), Tehran, Iran
2 Department of Statistics, Payame Noor University , Tehran, Iran
Abstract
Abstract. This paper explores the implementation and comparison of various optimization algorithms for training Variational Autoencoders (VAEs). We introduce a shrinkage-based post-hoc weight-update rule and compare it against SGD, Adam, RMSprop, Adagrad, AdamW, and Nadam, using a multi-seed evaluation protocol (5–10 random seeds per configuration) on both a small synthetic dataset and real MNIST digits, together with a per-dimension KL-divergence diagnostic for posterior collapse. At each dataset’s default configuration, the shrinkage step appears to have a large and opposite effect: on synthetic data it induces complete posterior collapse in every trained model, while on MNIST it instead avoids the partial posterior collapse exhibited by standard adaptive optimizers, at a substantial cost to reconstruction quality. However, directly testing the robustness of the MNIST finding, by sweeping the learning rate and by testing a nonlinear architecture, shows that this apparent separation between collapsing and non-collapsing optimizers is not stable: SGD collapses at higher learning rates, the adaptive methods’ collapse is a non-monotonic function of learning rate that vanishes outside a narrow band, and two of the four shrinkage variants collapse completely under a nonlinear architecture. We conclude that posterior collapse in this setting depends on an interaction between optimizer, learning rate, and architecture that single-configuration comparisons cannot capture, and we report a fully reproducible implementation, including the robustness-check scripts, as a cautionary and diagnostic contribution rather than as a general-purpose optimizer recommendation.
Highlights
-
Compares six standard optimizers with a shrinkage-based weight-update rule for VAE training.
-
Evaluates posterior collapse using synthetic data and real MNIST digits.
-
Introduces per-dimension KL-divergence diagnostics for posterior collapse.
-
Demonstrates sensitivity to learning rate and neural network architecture.
-
Provides reproducible implementations and robustness-check scripts for VAE optimization.
Keywords
[2] Kingma, D. P., & Welling, M. (2019). An Introduction to Variational Autoencoders. Foundations and Trends in Machine Learning, 12(4):307–392. https://doi.org/10.1561/2200000056
[3] Rezende, D. J., & Mohamed, S. (2015). Variational Inference with Normalizing Flows. Proceedings of the 32nd International Conference on Machine Learning (ICML), PMLR 37:1530–1538. https://doi.org/10.48550/arXiv.1505.05770
[4] Robbins, H., & Monro, S. (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400–407. https://doi.org/10.1214/aoms/1177729586
[5] Ruder, S. (2016). An Overview of Gradient Descent Optimization Algorithms. arXiv preprint arXiv:1609.04747. https://doi.org/10.48550/arXiv.1609.04747
[6] Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the 31st International Conference on Machine Learning (ICML), PMLR 32(2):1278– 1286. https://doi.org/10.48550/arXiv.1401.4082
[7] Polyak, B. T. (1964). Some Methods of Speeding Up the Convergence of Iterative Methods. USSR Computational Mathematics and Mathematical Physics, 4(5), 1–17. https://doi.org/10.1016/0041-5553(64)90137-5
[8] Sutskever, I., Martens, J., Dahl, G. E., & Hinton, G. E. (2013). On the Importance of Initialization and Momentum in Deep Learning. Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3):1139–1147.
[10] Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR), 1–13. https://doi.org/10.48550/arXiv.1412.6980
[11] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
[13] Tieleman, T., & Hinton, G. (2012). Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.
[14] Hinton, G. E. (2012). Neural Networks for Machine Learning. Coursera.
[15] Higgins, I., Matthey, L., Pal, A., et al. (2017). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. International Conference on Learning Representations (ICLR).
[16] Doersch, C. (2016). Tutorial on Variational Autoencoders. arXiv preprint arXiv:1606.05908.https://doi.org/10.48550/arXiv.1606.05908
[17] Duchi, J. C., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12, 2121–2159.
[18] Duchi, J., & Singer, Y. (2009). Efficient Online and Batch Learning Using Forward Backward Splitting. Journal of Machine Learning Research, 10, 2899–2934.
[19] Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2016). Generating Sentences from a Continuous Space. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), 10–21. https://doi.org/10.18653/v1/K16-1002
[20] Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., & Welling, M. (2016). Improved Variational Inference with Inverse Autoregressive Flow. Advances in Neural Information Processing Systems (NeurIPS), 29:4743–751.https://doi.org/10.48550/arXiv.1606.04934
[21] Rezende, D. J., & Viola, F. (2018). Taming VAEs. arXiv preprint arXiv:1810.00597. https://doi.org/10.48550/arXiv.1810.00597
[22] Zhao, S., Song, J., & Ermon, S. (2017). InfoVAE: Information Maximizing Variational Autoencoders. arXiv preprint arXiv:1706.02262. https://doi.org/10.48550/arXiv.1706.02262
[23] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278–2324. https://doi.org/10.1109/5.726791
[24] Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR). https://doi.org/10.48550/arXiv.1711.05101
[25] Dozat, T. (2016). Incorporating Nesterov Momentum into Adam. ICLR Workshop Track.