Document Type : Original Article

Authors

1 Iran University of Science and Technology (IUST), Tehran, Iran

2 Department of Statistics, Payame Noor University , Tehran, Iran

Abstract

Abstract. This paper explores the implementation and comparison of various optimization algorithms for training Variational Autoencoders (VAEs). We introduce a shrinkage-based post-hoc weight-update rule and compare it against SGD, Adam, RMSprop, Adagrad, AdamW, and Nadam, using a multi-seed evaluation protocol (5–10 random seeds per configuration) on both a small synthetic dataset and real MNIST digits, together with a per-dimension KL-divergence diagnostic for posterior collapse. At each dataset’s default configuration, the shrinkage step appears to have a large and opposite effect: on synthetic data it induces complete posterior collapse in every trained model, while on MNIST it instead avoids the partial posterior collapse exhibited by standard adaptive optimizers, at a substantial cost to reconstruction quality. However, directly testing the robustness of the MNIST finding, by sweeping the learning rate and by testing a nonlinear architecture, shows that this apparent separation between collapsing and non-collapsing optimizers is not stable: SGD collapses at higher learning rates, the adaptive methods’ collapse is a non-monotonic function of learning rate that vanishes outside a narrow band, and two of the four shrinkage variants collapse completely under a nonlinear architecture. We conclude that posterior collapse in this setting depends on an interaction between optimizer, learning rate, and architecture that single-configuration comparisons cannot capture, and we report a fully reproducible implementation, including the robustness-check scripts, as a cautionary and diagnostic contribution rather than as a general-purpose optimizer recommendation. 

Highlights

  • Compares six standard optimizers with a shrinkage-based weight-update rule for VAE training.

  • Evaluates posterior collapse using synthetic data and real MNIST digits.

  • Introduces per-dimension KL-divergence diagnostics for posterior collapse.

  • Demonstrates sensitivity to learning rate and neural network architecture.

  • Provides reproducible implementations and robustness-check scripts for VAE optimization.

Keywords

[1] Kingma, D. P., & Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114. https://doi.org/10.48550/arXiv.1312.6114

[2] Kingma, D. P., & Welling, M. (2019). An Introduction to Variational Autoencoders. Foundations and Trends in Machine Learning, 12(4):307–392. https://doi.org/10.1561/2200000056

[3] Rezende, D. J., & Mohamed, S. (2015). Variational Inference with Normalizing Flows. Proceedings of the 32nd International Conference on Machine Learning (ICML), PMLR 37:1530–1538. https://doi.org/10.48550/arXiv.1505.05770

[4] Robbins, H., & Monro, S. (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400–407. https://doi.org/10.1214/aoms/1177729586

[5] Ruder, S. (2016). An Overview of Gradient Descent Optimization Algorithms. arXiv preprint arXiv:1609.04747. https://doi.org/10.48550/arXiv.1609.04747

[6] Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the 31st International Conference on Machine Learning (ICML), PMLR 32(2):1278– 1286. https://doi.org/10.48550/arXiv.1401.4082

[7] Polyak, B. T. (1964). Some Methods of Speeding Up the Convergence of Iterative Methods. USSR Computational Mathematics and Mathematical Physics, 4(5), 1–17. https://doi.org/10.1016/0041-5553(64)90137-5

[8] Sutskever, I., Martens, J., Dahl, G. E., & Hinton, G. E. (2013). On the Importance of Initialization and Momentum in Deep Learning. Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3):1139–1147.
[9] Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.https://doi.org/10.48550/arXiv.1412.6980

[10] Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR), 1–13. https://doi.org/10.48550/arXiv.1412.6980

[11] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
[12] Tipping, M. E., & Bishop, C. M. (1999). Probabilistic Principal Component Analysis. Journal of the Royal Statistical Society: Series B, 61(3):611–622. https://doi.org/10.1111/1467-9868.00196

[13] Tieleman, T., & Hinton, G. (2012). Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.

[14] Hinton, G. E. (2012). Neural Networks for Machine Learning. Coursera.

[15] Higgins, I., Matthey, L., Pal, A., et al. (2017). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. International Conference on Learning Representations (ICLR).

[16] Doersch, C. (2016). Tutorial on Variational Autoencoders. arXiv preprint arXiv:1606.05908.https://doi.org/10.48550/arXiv.1606.05908

[17] Duchi, J. C., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12, 2121–2159.

[18] Duchi, J., & Singer, Y. (2009). Efficient Online and Batch Learning Using Forward Backward Splitting. Journal of Machine Learning Research, 10, 2899–2934.

[19] Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2016). Generating Sentences from a Continuous Space. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), 10–21. https://doi.org/10.18653/v1/K16-1002

[20] Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., & Welling, M. (2016). Improved Variational Inference with Inverse Autoregressive Flow. Advances in Neural Information Processing Systems (NeurIPS), 29:4743–751.https://doi.org/10.48550/arXiv.1606.04934

[21] Rezende, D. J., & Viola, F. (2018). Taming VAEs. arXiv preprint arXiv:1810.00597. https://doi.org/10.48550/arXiv.1810.00597

[22] Zhao, S., Song, J., & Ermon, S. (2017). InfoVAE: Information Maximizing Variational Autoencoders. arXiv preprint arXiv:1706.02262. https://doi.org/10.48550/arXiv.1706.02262

[23] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278–2324. https://doi.org/10.1109/5.726791

[24] Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR). https://doi.org/10.48550/arXiv.1711.05101

[25] Dozat, T. (2016). Incorporating Nesterov Momentum into Adam. ICLR Workshop Track.