Применение глубоких генеративных моделей для задач прогнозирования в машинном обучении тема диссертации и автореферата по ВАК РФ 00.00.00, кандидат наук Баранчук Дмитрий Александрович

• Баранчук Дмитрий Александрович
• кандидат наук
• 2024

Оглавление диссертации кандидат наук Баранчук Дмитрий Александрович

1 Introduction 4 Topic of the thesis 4 Relevance

2 Main Results and Conclusions

3 Content of the work

3.1 GP-VAE: Deep Probabilistic Time Series Imputation

3.2 Label-efficient Semantic Segmentation with Diffusion Models

3.3 TabDDPM: Modelling Tabular Data with Diffusion Models

4 Conclusion 22 References 23 A Article. GP-VAE: Deep Probabilistic Time Series Imputation 29 B Article. Label-efficient Semantic Segmentation with Diffusion Models 40 C Article. TabDDPM: Modelling Tabular Data with Diffusion Models 52 D Russian Translation of the Ph.D. dissertation

Введение диссертации (часть автореферата) на тему «Применение глубоких генеративных моделей для задач прогнозирования в машинном обучении»

1 Introduction

Topic of the thesis

For the past decade, deep neural networks have continuously grown in capability and capacity and excelled in various machine learning tasks, such as language processing, image recognition, speech synthesis, video generation and others. The deep learning methods can be grouped into two main classes: discriminative and generative approaches.

Discriminative models aim to answer specific questions about the data objects. For example, determine what is depicted in a picture, count the number of people on CCTV snapshots, and suggest an effective treatment for a patient given their measurements. More formally, the discriminative methods model the conditional distribution p(y |x) given the observed pairs (x, y), where x is an input object and y is a target label. Neural networks have rapidly demonstrated remarkable performance in a wide range of predictive tasks due to the emergence of large labeled datasets and the development of specialized hardware, e.g., graphics processing units (GPU). However, there are still many practical challenges in discriminative problems. For example, the data objects can have missing observations that could be informative for more accurate model predictions. Sometimes, collecting a large labeled dataset can be challenging and costly and hence, one requires the top-performing methods that have access only to few labeled samples during training. Also, in some areas and applications, data may be subject to the General Data Protection Regulation (GDPR) and contain private or sensitive user data. This problem might limit the use and collection of such data for developing machine learning methods.

ontrary to discriminative modeling, the fundamental goal of generative models is to approximate the data distribution pdata given the finite set of observed objects D = {x0,...,xN} from this distribution. Deep generative methods approximate pdata using a deep neural network with parameters 9. The parameters are learned to minimize the distance between the model distribution pg and pdata:

9*=min d(pdata,pg). The distance d(•, •) may be an arbitrary similarity measure between distributions,

g

e.g., KL divergence. An illustrative example of the generative problem: given Vincent van Gogh's paintings, learn the model 9 to draw the new paintings in the same style. Compared to the similar predictive problem "Who is the author of the painting?", one can correctly conclude that generative tasks are usually significantly more sophisticated than discriminative ones.

There exist many classes of deep generative models, and they can be grouped into two major categories: likelihood-based models and implicit generative models. Likelihood-based models explicitly learn pg via maximizing the likelihood directly or its lower bound. The examples of likelihood-based methods include autoregressive models [1], diffusion models [2, 3], normalizing flows [4, 5, 6], variational autoencoders [7]. On the other hand, implicit models do not have direct access to the density function but can still produce plausible samples from the target distribution. The prominent representative is generative adversarial networks (GANs) [8]. Each class of generative models has its strengths and weaknesses. For this reason,

different kinds of generative models can be preferable in different practical applications and domains. We refer the reader to the comprehensive overviews of the existing generative models [9, 10, 11] for details.

Deep generative modeling has been thoroughly investigated in recent years and achieved impressive results in various areas. Today, people can generate highly realistic images based on text descriptions [12, 13], produce advertising videos [14] and chat with "intelligent" systems like GPT4 [15]. These successes raise the question of whether so powerful deep generative models can complement established discriminative solutions. Many research works have already provided an affirmative answer to this question in different areas [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]. This thesis extends this line of works and considers deep generative models for the following practical applications: i) missing data imputation in time series to improve the performance of classification and regression methods; ii) image semantic segmentation when the amount of labeled data is scarce; iii) tabular data generation to design high-quality and private synthetic datasets for downstream tabular tasks.

Relevance

The thesis addresses the applications of deep generative models for three different fundamental machine learning problems. Below, we briefly discuss each of them in more detail.

The first work focuses on the time series imputation problem. The task aims to fill in missing observations in real-world time series data. Multivariate time series with missing values are prevalent in areas such as healthcare and finance and have increased in number and complexity over the past years. Missing values often occur due to faulty measurement devices, costly procedures, and human mistakes. As a result, the data can lack some informative features, causing machine learning methods to make incorrect predictions. Recent research has shown that accurate time series imputation significantly enhances the performance on downstream tasks [27, 28, 29].

Popular deep learning imputation approaches usually apply recurrent neural networks (RNNs) for sequence modeling [27, 30, 31, 29]. Other works combine RNNs with an adversarial objective [32, 28, 33] to improve the imputation performance. In this thesis, we make the first attempt to use deep probabilistic generative models for time series imputation. Specifically, we propose a variational autoencoder (VAE) with Gaussian process (GP) prior and demonstrate its effectiveness on the image and healthcare datasets with a temporal component. The follow-up work [20] proposes the probabilistic imputation method based on diffusion models and further enhances the imputation performance. Moreover, the appealing property of probabilistic imputation methods is that they can provide uncertainty estimations for the predicted values. This property is crucial for interpretive estimates and the trustworthiness of the method, especially if one aims to integrate it into medical applications.

The second work investigates generative models in the context of image semantic segmentation. Semantic segmentation is a fundamental computer vision problem that aims to recognize elements in an image at the pixel level. Opposed to image classification, where the model typically predicts a single label for an image, semantic segmentation seeks to assign each pixel to the class label. This makes semantic segmentation a highly challenging problem that would benefit from large labeled datasets. However, the

accurate and consistent annotation of many images requires tremendous human effort and cost. For this reason, the methods that can provide strong segmentation performance given only few labeled images are in high demand [18, 19, 34].

Deep generative models have already been applied for semantic segmentation. Most methods leverage state-of-the-art GANs [35] as infinite generators of synthetic labeled data. This data is then used to train the semantic segmentation models. Some methods [36, 37, 38] exploit the evidence that the latent space of the GANs contains a direction that allows producing synthetic images along with foreground/background segmentation masks. Other works [18, 19] exploit intermediate pixel-level representations of GANs to predict segmentation masks for generated images. These methods demonstrate promising results in the setting when there is a limited number of human-annotated images.

Diffusion probabilistic models (DPMs) demonstrate state-of-the-art image generation in terms of both image quality and diversity [39, 12, 13]. The advantages of DPM are successfully exploited in generative tasks such as image colorization [40], inpainting [40], super-resolution [41, 42], and semantic editing [43], where DPMs often achieve more impressive results than GANs. However, it has not been explored whether DPMs can be effectively applied to discriminative vision problems. We have investigated intermediate representations of DPMs and revealed that they contain the pixel-level semantic information of the input image. Following [18], we propose a novel semantic segmentation method that exploits these image representations. We demonstrate its superiority over GAN-based and self-supervised approaches in the label-efficient setting.

Finally, we extend the framework of diffusion probabilistic models to the tabular domain. Tabular datasets are usually isolated and limited in size, as opposed to textual or image data that is massively available on the Internet. Often, tabular data contains personal, private or sensitive information and hence cannot be publicly shared without violating GDPR-like regulations. Deep generative models in the tabular domain are mainly used to mitigate this problem by replacing real user data with synthetic data. At the same time, the synthetic dataset has to inherit the properties of the real distribution to be useful for downstream applications. The recent works have developed many generative modeling methods, including tabular VAEs [44] and GAN-based approaches [44, 22, 23, 24, 25, 26, 45, 46, 47, 48] Motivated by the success of diffusion models in other domains, we introduce TabDDPM — a diffusion model that can be applied to arbitrary tabular datasets and handles various feature distributions. We extensively evaluate TabDDPM on a wide set of benchmarks and demonstrate its superiority over existing GAN/VAE alternatives.

2 Main Results and Conclusions

Contribution. The main results of the work are formulated below.

1. We propose a novel probabilistic model: a variational autoencoder with a Gaussian process prior in the latent space for effective time series data modeling. The designed model is applied for the time series imputation task. We demonstrate that our approach outperforms several classical and deep learning-based data imputation methods on multivariate time series from the computer vision and healthcare domains. In addition, the method improves the smoothness of the imputations and provides interpretable uncertainty estimates.

2. We reveal that the state-of-the-art diffusion models have meaningful pixel-level image representations. Based on this knowledge, we propose a novel semantic segmentation approach that outperforms previous state-of-the-art generative and self-supervised methods when few annotated images are available.

3. We propose TabDDPM — a diffusion model for tabular data generation. This model outperforms other generative models for this task and can be useful for practitioners to replace private and sensitive data with generated data. This potentially takes a step toward the safe sharing of a company's internal data to develop high-quality prediction methods.

Theoretical and practical significance. The proposed methods and empirical findings contribute to the increasing prevalence of generative models for predictive tasks in machine learning. In scenarios characterized by a scarcity of labeled data, we demonstrate that a pretrained diffusion model can serve either as an effective data engine or as a strong discriminative model out of the box. For missing data imputation, we provide evidence that deep probabilistic modeling is a promising paradigm in healthcare applications, where it can recover missing patient measurements in an interpretable manner. Moreover, the thesis introduces a novel state-of-the-art approach for tabular data synthesis, enabling the training of highly effective machine learning methods in privacy-concerned scenarios.

Key aspects/ideas to be defended:

1. A deep probabilistic time series imputation method based on a variational autoencoder that uses a Gaussian process prior for better time series modeling;

2. Investigation of the internal representations of diffusion models, revealing the presence of useful finegrained semantic information about input images. A semantic segmentation method that effectively utilizes the image representations extracted from pretrained diffusion models when labeled data is limited;

3. A diffusion-based generative approach for tabular data modeling.

Personal contribution. In the first work, the author was responsible for the technical contribution of the paper: developing the method and conducting most experiments and analysis. In the second work,

the author proposed the core scientific ideas, collected the datasets, implemented the method, conducted most experiments and analysis and wrote the text. In the third work, the author formulated the key ideas, organized the research project, designed the experiment pipelines, and contributed to writing the paper.

Publications and probation of the work Top-tier publications

1. Vincent Fortuin*, Dmitry Baranchuk*, Gunnar Ratsch, Stephan Mandt GP-VAE: Deep probabilistic time series imputation. International Conference on Artificial Intelligence and Statistics, 2020 (AISTATS 2020). CORE A conference;

2. Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, Artem Babenko Label-Efficient Semantic Segmentation with Diffusion Models. International Conference on Learning Representations, 2022 (ICLR 2022). CORE A* conference;

3. Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, Artem Babenko TabDDPM: Modelling Tabular Data with Diffusion Models. International Conference on Machine Learning, 2023 (ICML 2023). CORE A* conference.

Reports at seminars

1. Seminar of the research group in Biomedical Informatics at ETH Zurich, Zurich, August 20, 2019. Topic: "Variational Autoencoders with Gaussian Process Priors for Time Series Modeling";

2. Christmas Colloquium on Computer Vision. Moscow, December 27, 2021. Topic: "Label-Efficient Semantic Segmentation with Diffusion Models";

3. Yandex Research Seminar, Moscow, July 24, 2022. Topic: "Applications of Diffusion Probabilistic Models in Practical Machine Learning Problems".

Volume and structure of the work. The thesis contains an introduction, the content of publications, a conclusion and includes the text of publications. The total volume of the thesis is 61 pages.

Заключение диссертации по теме «Другие cпециальности», Баранчук Дмитрий Александрович

Заключение

В заключительном разделе мы суммаризируем основные результаты, предложенные в диссертации:

1. Разработан подход для заполнения пропущенных значений в многомерных временных рядах с использованием новой глубокой вероятностной модели, GP-VAE, которая сочетает преимущества вариационных автоавтокодировщиков и гауссовских процессов. Модель переводит входные данные с пропусками в латентное пространство, где каждое измерение известно. Затем временные зависимости в латентном пространстве моделируются с помощью гауссовского процесса. В экспериментах показано, что предложенная модель заполняет пропуски в данных лучше чем ранее предложенные методы, и тем самым удается повысить эффективность методов прогнозирования для данных с высокой долей пропусков.

2. Проведено исследование, которое показывает, что предварительно обученные диффузионные модели могут быть эффективно использованы для извлечения семантических представлений из картинок для распознавания изображений, например для задачи семантической сегментации. Этот подход имеет ряд преимуществ по сравнению с GAN-ами: (1) более высокое качество диффузионных моделей транслируется в более информативные признаки, и (2) диффузионные модели предоставляют возможность извлекать признаки из реальных изображений напрямую. Эти преимущества позволяют диффузионным моделям обеспечивать наилучшие результаты в задаче семантической сегментации, когда есть только несколько размеченных примеров, и превосходить многие современные подходы обучения без учителя.

3. Изучен потенциал диффузионных моделей для моделирования табличных данных и предложен новый метод, TabDDPM, который генерирует данные с различными типами признаков одновременно: числовыми, порядковыми или категориальными. Показано, что TabDDPM генерирует значительно более реалистичные табличные данные, чем предыдущие подходы, основанные на GAN и VAE. В результате полученные синтетические данные могут быть использованы для обучения методов прогнозирования для задач классификации и регрессии в условиях, когда важна приватность пользовательских данных.

Список литературы диссертационного исследования кандидат наук Баранчук Дмитрий Александрович, 2024 год

Список литературы

[1] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1), 2016.

[2] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.

[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. 2020.

[4] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/ 69386f6bb1dfed68692a24c8686939b9-Paper.pdf.

[5] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. CoRR, abs/1410.8516, 2015.

[6] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research. PMLR, 07-09 Jul 2015.

[7] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.

[8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014. URL http://papers.nips.cc/paper/ 5423-generative-adversarial-nets.pdf.

[9] Staphord Bengesi, Hoda El-Sayed, Md Kamruzzaman Sarker, Yao Houkpati, John Irungu, and Timothy Oladunni. Advancements in generative ai: A comprehensive review of gans, gpt, autoencoders, diffusion model, and transformers, 2023.

[10] Harshvardhan GM, Mahendra Kumar Gourisaria, Manjusha Pandey, and Siddharth Swarup Rautaray. A comprehensive survey and analysis of generative models in machine learning. Computer Science Review, 38:100285, 2020.

[11] Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z. Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering, 2024.

[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.

[13] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=08Yk-n5l2Al.

[14] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/ video-generation-models-as-world-simulators.

[15] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.

[16] Anonymous. Synthetic data from diffusion models improves imagenet classification. Submitted to Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id= DlRsoxjyPm. Under review.

[17] Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023. URL https: //openreview.net/forum?id=laWYA-LXlNb.

[18] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021.

[19] Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn. Repurposing gans for one-shot semantic part segmentation. In CVPR, 2021.

[20] YUSUKE TASHIRO, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.

[21] Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. Generative hierarchical features from synthesizing images. In CVPR, 2021.

[22] Justin Engelmann and Stefan Lessmann. Conditional wasserstein gan-based oversampling of tabular data for imbalanced learning. Expert Systems with Applications, 174:114582, 2021.

[23] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference on learning representations, 2018.

[24] Ju Fan, Tongyu Liu, Guoliang Li, Junyou Chen, Yuwei Shen, and Xiaoyong Du. Relational data synthesis using generative adversarial networks: A design space exploration. arXiv preprint arXiv:2008.12763, 2020.

[25] Amirsina Torfi, Edward A Fox, and Chandan K Reddy. Differentially private synthetic medical data generation using convolutional gans. Information Sciences, 586:485-500, 2022.

[26] Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y Chen. Ctab-gan: Effective table data synthesizing. In Asian Conference on Machine Learning, pages 97-112. PMLR, 2021.

[27] Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pages 67756785, 2018.

[28] Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. E2gan: End-to-end generative adversarial network for multivariate time series imputation. IJCAI'19, page 3094-3100. AAAI Press, 2019. ISBN 9780999241141.

[29] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021. URL https:// openreview.net/forum?id=4c0J6lwQ4_.

[30] Jinsung Yoon, William R. Zame, and Mihaela van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 2019.

[31] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values, 2017.

[32] Steven Cheng-Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. International Conference on Learning Representations, 2019.

[33] Chao Ma, Sebastian Tschiatschek, Konstantina Palla, Jose Miguel Hernandez Lobato, Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. International Conference on Machine Learning, 2018.

[34] Nico Catalano and Matteo Matteucci. Few shot semantic segmentation: a review of methodologies and open challenges, 2023.

[35] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.

[36] Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentation without labels with large-scale generative models. ICML, 2021.

[37] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In ICML, 2020.

[38] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding an unsupervised image segmenter in each of your deep generative models. arXiv preprint arXiv:2105.08127, 2021.

[39] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. 2021.

[40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. 2021.

[41] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. 2021.

[42] Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. 2021.

[43] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. 2021.

[44] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, 32, 2019.

[45] Jayoung Kim, Jinsung Jeon, Jaehoon Lee, Jihyeon Hyeong, and Noseong Park. Oct-gan: Neural ode-based conditional tabular gans. In Proceedings of the Web Conference 2021, pages 1506-1515, 2021.

[46] Yishuo Zhang, Nayyar A Zaidi, Jiahui Zhou, and Gang Li. Ganblr: a tabular data generation model. In 2021 IEEE International Conference on Data Mining (ICDM), 2021.

[47] Richard Nock and Mathieu Guillame-Bert. Generative trees: Adversarial and copycat. ICML, 2022.

[48] Bingyang Wen, Yupeng Cao, Fan Yang, Koduvayur Subbalakshmi, and Rajarathnam Chandramouli. Causal-tgan: Modeling tabular data using causally-aware gan. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.

[49] Patrick Jahnichen, Florian Wenzel, Marius Kloft, and Stephan Mandt. Scalable generalized dynamic topic models. Conference on Artificial Intelligence and Statistics, 2018.

[50] Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233, 1999.

[51] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-877, 2017.

[52] Cheng Zhang, Judith Butepage, Hedvig Kjellstrom, and Stephan Mandt. Advances in variational inference. IEEE transactions on pattern analysis and machine intelligence, 2018.

[53] Martin J Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 2008.

[54] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International Conference on Machine Learning,

2014.

[55] Y Huang and WF McColl. Analytical inversion of general tridiagonal matrices. Journal of Physics A: Mathematical and General, 30(22):7919, 1997.

[56] Ranjan K Mallik. The inverse of a tridiagonal matrix. Linear Algebra and its Applications, 325(1-3): 109-139, 2001.

[57] Robert Bamler and Stephan Mandt. Structured black box variational inference for latent time series models. arXiv preprint arXiv:1707.01069, 2017.

[58] Roderick JA Little and Donald B Rubin. Single imputation methods. Statistical analysis with missing data, pages 59-74, 2002.

[59] Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. arXiv preprint arXiv:1807.03653, 2018.

[60] Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. arXiv preprint arXiv:1511.05121, 2015.

[61] Yann LeCun, Leon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.

[62] Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. International Conference on Machine Learning, 2018.

[63] Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology, pages 245-248. IEEE, 2012.

[64] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pages 63-71. Springer, 2003.

[65] Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems, pages 15961607, 2018.

[66] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.

[67] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365,

2015.

[68] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377, 2021.

[69] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020.

[70] Rewon Child. Very deep {vae}s generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.

[71] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.

[72] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.

[73] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-18943, 2021.

[74] Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y Chen. Ctab-gan+: Enhancing tabular data synthesis. arXiv preprint arXiv:2204.00401, 2022.

[75] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.