Bayesian graph convolutional neural networks via tempered MCMC

Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks, particularly those with structured data. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories. Graph convolutional neural networks have recently gained attention in the field of deep learning that takes advantage of graph-based data representation with automatic feature extraction via convolutions. Given the popularity of these methods in a wide range of applications, robust uncertainty quantification is vital. This remains a challenge for large models and unstructured datasets. Bayesian inference provides a principled and robust approach to uncertainty quantification of model parameters for deep learning models. Although Bayesian inference has been used extensively elsewhere, its application to deep learning remains limited due to the computational requirements of the Markov Chain Monte Carlo (MCMC) methods. Recent advances in parallel computing and advanced proposal schemes in sampling, such as incorporating gradients has allowed Bayesian deep learning methods to be implemented. In this paper, we present Bayesian graph deep learning techniques that employ state-of-art methods such as tempered MCMC sampling and advanced proposal schemes. Our results show that Bayesian graph convolutional methods can provide accuracy similar to advanced learning methods while providing a better alternative for robust uncertainty quantification for key benchmark problems.

Revisiting Bayesian Autoencoders with MCMC

Autoencoders gained popularity in the deep learning revolution given their ability to compress data and provide dimensionality reduction. Although prominent deep learning methods have been used to enhance autoencoders, the need to provide robust uncertainty quantification remains a challenge. This has been addressed with variational autoencoders so far. Bayesian inference via MCMC methods have faced limitations but recent advances with parallel computing and advanced proposal schemes that incorporate gradients have opened routes less travelled. In this paper, we present Bayesian autoencoders powered MCMC sampling implemented using parallel computing and Langevin gradient proposal scheme. Our proposed Bayesian autoencoder provides similar performance accuracy when compared to related methods from the literature, with the additional feature of robust uncertainty quantification in compressed datasets. This motivates further application of the Bayesian autoencoder framework for other deep learning models.

Memory Capacity of Neural Turing Machines with Matrix Representation

It is well known that recurrent neural networks (RNNs) faced limitations in learning long-term dependencies that have been addressed by memory structures in long short-term memory (LSTM) networks. Matrix neural networks feature matrix representation which inherently preserves the spatial structure of data and has the potential to provide better memory structures when compared to canonical neural networks that use vector representation. Neural Turing machines (NTMs) are novel RNNs that implement notion of programmable computers with neural network controllers to feature algorithms that have copying, sorting, and associative recall tasks. In this paper, we study the augmentation of memory capacity with a matrix representation of RNNs and NTMs (MatNTMs). We investigate if matrix representation has a better memory capacity than the vector representations in conventional neural networks. We use a probabilistic model of the memory capacity using Fisher information and investigate how the memory capacity for matrix representation networks are limited under various constraints, and in general, without any constraints. In the case of memory capacity without any constraints, we found that the upper bound on memory capacity to be $N^2$ for an $N\times N$ state matrix. The results from our experiments using synthetic algorithmic tasks show that MatNTMs have a better learning capacity when compared to its counterparts.

COVID-19 sentiment analysis via deep learning during the rise of novel cases

Social scientists and psychologists take interest in understanding how people express emotions or sentiments when dealing with catastrophic events such as natural disasters, political unrest, and terrorism. The COVID-19 pandemic is a catastrophic event that has raised a number of psychological issues such as depression given abrupt social changes and lack of employment. During the rise of COVID-19 cases with stricter lock downs, people have been expressing their sentiments in social media which can provide a deep understanding of how people physiologically react to catastrophic events. In this paper, we use deep learning based language models via long short-term memory (LSTM) recurrent neural networks for sentiment analysis on Twitter with a focus of rise of novel cases in India. We use the LSTM model with a global vector (GloVe) for word representation in building a language model. We review the sentiments expressed for selective months covering the major peak of new cases in 2020. We present a framework that focuses on multi-label sentiment classification using LSTM model and GloVe embedding, where more than one sentiment can be expressed at once. Our results show that the majority of the tweets have been positive with high levels of optimism during the rise of the COVID-19 cases in India. We find that the number of tweets significantly lowered towards the peak of new cases. We find that the optimistic and joking tweets mostly dominated the monthly tweets and there was a much lower number of negative sentiments expressed. This could imply that the majority were generally positive and some annoyed towards the way the pandemic was handled by the authorities as their peak was reached.

Evaluation of deep learning models for multi-step ahead time series prediction

Time series prediction with neural networks have been focus of much research in the past few decades. Given the recent deep learning revolution, there has been much attention in using deep learning models for time series prediction, and hence it is important to evaluate their strengths and weaknesses. In this paper, we present an evaluation study that compares the performance of deep learning models for multi-step ahead time series prediction. Our deep learning methods compromise of simple recurrent neural networks, long short term memory (LSTM) networks, bidirectional LSTM, encoder-decoder LSTM networks, and convolutional neural networks. We also provide comparison with simple neural networks use stochastic gradient descent and adaptive gradient method (Adam) for training. We focus on univariate and multi-step-ahead prediction from benchmark time series datasets and compare with results from from the literature. The results show that bidirectional and encoder-decoder LSTM provide the best performance in accuracy for the given time series problems with different properties.

A review of machine learning in processing remote sensing data for mineral exploration

As a primary step in mineral exploration, a variety of features are mapped such as lithological units, alteration types, structures, and minerals. These features are extracted to aid decision-making in targeting ore deposits. Different types of remote sensing data including satellite optical and radar, airborne, and drone-based data make it possible to overcome problems associated with mapping these important parameters on the field. The rapid increase in the volume of remote sensing data obtained from different platforms has allowed scientists to develop advanced, innovative, and powerful data processing methodologies. Machine learning methods can help in processing a wide range of remote sensing data and in determining the relationship between the reflectance continuum and features of interest. Moreover, these methods are robust in processing spectral and ground truth measurements against noise and uncertainties. In recent years, many studies have been carried out by supplementing geological surveys with remote sensing data, and this area is now considered a hotspot in geoscience research. This paper reviews the implementation and adaptation of some popular and recently established machine learning methods for remote sensing data processing and investigates their applications for exploring different ore deposits. Lastly, the challenges and future directions in this critical interdisciplinary field are discussed.

Delhi air quality prediction using LSTM deep learning models with a focus on COVID-19 lockdown

Air pollution has a wide range of implications on agriculture, economy, road accidents, and health. In this paper, we use novel deep learning methods for short-term (multi-step-ahead) air-quality prediction in selected parts of Delhi, India. Our deep learning methods comprise of long short-term memory (LSTM) network models which also include some recent versions such as bidirectional-LSTM and encoder-decoder LSTM models. We use a multivariate time series approach that attempts to predict air quality for 10 prediction horizons covering total of 80 hours and provide a long-term (one month ahead) forecast with uncertainties quantified. Our results show that the multivariate bidirectional-LSTM model provides best predictions despite COVID-19 impact on the air-quality during full and partial lockdown periods. The effect of COVID-19 on the air quality has been significant during full lockdown; however, there was unprecedented growth of poor air quality afterwards.

Deep learning via LSTM models for COVID-19 infection forecasting in India

We have entered an era of a pandemic that has shaken the world with major impact to medical systems, economics and agriculture. Prominent computational and mathematical models have been unreliable due to the complexity of the spread of infections. Moreover, lack of data collection and reporting makes any such modelling attempts unreliable. Hence we need to re-look at the situation with the latest data sources and most comprehensive forecasting models. Deep learning models such as recurrent neural networks are well suited for modelling temporal sequences. In this paper, prominent recurrent neural networks, in particular \textit{long short term memory} (LSTMs) networks, bidirectional LSTM, and encoder-decoder LSTM models for multi-step (short-term) forecasting the spread of COVID-infections among selected states in India. We select states with COVID-19 hotpots in terms of the rate of infections and compare with states where infections have been contained or reached their peak and provide two months ahead forecast that shows that cases will slowly decline. Our results show that long-term forecasts are promising which motivates the application of the method in other countries or areas. We note that although we made some progress in forecasting, the challenges in modelling remain due to data and difficulty in capturing factors such as population density, travel logistics, and social aspects such culture and lifestyle.

Surrogate-assisted Bayesian inversion for landscape and basin evolution models

The complex and computationally expensive features of the forward landscape and sedimentary basin evolution models pose a major challenge in the development of efficient inference and optimization methods. Bayesian inference provides a methodology for estimation and uncertainty quantification of free model parameters. In our previous work, parallel tempering Bayeslands was developed as a framework for parameter estimation and uncertainty quantification for the landscape and basin evolution modelling software Badlands. Parallel tempering Bayeslands features high-performance computing with dozens of processing cores running in parallel to enhance computational efficiency. Although parallel computing is used, the procedure remains computationally challenging since thousands of samples need to be drawn and evaluated. In large-scale landscape and basin evolution problems, a single model evaluation can take from several minutes to hours, and in certain cases, even days. Surrogate-assisted optimization has been with successfully applied to a number of engineering problems. This motivates its use in optimisation and inference methods suited for complex models in geology and geophysics. Surrogates can speed up parallel tempering Bayeslands by developing computationally inexpensive surrogates to mimic expensive models. In this paper, we present an application of surrogate-assisted parallel tempering where that surrogate mimics a landscape evolution model including erosion, sediment transport and deposition, by estimating the likelihood function that is given by the model. We employ a machine learning model as a surrogate that learns from the samples generated by the parallel tempering algorithm. The results show that the methodology is effective in lowering the overall computational cost significantly while retaining the quality of solutions.

Efficiency and robustness in Monte Carlo sampling of 3-D geophysical inversions with Obsidian v0.1.2: Setting up for success

The rigorous quantification of uncertainty in geophysical inversions is a challenging problem. Inversions are often ill-posed and the likelihood surface may be multi-modal; properties of any single mode become inadequate uncertainty measures, and sampling methods become inefficient for irregular posteriors or high-dimensional parameter spaces. We explore the influences of different choices made by the practitioner on the efficiency and accuracy of Bayesian geophysical inversion methods that rely on Markov chain Monte Carlo sampling to assess uncertainty, using a multi-sensor inversion of the three-dimensional structure and composition of a region in the Cooper Basin of South Australia as a case study. The inversion is performed using an updated version of the Obsidian distributed inversion software. We find that the posterior for this inversion has complex local covariance structure, hindering the efficiency of adaptive sampling methods that adjust the proposal based on the chain history. Within the context of a parallel-tempered Markov chain Monte Carlo scheme for exploring high-dimensional multi-modal posteriors, a preconditioned Crank-Nicholson proposal outperforms more conventional forms of random walk. Aspects of the problem setup, such as priors on petrophysics or on 3-D geological structure, affect the shape and separation of posterior modes, influencing sampling performance as well as the inversion results. Use of uninformative priors on sensor noise can improve inversion results by enabling optimal weighting among multiple sensors even if noise levels are uncertain. Efficiency could be further increased by using posterior gradient information within proposals, which Obsidian does not currently support, but which could be emulated using posterior surrogates.