A Stable Variational Autoencoder for Text Modelling

Variational Autoencoder (VAE) is a powerful method for learning representations of high-dimensional data. However, VAEs can suffer from an issue known as latent variable collapse (or KL loss vanishing), where the posterior collapses to the prior and the model will ignore the latent codes in generative tasks. Such an issue is particularly prevalent when employing VAE-RNN architectures for text modelling (Bowman et al., 2016). In this paper, we present a simple architecture called holistic regularisation VAE (HR-VAE), which can effectively avoid latent variable collapse. Compared to existing VAE-RNN architectures, we show that our model can achieve much more stable training process and can generate text with significantly better quality.

A Dual-Attention Hierarchical Recurrent Neural Network for Dialogue Act Classification

Recognising dialogue acts (DA) is important for many natural language processing tasks such as dialogue generation and intention recognition. In this paper, we propose a dual-attention hierarchical recurrent neural network for dialogue act classification. Our model is partially inspired by the observation that conversational utterances are normally associated with both a dialogue act and a topic, where the former captures the social act and the latter describes the subject matter. However, such a dependency between dialogue acts and topics has not been utilised by most existing systems for DA classification. With a novel dual task-specific attention mechanism, our model is able, for utterances, to capture information about both dialogue acts and topics, as well as information about the interactions between them. We evaluate the performance of our model on two publicly available datasets, i.e., Switchboard and DailyDialog. Experimental results show that by modelling topic as an auxiliary task, our model can significantly improve DA classification.