Generalized Random Direction Newton Algorithms for Stochastic Optimization

We present a family of generalized Hessian estimators of the objective using random direction stochastic approximation (RDSA) by utilizing only noisy function measurements. The form of each estimator and the order of the bias depend on the number of function measurements. In particular, we demonstrate that estimators with more function measurements exhibit lower-order estimation bias. We show the asymptotic unbiasedness of the estimators. We also perform asymptotic and non-asymptotic convergence analyses for stochastic Newton methods that incorporate our generalized Hessian estimators. Finally, we perform numerical experiments to validate our theoretical findings.

Risk-sensitive reinforcement learning using expectiles, shortfall risk and optimized certainty equivalent risk

We propose risk-sensitive reinforcement learning algorithms catering to three families of risk measures, namely expectiles, utility-based shortfall risk and optimized certainty equivalent risk. For each risk measure, in the context of a finite horizon Markov decision process, we first derive a policy gradient theorem. Second, we propose estimators of the risk-sensitive policy gradient for each of the aforementioned risk measures, and establish $\mathcal{O}\left(1/m\right)$ mean-squared error bounds for our estimators, where $m$ is the number of trajectories. Further, under standard assumptions for policy gradient-type algorithms, we establish smoothness of the risk-sensitive objective, in turn leading to stationary convergence rate bounds for the overall risk-sensitive policy gradient algorithm that we propose. Finally, we conduct numerical experiments to validate the theoretical findings on popular RL benchmarks.

Policy Newton methods for Distortion Riskmetrics

We consider the problem of risk-sensitive control in a reinforcement learning (RL) framework. In particular, we aim to find a risk-optimal policy by maximizing the distortion riskmetric (DRM) of the discounted reward in a finite horizon Markov decision process (MDP). DRMs are a rich class of risk measures that include several well-known risk measures as special cases. We derive a policy Hessian theorem for the DRM objective using the likelihood ratio method. Using this result, we propose a natural DRM Hessian estimator from sample trajectories of the underlying MDP. Next, we present a cubic-regularized policy Newton algorithm for solving this problem in an on-policy RL setting using estimates of the DRM gradient and Hessian. Our proposed algorithm is shown to converge to an $\epsilon$-second-order stationary point ($\epsilon$-SOSP) of the DRM objective, and this guarantee ensures the escaping of saddle points. The sample complexity of our algorithms to find an $ \epsilon$-SOSP is $\mathcal{O}(\epsilon^{-3.5})$. Our experiments validate the theoretical findings. To the best of our knowledge, our is the first work to present convergence to an $\epsilon$-SOSP of a risk-sensitive objective, while existing works in the literature have either shown convergence to a first-order stationary point of a risk-sensitive objective, or a SOSP of a risk-neutral one.

Public Transit Arrival Prediction: a Seq2Seq RNN Approach

Arrival/Travel times for public transit exhibit variability on account of factors like seasonality, dwell times at bus stops, traffic signals, travel demand fluctuation etc. The developing world in particular is plagued by additional factors like lack of lane discipline, excess vehicles, diverse modes of transport and so on. This renders the bus arrival time prediction (BATP) to be a challenging problem especially in the developing world. A novel data-driven model based on recurrent neural networks (RNNs) is proposed for BATP (in real-time) in the current work. The model intelligently incorporates both spatial and temporal correlations in a unique (non-linear) fashion distinct from existing approaches. In particular, we propose a Gated Recurrent Unit (GRU) based Encoder-Decoder(ED) OR Seq2Seq RNN model (originally introduced for language translation) for BATP. The geometry of the dynamic real time BATP problem enables a nice fit with the Encoder-Decoder based RNN structure. We feed relevant additional synchronized inputs (from previous trips) at each step of the decoder (a feature classically unexplored in machine translation applications). Further motivated from accurately modelling congestion influences on travel time prediction, we additionally propose to use a bidirectional layer at the decoder (something unexplored in other time-series based ED application contexts). The effectiveness of the proposed algorithms is demonstrated on real field data collected from challenging traffic conditions. Our experiments indicate that the proposed method outperforms diverse existing state-of-art data-driven approaches proposed for the same problem.

Sequence Prediction Under Missing Data : An RNN Approach Without Imputation

Missing data scenarios are very common in ML applications in general and time-series/sequence applications are no exceptions. This paper pertains to a novel Recurrent Neural Network (RNN) based solution for sequence prediction under missing data. Our method is distinct from all existing approaches. It tries to encode the missingness patterns in the data directly without trying to impute data either before or during model building. Our encoding is lossless and achieves compression. It can be employed for both sequence classification and forecasting. We focus on forecasting here in a general context of multi-step prediction in presence of possible exogenous inputs. In particular, we propose novel variants of Encoder-Decoder (Seq2Seq) RNNs for this. The encoder here adopts the above mentioned pattern encoding, while at the decoder which has a different structure, multiple variants are feasible. We demonstrate the utility of our proposed architecture via multiple experiments on both single and multiple sequence (real) data-sets. We consider both scenarios where (i)data is naturally missing and (ii)data is synthetically masked.

Seasonal Encoder-Decoder Architecture for Forecasting

Deep learning (DL) in general and Recurrent neural networks (RNNs) in particular have seen high success levels in sequence based applications. This paper pertains to RNNs for time series modelling and forecasting. We propose a novel RNN architecture capturing (stochastic) seasonal correlations intelligently while capable of accurate multi-step forecasting. It is motivated from the well-known encoder-decoder (ED) architecture and multiplicative seasonal auto-regressive model. It incorporates multi-step (multi-target) learning even in the presence (or absence) of exogenous inputs. It can be employed on single or multiple sequence data. For the multiple sequence case, we also propose a novel greedy recursive procedure to build (one or more) predictive models across sequences when per-sequence data is less. We demonstrate via extensive experiments the utility of our proposed architecture both in single sequence and multiple sequence scenarios.