Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis

Reinforcement Learning (RL) has shown exceptional performance across various applications, enabling autonomous agents to learn optimal policies through interaction with their environments. However, traditional RL frameworks often face challenges in terms of iteration complexity and robustness. Risk-sensitive RL, which balances expected return and risk, has been explored for its potential to yield probabilistically robust policies, yet its iteration complexity analysis remains underexplored. In this study, we conduct a thorough iteration complexity analysis for the risk-sensitive policy gradient method, focusing on the REINFORCE algorithm and employing the exponential utility function. We obtain an iteration complexity of $\mathcal{O}(\epsilon^{-2})$ to reach an $\epsilon$-approximate first-order stationary point (FOSP). We investigate whether risk-sensitive algorithms can achieve better iteration complexity compared to their risk-neutral counterparts. Our theoretical analysis demonstrates that risk-sensitive REINFORCE can have a reduced number of iterations required for convergence. This leads to improved iteration complexity, as employing the exponential utility does not entail additional computation per iteration. We characterize the conditions under which risk-sensitive algorithms can achieve better iteration complexity. Our simulation results also validate that risk-averse cases can converge and stabilize more quickly after approximately half of the episodes compared to their risk-neutral counterparts.

Cooperative Bidirectional Mixed-Traffic Overtaking

Safe overtaking, especially in a bidirectional mixed-traffic setting, remains a key challenge for Connected Autonomous Vehicles (CAVs). The presence of human-driven vehicles (HDVs), behavior unpredictability, and blind spots resulting from sensor occlusion make this a challenging control problem. To overcome these difficulties, we propose a cooperative communication-based approach that utilizes the information shared between CAVs to reduce the effects of sensor occlusion while benefiting from the local velocity prediction based on past tracking data. Our control framework aims to perform overtaking maneuvers with the objective of maximizing velocity while prioritizing safety and passenger comfort. Our method is also capable of reactively adjusting its plan to dynamic changes in the environment. The performance of the proposed approach is verified using realistic traffic simulations.

Safe Collective Control under Noisy Inputs and Competing Constraints via Non-Smooth Barrier Functions

We consider the problem of safely coordinating ensembles of identical autonomous agents to conduct complex missions with conflicting safety requirements and under noisy control inputs. Using non-smooth control barrier functions (CBFs) and stochastic model-predictive control as springboards and by adopting an extrinsic approach where the ensemble is treated as a unified dynamic entity, we devise a method to synthesize safety-aware control inputs for uncertain collectives, drawing upon recent developments in Boolean CBF composition and extensions of CBFs to stochastic systems. Specifically, we approximate the combined CBF by a smooth function and solve a stochastic optimization problem, with agent-level forcing terms restricted to the resulting affine subspace of safe control inputs. For the smoothing step, we employ a polynomial approximation scheme, providing evidence for its advantage in generating more conservative yet sufficiently-filtered control signals than the smoother but more aggressive equivalents realized via an approximation technique based on the log-sum-exp function. To further demonstrate the utility of the proposed method, we present bounds for the expected value of the CBF approximation error, along with results from simulations of a single-integrator collective under velocity perturbations, comparing these results with those obtained using a naive state-feedback controller lacking safety filters.

RCMS: Risk-Aware Crash Mitigation System for Autonomous Vehicles

We propose a risk-aware crash mitigation system (RCMS), to augment any existing motion planner (MP), that enables an autonomous vehicle to perform evasive maneuvers in high-risk situations and minimize the severity of collision if a crash is inevitable. In order to facilitate a smooth transition between RCMS and MP, we develop a novel activation mechanism that combines instantaneous as well as predictive collision risk evaluation strategies in a unified hysteresis-band approach. For trajectory planning, we deploy a modular receding horizon optimization-based approach that minimizes a smooth situational risk profile, while adhering to the physical road limits as well as vehicular actuator limits. We demonstrate the performance of our approach in a simulation environment.

Blind Cyclic Prefix-based CFO Estimation in MIMO-OFDM Systems

Low-complexity estimation and correction of carrier frequency offset (CFO) are essential in orthogonal frequency division multiplexing (OFDM). In this paper, we propose a low-overhead blind CFO estimation technique based on cyclic prefix (CP), in multi-input multi-output (MIMO)-OFDM systems. We propose to use antenna diversity for CFO estimation. Given that the RF chains for all antenna elements at a communication node share the same clock, the carrier frequency offset (CFO) between two points may be estimated by using the combination of the received signal at all antennas. We improve our method by combining the antenna diversity with time diversity by considering the CP for multiple OFDM symbols. We provide a closed-form expression for CFO estimation and present algorithms that can considerably improve the CFO estimation performance at the expense of a linear increase in computational complexity. We validate the effectiveness of our estimation scheme via extensive numerical analysis.

SLAS: Speed and Lane Advisory System for Highway Navigation

This paper proposes a hierarchical autonomous vehicle navigation architecture, composed of a high-level speed and lane advisory system (SLAS) coupled with low-level trajectory generation and trajectory following modules. Specifically, we target a multi-lane highway driving scenario where an autonomous ego vehicle navigates in traffic. We propose a novel receding horizon mixed-integer optimization based method for SLAS with the objective to minimize travel time while accounting for passenger comfort. We further incorporate various modifications in the proposed approach to improve the overall computational efficiency and achieve real-time performance. We demonstrate the efficacy of the proposed approach in contrast to the existing methods, when applied in conjunction with state-of-the-art trajectory generation and trajectory following frameworks, in a CARLA simulation environment.

GAMEOPT: Optimal Real-time Multi-Agent Planning and Control for Dynamic Intersections

We propose GameOpt: a novel hybrid approach to cooperative intersection control for dynamic, multi-lane, unsignalized intersections. Safely navigating these complex and accident prone intersections requires simultaneous trajectory planning and negotiation among drivers. GameOpt is a hybrid formulation that first uses an auction mechanism to generate a priority entrance sequence for every agent, followed by an optimization-based trajectory planner that computes velocity controls that satisfy the priority sequence. This coupling operates at real-time speeds of less than 10 milliseconds in high density traffic of more than 10,000 vehicles/hr, 100 times faster than other fully optimization-based methods, while providing guarantees in terms of fairness, safety, and efficiency. Tested on the SUMO simulator, our algorithm improves throughput by at least 25%, time taken to reach the goal by 75%, and fuel consumption by 33% compared to auction-based approaches and signaled approaches using traffic-lights and stop signs.

GAMEOPT: Optimal Real-time Multi-Agent Planning and Control at Dynamic Intersections

We propose GameOpt: a novel hybrid approach to cooperative intersection control for dynamic, multi-lane, unsignalized intersections. Safely navigating these complex and accident prone intersections requires simultaneous trajectory planning and negotiation among drivers. GameOpt is a hybrid formulation that first uses an auction mechanism to generate a priority entrance sequence for every agent, followed by an optimization-based trajectory planner that computes velocity controls that satisfy the priority sequence. This coupling operates at real-time speeds of less than $10$ milliseconds in high density traffic of more than $10,000$ vehicles/hr, $100\times$ faster than other fully optimization-based methods, while providing guarantees in terms of fairness, safety, and efficiency. Tested on the SUMO simulator, our algorithm improves throughput by at least $25\%$, time taken to reach the goal by $75\%$, and fuel consumption by $33\%$ compared to auction-based approaches and signaled approaches using traffic-lights and stop signs.

Multi-user Beam Alignment in Presence of Multi-path

To overcome the high path-loss and the intense shadowing in millimeter-wave (mmWave) communications, effective beamforming schemes are required which incorporate narrow beams with high beamforming gains. The mmWave channel consists of a few spatial clusters each associated with an angle of departure (AoD). The narrow beams must be aligned with the channel AoDs to increase the beamforming gain. This is achieved through a procedure called beam alignment (BA). Most of the BA schemes in the literature consider channels with a single dominant path while in practice the channel has a few resolvable paths with different AoDs, hence, such BA schemes may not work correctly in the presence of multi-path or at the least do not exploit such multipath to achieve diversity or increase robustness. In this paper, we propose an efficient BA scheme in presence of multi-path. The proposed BA scheme transmits probing packets using a set of scanning beams and receives feedback for all the scanning beams at the end of the probing phase from each user. We formulate the BA scheme as minimizing the expected value of the average transmission beamwidth under different policies. The policy is defined as a function from the set of received feedback to the set of transmission beams (TB). In order to maximize the number of possible feedback sequences, we prove that the set of scanning beams (SB) has a special form, namely, Tulip Design. Consequently, we rewrite the minimization problem with a set of linear constraints and a reduced number of variables which is solved by using an efficient greedy algorithm.