Leveraging Digital Cousins for Ensemble Q-Learning in Large-Scale Wireless Networks

Optimizing large-scale wireless networks, including optimal resource management, power allocation, and throughput maximization, is inherently challenging due to their non-observable system dynamics and heterogeneous and complex nature. Herein, a novel ensemble Q-learning algorithm that addresses the performance and complexity challenges of the traditional Q-learning algorithm for optimizing wireless networks is presented. Ensemble learning with synthetic Markov Decision Processes is tailored to wireless networks via new models for approximating large state-space observable wireless networks. In particular, digital cousins are proposed as an extension of the traditional digital twin concept wherein multiple Q-learning algorithms on multiple synthetic Markovian environments are run in parallel and their outputs are fused into a single Q-function. Convergence analyses of key statistics and Q-functions and derivations of upper bounds on the estimation bias and variance are provided. Numerical results across a variety of real-world wireless networks show that the proposed algorithm can achieve up to 50% less average policy error with up to 40% less runtime complexity than the state-of-the-art reinforcement learning algorithms. It is also shown that theoretical results properly predict trends in the experimental results.

Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization

Reinforcement learning (RL) is a classical tool to solve network control or policy optimization problems in unknown environments. The original Q-learning suffers from performance and complexity challenges across very large networks. Herein, a novel model-free ensemble reinforcement learning algorithm which adapts the classical Q-learning is proposed to handle these challenges for networks which admit Markov decision process (MDP) models. Multiple Q-learning algorithms are run on multiple, distinct, synthetically created and structurally related Markovian environments in parallel; the outputs are fused using an adaptive weighting mechanism based on the Jensen-Shannon divergence (JSD) to obtain an approximately optimal policy with low complexity. The theoretical justification of the algorithm, including the convergence of key statistics and Q-functions are provided. Numerical results across several network models show that the proposed algorithm can achieve up to 55% less average policy error with up to 50% less runtime complexity than the state-of-the-art Q-learning algorithms. Numerical results validate assumptions made in the theoretical analysis.

Optimized Parameter Design for Channel State Information-Free Location Spoofing

In this paper, an augmented analysis of a delay-angle information spoofing (DAIS) is provided for location-privacy preservation, where the location-relevant delays and angles are artificially shifted to obfuscate the eavesdropper with an incorrect physical location. A simplified mismatched Cramer-Rao bound (MCRB) is derived, which clearly manifests that not only estimation error, but also the geometric mismatch introduced by DAIS can lead to a significant increase in localization error for an eavesdropper. Given an assumption of the orthogonality among wireless paths, the simplified MCRB can be further expressed as a function of delay-angle shifts in a closed-form, which enables the more straightforward optimization of these design parameters for location-privacy enhancement. Numerical results are provided, validating the theoretical analysis and showing that the root-mean-square error for eavesdropper's localization can be more than 150 m with the optimized delay-angle shifts for DAIS.

Channel State Information-Free Location-Privacy Enhancement: Delay-Angle Information Spoofing

In this paper, a delay-angle information spoofing (DAIS) strategy is proposed for location-privacy enhancement. By shifting the location-relevant delays and angles without the aid of channel state information (CSI) at the transmitter, the eavesdropper is obfuscated by a physical location that is distinct from the true one. A precoder is designed to preserve location-privacy while the legitimate localizer can remove the obfuscation with the securely shared information. Then, a lower bound on the localization error is derived via the analysis of the geometric mismatch caused by DAIS, validating the enhanced location-privacy. The statistical hardness for the estimation of the shared information is also investigated to assess the robustness to the potential leakage of the designed precoder structure. Numerical comparisons show that the proposed DAIS scheme results in more than 15 dB performance degradation for the illegitimate localizer at high signal-to-noise ratios, which is comparable to a recently proposed CSI-free location-privacy enhancement strategy and is less sensitive to the precoder structure leakage than the prior approach.

Guaranteed Private Communication with Secret Block Structure

A novel private communication framework is proposed where privacy is induced by transmitting over channel instances of linear inverse problems that are identifiable to the legitimate receiver, but unidentifiable to an eavesdropper. The gap in identifiability is created in the framework by leveraging secret knowledge between the transmitter and the legitimate receiver. Specifically, the case where the legitimate receiver harnesses a secret block structure to decode a transmitted block-sparse message from underdetermined linear measurements in conditions where classical compressed sensing would provably fail is examined. The applicability of the proposed scheme to practical multiple access wireless communication systems is discussed. The protocol's privacy is studied under a single transmission, and under multiple transmissions without refreshing the secret block structure. It is shown that, under a specific scaling of the channel dimensions and transmission parameters, the eavesdropper can attempt to overhear the block structure from the fourth-order moments of the channel output. Computation of a statistical lower bound, suggests that the proposed fourth-order moment secret block estimation strategy is near optimal. The performance of a spectral clustering algorithm is studied to that end, defining scaling laws on the lifespan of the secret key before the communication is compromised. Finally, numerical experiments corroborating the theoretical findings are conducted.

Channel State Information-Free Location-Privacy Enhancement: Fake Path Injection

In this paper, a channel state information-free, fake path injection (FPI) scheme is proposed for location-privacy preservation. Specifically, structured artificial noise is designed to introduce virtual fake paths into the channels of the illegitimate devices. By leveraging the geometrical feasibility of the fake paths, under mild conditions, it can be proved that the illegitimate device cannot distinguish between a fake and true path, thus degrading the illegitimate devices' ability to localize. Two closed-form, lower bounds on the illegitimate devices' estimation error are derived via the analysis of the Fisher information of the location-relevant channel parameters, thus characterizing the enhanced location-privacy. A transmit beamformer is proposed, which efficiently injects the virtual fake paths. The intended device receives the two parameters of the beamformer design over a secure channel in order to enable localization. The impact of leaking the beamformer structure and associated localization leakage are analyzed. Theoretical analyses are verified via simulation. Numerical results show that a 20dB degradation of the illegitimate devices' localization accuracy can be achieved thus validating the efficacy of the proposed FPI versus using unstructured Gaussian noise.

Matrix Approximation with Side Information: When Column Sampling is Enough

A novel matrix approximation problem is considered herein: observations based on a few fully sampled columns and quasi-polynomial structural side information are exploited. The framework is motivated by quantum chemistry problems wherein full matrix computation is expensive, and partial computations only lead to column information. The proposed algorithm successfully estimates the column and row-space of a true matrix given a priori structural knowledge of the true matrix. A theoretical spectral error bound is provided, which captures the possible inaccuracies of the side information. The error bound proves it scales in its signal-to-noise (SNR) ratio. The proposed algorithm is validated via simulations which enable the characterization of the amount of information provided by the quasi-polynomial side information.

Channel State Information-Free Artificial Noise-Aided Location-Privacy Enhancement

In this paper, an artificial noise-aided strategy is presented for location-privacy preservation. A novel framework for the reduction of location-privacy leakage is introduced, where structured artificial noise is designed to degrade the structure of the illegitimate devices' channel, without the aid of channel state information at the transmitter. Then, based on the location-privacy enhancement framework, a transmit beamformer is proposed to efficiently inject the structured artificial noise. Furthermore, the securely shared information is characterized to enable the legitimate devices to localize accurately. Numerical results show a 9dB degradation of illegitimate devices' localization accuracy is achieved, and validate the efficacy of structured artificial noise versus unstructured Gaussian noise.

UAV-aided RF Mapping for Sensing and Connectivity in Wireless Networks

The use of unmanned aerial vehicles (UAV) as flying radio access network (RAN) nodes offers a promising complement to traditional fixed terrestrial deployments. More recently yet still in the context of wireless networks, drones have also been envisioned for use as radio frequency (RF) sensing and localization devices. In both cases, the advantage of using UAVs lies in their ability to navigate themselves freely in 3D and in a timely manner to locations of space where the obtained network throughput or sensing performance is optimal. In practice, the selection of a proper location or trajectory for the UAV very much depends on local terrain features, including the position of surrounding radio obstacles. Hence, the robot must be able to map the features of its radio environment as it performs its data communication or sensing services. The challenges related to this task, referred here as radio mapping, are discussed in this paper. Its promises related to efficient trajectory design for autonomous radio-aware UAVs are highlighted, along with algorithm solutions. The advantages induced by radio-mapping in terms of connectivity, sensing, and localization performance are illustrated.

Uncertainty-Based Non-Parametric Active Peak Detection

Active, non-parametric peak detection is considered. As a use case, active source localization is examined and an uncertainty-based sampling scheme algorithm to effectively localize the peak from a few energy measurements is designed. It is shown that under very mild conditions, the source localization error with $m$ actively chosen energy measurements scales as $O(\log^2 m/m)$. Numerically, it is shown that in low-sample regimes, the proposed method enjoys superior performance on several types of data and outperforms the state-of-the-art passive source localization approaches and in the low sample regime, can outperform greedy methods as well.