In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates the gradients by only using a subset of feature maps to save the GPU memory and computational cost. We interpret SBP as an efficient way to implement stochastic gradient decent by performing backpropagation dropout, which leads to considerable memory saving and training process speedup, with a minimal impact on the overall model accuracy. We offer some good practices to apply SBP in training image recognition models, which can be adopted in learning a wide range of deep neural networks. Experiments on image classification and object detection show that SBP can save up to 40% of GPU memory with less than 1% accuracy degradation.
With the shift towards on-device deep learning, ensuring a consistent behavior of an AI service across diverse compute platforms becomes tremendously important. Our work tackles the emergent problem of reducing predictive inconsistencies arising as negative flips: test samples that are correctly predicted by a less accurate model, but incorrectly by a more accurate one. We introduce REGression constrained Neural Architecture Search (REG-NAS) to design a family of highly accurate models that engender fewer negative flips. REG-NAS consists of two components: (1) A novel architecture constraint that enables a larger model to contain all the weights of the smaller one thus maximizing weight sharing. This idea stems from our observation that larger weight sharing among networks leads to similar sample-wise predictions and results in fewer negative flips; (2) A novel search reward that incorporates both Top-1 accuracy and negative flips in the architecture search metric. We demonstrate that \regnas can successfully find desirable architectures with few negative flips in three popular architecture search spaces. Compared to the existing state-of-the-art approach, REG-NAS enables 33-48% relative reduction of negative flips.
Federated learning (FL) is an emerging machine learning paradigm that allows to accomplish model training without aggregating data at a central server. Most studies on FL consider a centralized framework, in which a single server is endowed with a central authority to coordinate a number of devices to perform model training in an iterative manner. Due to stringent communication and bandwidth constraints, such a centralized framework has limited scalability as the number of devices grows. To address this issue, in this paper, we propose a ConFederated Learning (CFL) framework. The proposed CFL consists of multiple servers, in which each server is connected with an individual set of devices as in the conventional FL framework, and decentralized collaboration is leveraged among servers to make full use of the data dispersed throughout the network. We develop an alternating direction method of multipliers (ADMM) algorithm for CFL. The proposed algorithm employs a random scheduling policy which randomly selects a subset of devices to access their respective servers at each iteration, thus alleviating the need of uploading a huge amount of information from devices to servers. Theoretical analysis is presented to justify the proposed method. Numerical results show that the proposed method can converge to a decent solution significantly faster than gradient-based FL algorithms, thus boasting a substantial advantage in terms of communication efficiency.
To achieve communication-efficient federated multitask learning (FMTL), we propose an over-the-air FMTL (OAFMTL) framework, where multiple learning tasks deployed on edge devices share a non-orthogonal fading channel under the coordination of an edge server (ES). In OA-FMTL, the local updates of edge devices are sparsified, compressed, and then sent over the uplink channel in a superimposed fashion. The ES employs over-the-air computation in the presence of intertask interference. More specifically, the model aggregations of all the tasks are reconstructed from the channel observations concurrently, based on a modified version of the turbo compressed sensing (Turbo-CS) algorithm (named as M-Turbo-CS). We analyze the performance of the proposed OA-FMTL framework together with the M-Turbo-CS algorithm. Furthermore, based on the analysis, we formulate a communication-learning optimization problem to improve the system performance by adjusting the power allocation among the tasks at the edge devices. Numerical simulations show that our proposed OAFMTL effectively suppresses the inter-task interference, and achieves a learning performance comparable to its counterpart with orthogonal multi-task transmission. It is also shown that the proposed inter-task power allocation optimization algorithm substantially reduces the overall communication overhead by appropriately adjusting the power allocation among the tasks.
We consider the problem of spatial channel covariance matrix (CCM) estimation for intelligent reflecting surface (IRS)-assisted millimeter wave (mmWave) communication systems. Spatial CCM is essential for two-timescale beamforming in IRS-assisted systems; however, estimating the spatial CCM is challenging due to the passive nature of reflecting elements and the large size of the CCM resulting from massive reflecting elements of the IRS. In this paper, we propose a CCM estimation method by exploiting the low-rankness as well as the positive semi-definite (PSD) 3-level Toeplitz structure of the CCM. Estimation of the CCM is formulated as a semidefinite programming (SDP) problem and an alternating direction method of multipliers (ADMM) algorithm is developed. Our analysis shows that the proposed method is theoretically guaranteed to attain a reliable CCM estimate with a sample complexity much smaller than the dimension of the CCM. Thus the proposed method can help achieve a significant training overhead reduction. Simulation results are presented to illustrate the effectiveness of our proposed method and the performance of two-timescale beamforming scheme based on the estimated CCM.
We consider the problem of downlink channel estimation for intelligent reflecting surface (IRS)-assisted millimeter Wave (mmWave) orthogonal frequency division multiplexing (OFDM) systems. By exploring the inherent sparse scattering characteristics of mmWave channels, we show that the received signals can be expressed as a low-rank third-order tensor that admits a tensor rank decomposition, also known as canonical polyadic decomposition (CPD). A structured CPD-based method is then developed to estimate the channel parameters. Our analysis reveals that the training overhead required by our proposed method is as low as O(U^2), where U denotes the sparsity of the cascade channel. Simulation results are provided to illustrate the efficiency of the proposed method.
Reconfigurable intelligent surface (RIS) has recently emerged as a promising paradigm for future cellular networks. Specifically, due to its capability in reshaping the propagation environment, RIS was introduced to address the blockage issue in millimeter Wave (mmWave) or even Terahertz (THz) communications. The deployment of RIS, however, complicates the system architecture and poses a significant challenge for beam training (BT)/ beam alignment (BA), a process that is required to establish a reliable link between the transmitter and the receiver. In this article, we first review several state-of-the-art beam training solutions for RIS-assisted mmWave systems and discuss their respective advantages and limitations. We also present a new multi-directional BT method, which can achieve a decent BA performance with only a small amount of training overhead. Finally, we outline several important open issues in BT for RIS-assisted mmWave systems.
The great success of deep learning (DL) has inspired researchers to develop more accurate and efficient symbol detectors for multi-input multi-output (MIMO) systems. Existing DL-based MIMO detectors, however, suffer several drawbacks. To address these issues, in this paper, we develop a modeldriven DL detector based on variational Bayesian inference. Specifically, the proposed unrolled DL architecture is inspired by an inverse-free variational Bayesian learning framework which circumvents matrix inversion via maximizing a relaxed evidence lower bound. Two networks are respectively developed for independent and identically distributed (i.i.d.) Gaussian channels and arbitrarily correlated channels. The proposed networks, referred to as VBINet, have only a few learnable parameters and thus can be efficiently trained with a moderate amount of training samples. The proposed VBINet-based detectors can work in both offline and online training modes. An important advantage of our proposed networks over state-of-the-art MIMO detection networks such as OAMPNet and MMNet is that the VBINet can automatically learn the noise variance from data, thus yielding a significant performance improvement over the OAMPNet and MMNet in the presence of noise variance uncertainty. Simulation results show that the proposed VBINet-based detectors achieve competitive performance for both i.i.d. Gaussian and realistic 3GPP MIMO channels.
Intelligent reflecting surface (IRS) is a promising technology being considered for future wireless communications due to its ability to control signal propagation. This paper considers the joint active and passive beamforming problem for an IRS-assisted radar, where multiple IRSs are deployed to assist the surveillance of multiple targets in cluttered environments. Specifically, we aim to maximize the minimum target illumination power at multiple target locations by jointly optimizing the active beamformer at the radar transmitter and the passive phase-shift matrices at the IRSs, subject to an upperbound on the clutter power at each clutter scatterer. The resulting optimization problem is nonconvex and solved with a sequential optimization procedure along with semedefinite relaxation (SDR). Simulation results show that IRSs can help create effective line-of-sight (LOS) paths and thus substantially improve the radar robustness against target blockage.
Cognitive radio (CR) is a promising technology enabling efficient utilization of the spectrum resource for future wireless systems. As future CR networks are envisioned to operate over a wide frequency range, advanced wideband spectrum sensing (WBSS) capable of quickly and reliably detecting idle spectrum bands across a wide frequency span is essential. In this article, we provide an overview of recent advances on sub-Nyquist sampling-based WBSS techniques, including compressed sensing-based methods and compressive covariance sensing-based methods. An elaborate discussion of the pros and cons of each approach is presented, along with some challenging issues for future research. A comparative study suggests that the compressive covariance sensing-based approach offers a more competitive solution for reliable real-time WBSS.