Guided Policy Optimization under Partial Observability

Reinforcement Learning (RL) in partially observable environments poses significant challenges due to the complexity of learning under uncertainty. While additional information, such as that available in simulations, can enhance training, effectively leveraging it remains an open problem. To address this, we introduce Guided Policy Optimization (GPO), a framework that co-trains a guider and a learner. The guider takes advantage of privileged information while ensuring alignment with the learner's policy that is primarily trained via imitation learning. We theoretically demonstrate that this learning scheme achieves optimality comparable to direct RL, thereby overcoming key limitations inherent in existing approaches. Empirical evaluations show strong performance of GPO across various tasks, including continuous control with partial observability and noise, and memory-based challenges, significantly outperforming existing methods.

Emergent Specialization: Rare Token Neurons in Language Models

Large language models struggle with representing and generating rare tokens despite their importance in specialized domains. In this study, we identify neuron structures with exceptionally strong influence on language model's prediction of rare tokens, termed as rare token neurons, and investigate the mechanism for their emergence and behavior. These neurons exhibit a characteristic three-phase organization (plateau, power-law, and rapid decay) that emerges dynamically during training, evolving from a homogeneous initial state to a functionally differentiated architecture. In the activation space, rare token neurons form a coordinated subnetwork that selectively co-activates while avoiding co-activation with other neurons. This functional specialization potentially correlates with the development of heavy-tailed weight distributions, suggesting a statistical mechanical basis for emergent specialization.

System Concept and Demonstration of Bistatic MIMO-OFDM-based ISAC

In future sixth-generation (6G) mobile networks, radar sensing is expected to be offered as an additional service to its original purpose of communication. Merging these two functions results in integrated sensing and communication (ISAC) systems. In this context, bistatic ISAC appears as a possibility to exploit the distributed nature of cellular networks while avoiding highly demanding hardware requirements such as full-duplex operation. Recent studies have introduced strategies to perform required synchronization and data exchange between nodes for bistatic ISAC operation, based on orthogonal frequency-division multiplexing (OFDM), however, only for single-input single-output architectures. In this article, a system concept for a bistatic multiple-input multiple-output (MIMO)-OFDM-based ISAC system with beamforming at both transmitter and receiver is proposed, and a distribution synchronization concept to ensure coherence among the different receive channels for direction-of-arrival estimation is presented. After a discussion on the ISAC processing chain, including relevant aspects for practical deployments such as transmitter digital pre-distortion and receiver calibration, a 4x8 MIMO measurement setup at 27.5 GHz and results are presented to validate the proposed system and distribution synchronization concepts.

Joint BS Deployment and Power Optimization for Minimum EMF Exposure with RL in Real-World Based Urban Scenario

Conventional base station (BS) deployments typically prioritize coverage, quality of service (QoS), or cost reduction, often overlooking electromagnetic field (EMF) exposure. Whereas EMF exposure triggers significant public concern due to its potential health implications, making it crucial to address when deploying BS in densely populated areas. To this end, this paper addresses minimizing average EMF exposure while maintaining coverage in a 3D urban scenario by jointly optimizing BS deployment and power. To address this, firstly, accurate EMF prediction is essential, as traditional empirical models lack the required accuracy, necessitating a deterministic channel model. A novel least-time shoot-and-bounce ray (SBR) ray-launching (RL) algorithm is therefore developed to overcome several limitations of current simulators and is validated with real-world measurements. Secondly, to further reduce computational complexity, unlike using a fixed grid size to discretize the target area, the adaptive grid refinement (AGR) algorithm is designed with a flexible grid to predict the overall EMF exposure. Finally, based on the EMF exposure predictions, the Nelder-Mead (NM) method is used in the joint optimization, and urban user equipment (UE) distributions are incorporated to better reflect real-world conditions. When evaluating the benefits of the whole process, the results are compared against using empirical channel models, revealing notable differences and underestimation of EMF exposure that highlight the importance of considering real-world scenario.

On the Sensing Performance of OFDM-based ISAC under the Influence of Oscillator Phase Noise

Integrated sensing and communication (ISAC) is a novel capability expected for sixth generation (6G) cellular networks. To that end, several challenges must be addressed to enable both mono- and bistatic sensing in existing deployments. A common impairment in both architectures is oscillator phase noise (PN), which not only degrades communication performance, but also severely impairs radar sensing. To enable a broader understanding of orthogonal-frequency division multiplexing (OFDM)-based sensing impaired by PN, this article presents an analysis of sensing peformance in OFDM-based ISAC for different waveform parameter choices and settings in both mono- and bistatic architectures. In this context, the distortion of the adopted digital constellation modulation is analyzed and the resulting PN-induced effects in range-Doppler radar images are investigated both without and with PN compensation. These effects include peak power loss of target reflections and higher sidelobe levels, especially in the Doppler shift direction. In the conducted analysis, these effects are measured by the peak power loss ratio, peak-to-sidelobe level ratio, and integrated sidelobe level ratio parameters, the two latter being evaluated in both range and Doppler shift directions. In addition, the signal-to-interference ratio is analyzed to allow not only quantifying the distortion of a target reflection, but also measuring the interference floor level in a radar image. The achieved results allow to quantify not only the PN-induced impairments to a single target, but also how the induced degradation may impair the sensing performance of OFDM-based ISAC systems in multi-target scenarios.

Pilot-Based SFO Estimation for Bistatic Integrated Sensing and Communication

Enabling bistatic radar sensing within the context of integrated sensing and communication (ISAC) for future sixth generation mobile networks demands strict synchronization accuracy, which is particularly challenging to be achieved with over-the-air synchronization. Existing algorithms handle time and frequency offsets adequately, but provide insufficiently accurate sampling frequency offset (SFO) estimates that result in degradation of obtained radar images in the form of signal-to-noise ratio loss and migration of range and Doppler shift. This article introduces an SFO estimation algorithm named tilt inference of time offset (TITO) for orthogonal frequency-division multiplexing (OFDM)-based ISAC. Using available pilot subcarriers, TITO obtains channel impulse response estimates and extracts information on the SFO-induced delay migration to a dominant reference path with constant range, Doppler shift, and angle between transmit and receive ISAC nodes. TITO then adaptively selects the delay estimates that are only negligibly impaired by SFO-induced intersymbol interference, ultimately employing them to estimate the SFO. Assuming a scenario without a direct line-of-sight (LoS) between the aforementioned transmitting and receiving ISAC nodes, a system concept with a relay reflective intelligent surface (RIS) is used to create the aforementioned reference path is proposed. Besides a mathematical derivation of accuracy bounds, simulation and measurements at 26.2 GHz are presented to demonstrate TITO's superiority over existing methods in terms of SFO estimation accuracy and robustness.

Bistatic OFDM-based ISAC with Over-the-Air Synchronization: System Concept and Performance Analysis

Integrated sensing and communication (ISAC) has been defined as one goal for 6G mobile communication systems. In this context, this article introduces a bistatic ISAC system based on orthogonal frequency-division multiplexing (OFDM). While the bistatic architecture brings advantages such as not demanding full duplex operation with respect to the monostatic one, the need for synchronizing transmitter and receiver is imposed. In this context, this article introuces a bistatic ISAC signal processing framework where an incoming OFDM-based ISAC signal undergoes over-the-air synchronization based on preamble symbols and pilots. Afterwards, bistatic radar processing is performed using either only pilot subcarriers or the full OFDM frame. The latter approach requires estimation of the originally transmitted frame based on communication processing and therefore error-free communication, which can be achieved via appropriate channel coding. The performance and limitations of the introduced system based on both aforementioned approaches are assessed via an analysis of the impact of residual synchronization mismatches and data decoding failures on both communication and radar performances. Finally, the performed analyses are validated by proof-of-concept measurement results.

Enabling Joint Radar-Communication Operation in Shift Register-Based PMCW Radars

This article introduces adaptations to the conventional frame structure in binary phase-modulated continuous wave (PMCW) radars with sequence generation via linear-feedbck shift registers and additional processing steps to enable joint radar-communication (RadCom) operation. In this context, a preamble structure based on pseudorandom binary sequences (PRBSs) that is compatible with existing synchronization algorithms is outlined, and the allocation of pilot PRBS blocks is discussed. Finally, results from proof-of-concept measurements are presented to illustrate the effects of the choice of system and signal parameters and validate the investigated PMCW-based RadCom system and synchronization strategy.

Breast Cancer Immunohistochemical Image Generation: a Benchmark Dataset and Challenge Review

For invasive breast cancer, immunohistochemical (IHC) techniques are often used to detect the expression level of human epidermal growth factor receptor-2 (HER2) in breast tissue to formulate a precise treatment plan. From the perspective of saving manpower, material and time costs, directly generating IHC-stained images from hematoxylin and eosin (H&E) stained images is a valuable research direction. Therefore, we held the breast cancer immunohistochemical image generation challenge, aiming to explore novel ideas of deep learning technology in pathological image generation and promote research in this field. The challenge provided registered H&E and IHC-stained image pairs, and participants were required to use these images to train a model that can directly generate IHC-stained images from corresponding H&E-stained images. We selected and reviewed the five highest-ranking methods based on their PSNR and SSIM metrics, while also providing overviews of the corresponding pipelines and implementations. In this paper, we further analyze the current limitations in the field of breast cancer immunohistochemical image generation and forecast the future development of this field. We hope that the released dataset and the challenge will inspire more scholars to jointly study higher-quality IHC-stained image generation.

Improving ABR Performance for Short Video Streaming Using Multi-Agent Reinforcement Learning with Expert Guidance

In the realm of short video streaming, popular adaptive bitrate (ABR) algorithms developed for classical long video applications suffer from catastrophic failures because they are tuned to solely adapt bitrates. Instead, short video adaptive bitrate (SABR) algorithms have to properly determine which video at which bitrate level together for content prefetching, without sacrificing the users' quality of experience (QoE) and yielding noticeable bandwidth wastage jointly. Unfortunately, existing SABR methods are inevitably entangled with slow convergence and poor generalization. Thus, in this paper, we propose Incendio, a novel SABR framework that applies Multi-Agent Reinforcement Learning (MARL) with Expert Guidance to separate the decision of video ID and video bitrate in respective buffer management and bitrate adaptation agents to maximize the system-level utilized score modeled as a compound function of QoE and bandwidth wastage metrics. To train Incendio, it is first initialized by imitating the hand-crafted expert rules and then fine-tuned through the use of MARL. Results from extensive experiments indicate that Incendio outperforms the current state-of-the-art SABR algorithm with a 53.2% improvement measured by the utility score while maintaining low training complexity and inference time.