Mode Hopping for Anti-Jamming in Radio Vortex Wireless Communications

Frequency hopping (FH) has been widely used as a powerful technique for antijamming in wireless communications. However, as the wireless spectrum is becoming more and more crowded, it is very difficult to achieve efficient antijamming results with FH-based schemes. Orbital angular momentum (OAM), which provides the new angular/mode dimension for wireless communications, offers an intriguing way for antijamming. In this paper, we propose to use the orthogonality of OAM-modes for antijamming in wireless communications. In particular, we propose the mode hopping (MH) scheme for antijamming within the narrow frequency band. We derive the closed-form expression of bit error rate (BER) for multiple users scenario with our developed MH scheme. Our developed MH scheme can achieve the same antijamming results within the narrow frequency band as compared with the conventional wideband FH scheme. Furthermore, we propose mode-frequency hopping (MFH) scheme, which jointly uses our developed MH scheme and the conventional FH scheme to further decrease the BER for wireless communication. Numerical results are presented to show that the BER of our developed MH scheme within the narrow frequency band is the same with that of the conventional wideband FH scheme. Moreover, the BER of our developed MFH schemes is much smaller than that of the conventional FH schemes for wireless communications.

Joint OAM Multiplexing and OFDM in Sparse Multipath Environments

The emerging orbital angular momentum (OAM) based wireless communication is expected to be a high spectrum-efficiency communication paradigm to solve the growing transmission data rate and limited bandwidth problem. Academic researchers mainly concentrate on the OAM-based line-of-sight (LoS) communications. However, there exist some surroundings around the transceiver in most practical wireless communication scenarios, thus forming multipath transmission. In this paper, a hybrid orthogonal division multiplexing (HODM) scheme by using OAM multiplexing and orthogonal frequency division multiplexing (OFDM) in conjunction is proposed to achieve high-capacity wireless communications in sparse multipath environments, where the scatterers are sparse. We first build the OAM-based wireless channel in a LoS path and several reflection paths combined sparse multipath environments. We concentrate on less than or equal to three-time reflection paths because of the severe energy attenuation. The phase difference among the channel amplitude gains of the LoS and reflection paths, which is caused by the reflection paths, makes it difficult to decompose the OAM signals. We propose the phase difference compensation to handle this problem and then calculate the corresponding capacity in radio vortex wireless communications. Numerical results illustrate that the capacity of wireless communications by using our proposed HODM scheme can be drastically increased in sparse multipath environments.

Mode Hopping with OAM-Based Index Modulation

Orbital angular momentum (OAM) based mode hopping (MH) scheme is expected to be a potential anti-jamming technology in radio vortex wireless communications. However, it only uses one OAM-mode for hopping, thus resulting in low spectrum efficiency (SE). Index modulation offers a trade-off balance between the SE and performance reliability. In this paper, we propose an MH with OAM-based index modulation scheme, where several OAM-modes are activated for hopping, to achieve high SE at a given bit error rate in radio vortex wireless communications. Based on the proposed scheme, we derive the upper bound and lower bound of achievable SEs. Furthermore, in order to take advantage of index information, we derive the optimal hopped OAM-modes to achieve the maximum SE. Numerical results show that our proposed MH with index modulation scheme can achieve high SE while satisfying a certain reliability of radio vortex wireless communications.

Index Modulation Embedded Mode Hopping for Anti-Jamming

Due to the crowded spectrum, frequency hopping (FH) techniques are now very difficult to achieve efficient antijamming and increase spectrum efficiency (SE) for wireless communications. The emerging orbital angular momentum (OAM), which is a property describing the helical phase fronts of electromagnetic waves, offers the potential to improve reliability and increase SE in wireless communications. To achieve efficient anti-jamming and increase SE of wireless communications with slight computational complexity cost, in this paper we propose an index-modulation embedded mode-hopping (IM-MH) scheme, which simultaneously activates several OAM-modes for hopping along with additional index information and signal information transmission. We analyze the average bit error rates (ABERs) for our proposed IM-MH scheme with perfect channel state information (CSI) and imperfect CSI, respectively. We also propose the index-modulation embedded double-serial MH (IMDSMH) scheme, which randomly activates one OAM-mode as the serial second hop to transmit the hopping signals in the IM-MH scheme, to further decrease the ABER of wireless communications. Extensive numerical results demonstrate that our proposed schemes within a narrowband can achieve the low ABER and significantly increase the SE. Also, the ABERs of our proposed IM-MH and IM-DSMH schemes are around 25% and 10%, respectively, compared with that of the mode hopping scheme.

Efficiently Training 7B LLM with 1 Million Sequence Length on 8 GPUs

Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges considering the constraint of GPU memory. It not only leads to substantial activation memory consumption during training, but also incurs considerable memory fragmentation. To facilitate long context training, existing frameworks have adopted strategies such as recomputation and various forms of parallelisms. Nevertheless, these techniques rely on redundant computation or extensive communication, resulting in low Model FLOPS Utilization (MFU). In this paper, we propose MEMO, a novel LLM training framework designed for fine-grained activation memory management. Given the quadratic scaling of computation and linear scaling of memory with sequence lengths when using FlashAttention, we offload memory-consuming activations to CPU memory after each layer's forward pass and fetch them during the backward pass. To maximize the swapping of activations without hindering computation, and to avoid exhausting limited CPU memory, we implement a token-wise activation recomputation and swapping mechanism. Furthermore, we tackle the memory fragmentation issue by employing a bi-level Mixed Integer Programming (MIP) approach, optimizing the reuse of memory across transformer layers. Empirical results demonstrate that MEMO achieves an average of 2.42x and 2.26x MFU compared to Megatron-LM and DeepSpeed, respectively. This improvement is attributed to MEMO's ability to minimize memory fragmentation, reduce recomputation and intensive communication, and circumvent the delays associated with the memory reorganization process due to fragmentation. By leveraging fine-grained activation memory management, MEMO facilitates efficient training of 7B LLM with 1 million sequence length on just 8 A800 GPUs, achieving an MFU of 52.30%.

OAM-SWIPT for IoE-Driven 6G

Simultaneous wireless information and power transfer (SWIPT), which achieves both wireless energy transfer (WET) and information transfer, is an attractive technique for future Internet of Everything (IoE) in the sixth-generation (6G) mobile communications. With SWIPT, battery-less IoE devices can be powered while communicating with other devices. Line-of-sight (LOS) RF transmission and near-field inductive coupling based transmission are typical SWIPT scenarios, which are both LOS channels and without enough degree of freedom for high spectrum efficiency as well as high energy efficiency. Due to the orthogonal wavefronts, orbital angular momentum (OAM) can facilitate the SWIPT in LOS channels. In this article, we introduce the OAM-based SWIPT as well as discuss some basic advantages and challenges for it. After introducing the OAM-based SWIPT for IoE, we first propose an OAM-based SWIPT system model with the OAM-modes assisted dynamic power splitting (DPS). Then, four basic advantages regarding the OAM-based SWIPT are reviewed with some numerical analyses for further demonstrating the advantages. Next, four challenges regarding integrating OAM into SWIPT and possible solutions are discussed. OAM technology provides multiple orthogonal streams to increase both spectrum and energy efficiencies for SWIPT, thus creating many opportunities for future WET and SWIPT researches.

Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling

In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optimizers typically update model parameters using the sign of gradients, resulting in more stable convergence curves. The learning rate and the batch size are the most critical hyperparameters for optimizers, which require careful tuning to enable effective convergence. Previous research has shown that the optimal learning rate increases linearly or follows similar rules with batch size for SGD style optimizers. However, this conclusion is not applicable to Adam style optimizers. In this paper, we elucidate the connection between optimal learning rates and batch sizes for Adam style optimizers through both theoretical analysis and extensive experiments. First, we raise the scaling law between batch sizes and optimal learning rates in the sign of gradient case, in which we prove that the optimal learning rate first rises and then falls as the batch size increases. Moreover, the peak value of the surge will gradually move toward the larger batch size as training progresses. Second, we conducted experiments on various CV and NLP tasks and verified the correctness of the scaling law.

Retrieval-Augmented Generation for AI-Generated Content: A Survey

The development of Artificial Intelligence Generated Content (AIGC) has been facilitated by advancements in model algorithms, scalable foundation model architectures, and the availability of ample high-quality datasets. While AIGC has achieved remarkable performance, it still faces challenges, such as the difficulty of maintaining up-to-date and long-tail knowledge, the risk of data leakage, and the high costs associated with training and inference. Retrieval-Augmented Generation (RAG) has recently emerged as a paradigm to address such challenges. In particular, RAG introduces the information retrieval process, which enhances AIGC results by retrieving relevant objects from available data stores, leading to greater accuracy and robustness. In this paper, we comprehensively review existing efforts that integrate RAG technique into AIGC scenarios. We first classify RAG foundations according to how the retriever augments the generator. We distill the fundamental abstractions of the augmentation methodologies for various retrievers and generators. This unified perspective encompasses all RAG scenarios, illuminating advancements and pivotal technologies that help with potential future progress. We also summarize additional enhancements methods for RAG, facilitating effective engineering and implementation of RAG systems. Then from another view, we survey on practical applications of RAG across different modalities and tasks, offering valuable references for researchers and practitioners. Furthermore, we introduce the benchmarks for RAG, discuss the limitations of current RAG systems, and suggest potential directions for future research. Project: https://github.com/hymie122/RAG-Survey

CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models

Recently, the growing memory demands of embedding tables in Deep Learning Recommendation Models (DLRMs) pose great challenges for model training and deployment. Existing embedding compression solutions cannot simultaneously meet three key design requirements: memory efficiency, low latency, and adaptability to dynamic data distribution. This paper presents CAFE, a Compact, Adaptive, and Fast Embedding compression framework that addresses the above requirements. The design philosophy of CAFE is to dynamically allocate more memory resources to important features (called hot features), and allocate less memory to unimportant ones. In CAFE, we propose a fast and lightweight sketch data structure, named HotSketch, to capture feature importance and report hot features in real time. For each reported hot feature, we assign it a unique embedding. For the non-hot features, we allow multiple features to share one embedding by using hash embedding technique. Guided by our design philosophy, we further propose a multi-level hash embedding framework to optimize the embedding tables of non-hot features. We theoretically analyze the accuracy of HotSketch, and analyze the model convergence against deviation. Extensive experiments show that CAFE significantly outperforms existing embedding compression methods, yielding 3.92% and 3.68% superior testing AUC on Criteo Kaggle dataset and CriteoTB dataset at a compression ratio of 10000x. The source codes of CAFE are available at GitHub.

Experimental Analysis of Large-scale Learnable Vector Storage Compression

Learnable embedding vector is one of the most important applications in machine learning, and is widely used in various database-related domains. However, the high dimensionality of sparse data in recommendation tasks and the huge volume of corpus in retrieval-related tasks lead to a large memory consumption of the embedding table, which poses a great challenge to the training and deployment of models. Recent research has proposed various methods to compress the embeddings at the cost of a slight decrease in model quality or the introduction of other overheads. Nevertheless, the relative performance of these methods remains unclear. Existing experimental comparisons only cover a subset of these methods and focus on limited metrics. In this paper, we perform a comprehensive comparative analysis and experimental evaluation of embedding compression. We introduce a new taxonomy that categorizes these techniques based on their characteristics and methodologies, and further develop a modular benchmarking framework that integrates 14 representative methods. Under a uniform test environment, our benchmark fairly evaluates each approach, presents their strengths and weaknesses under different memory budgets, and recommends the best method based on the use case. In addition to providing useful guidelines, our study also uncovers the limitations of current methods and suggests potential directions for future research.