Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey

Resource constraints increasingly determine what can be trained, fine-tuned, and deployed in large language models (LLMs), yet efficiency is often studied through isolated techniques rather than as an interacting system of limits. This survey adopts a constraint-centric perspective and organizes recent progress around three coupled bottlenecks: data efficiency (what to train on), memory efficiency (how to fit training), and compute budget awareness (when and where to spend FLOPs). On the data axis, we review selection and pruning methods that maximize learning per token, ranging from scalable proxy signals based on learning dynamics to gradient- and influence-based scoring, as well as difficulty-aware and curriculum-style strategies. We highlight emerging evidence that different notions of good data dominate in different regimes, implying that optimal subsets depend on the task objective and resource budget rather than being universal. On the systems side, we show that GPU memory, not raw compute, is often the dominant bottleneck in fine-tuning, and that effective scaling requires jointly reducing weight storage, optimizer states, and activation memory rather than optimizing any single component in isolation. Beyond memory, we frame training and inference as compute-governed processes in which optimization, data selection, and decoding must explicitly account for finite FLOP budgets. We review evidence for compute-optimal allocation and stopping rules, where computation should be halted or reallocated once marginal performance gains fall below a budget-dependent threshold. Together, these results unify compute-aware data selection, scaling laws, and adaptive inference under a common principle of resource-conditioned decision-making.

Benchmarking Wireless Representations: High-Dimensional vs. Compressed Embeddings for Efficiency and Robustness

Building on recent advances in representation learning for wireless channels, this work investigates the cost-benefit trade-offs of high-dimensional channel embeddings in practical systems. We benchmark multiple wireless representations: high-dimensional learned embeddings from a wireless foundation model, compact autoencoder-based representations with significantly lower dimensionality, and raw data baselines, evaluating their performance across diverse downstream tasks. We then systematically analyze data efficiency, noise robustness, and computational complexity, explicitly characterizing the resource overhead associated with high-dimensional embeddings. Beyond standard tasks such as line-of-sight/non-line-of-sight (LoS/NLoS) classification and beam selection, we introduce power allocation as a new downstream task. Our results reveal clear trade-offs: while high-dimensional embeddings can perform well in few-shot regimes for certain tasks, they incur substantial latency and parameter overhead. In contrast, compressed latent representations learned by autoencoders demonstrate improved noise robustness and more stable performance across tasks, while significantly reducing computational and transmission costs.

A Full Compression Pipeline for Green Federated Learning in Communication-Constrained Environments

Federated Learning (FL) enables collaborative model training across distributed clients without sharing raw data, thereby preserving privacy. However, FL often suffers from significant communication and computational overhead, limiting its scalability and sustainability. In this work, we introduce a Full Compression Pipeline (FCP) for FL in communication-constrained environments. FCP integrates three complementary deep compression techniques (pruning, quantization, and Huffman encoding) into a unified end-to-end framework. By compressing local models and communication payloads, FCP substantially reduces transmission costs and resource consumption while maintaining competitive accuracy. To quantify its impact, we develop an evaluation framework that captures both communication and computation overheads as a unified model cost, allowing a holistic assessment of efficiency trade-offs. The pipeline is evaluated in an independent and identically distributed (IID) and non-IID data setting. In one representative scenario, training a ResNet-12 model on the CIFAR-10 dataset with ten clients and a 2 Mbps bandwidth, the FCP achieves more than 11$\times$ reduction in model size, with only a 2% drop in accuracy compared to the uncompressed baseline. This results in an FL training that is more than 60% faster.

Conformal Cross-Modal Active Learning

Foundation models for vision have transformed visual recognition with powerful pretrained representations and strong zero-shot capabilities, yet their potential for data-efficient learning remains largely untapped. Active Learning (AL) aims to minimize annotation costs by strategically selecting the most informative samples for labeling, but existing methods largely overlook the rich multimodal knowledge embedded in modern vision-language models (VLMs). We introduce Conformal Cross-Modal Acquisition (CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture. CCMA employs a pretrained VLM as a teacher to provide semantically grounded uncertainty estimates, conformally calibrated to guide sample selection for a vision-only student model. By integrating multimodal conformal scoring with diversity-aware selection strategies, CCMA achieves superior data efficiency across multiple benchmarks. Our approach consistently outperforms state-of-the-art AL baselines, demonstrating clear advantages over methods relying solely on uncertainty or diversity metrics.

Active Learning Using Aggregated Acquisition Functions: Accuracy and Sustainability Analysis

Active learning (AL) is a machine learning (ML) approach that strategically selects the most informative samples for annotation during training, aiming to minimize annotation costs. This strategy not only reduces labeling expenses but also results in energy savings during neural network training, thereby enhancing both data and energy efficiency. In this paper, we implement and evaluate various state-of-the-art acquisition functions, analyzing their accuracy and computational costs, while discussing the advantages and disadvantages of each method. Our findings reveal that representativity-based acquisition functions effectively explore the dataset but do not prioritize boundary decisions, whereas uncertainty-based acquisition functions focus on refining boundary decisions already identified by the neural network. This trade-off is known as the exploration-exploitation dilemma. To address this dilemma, we introduce six aggregation structures: series, parallel, hybrid, adaptive feedback, random exploration, and annealing exploration. Our aggregated acquisition functions alleviate common AL pathologies such as batch mode inefficiency and the cold start problem. Additionally, we focus on balancing accuracy and energy consumption, contributing to the development of more sustainable, energy-aware artificial intelligence (AI). We evaluate our proposed structures on various models and datasets. Our results demonstrate the potential of these structures to reduce computational costs while maintaining or even improving accuracy. Innovative aggregation approaches, such as alternating between acquisition functions such as BALD and BADGE, have shown robust results. Sequentially running functions like $K$-Centers followed by BALD has achieved the same performance goals with up to 12\% fewer samples, while reducing the acquisition cost by almost half.

Efficient Joint Resource Allocation for Wireless Powered ISAC with Target Localization

Wireless powered integrated sensing and communication (ISAC) faces a fundamental tradeoff between energy supply, communication throughput, and sensing accuracy. This paper investigates a wireless powered ISAC system with target localization requirements, where users harvest energy from wireless power transfer (WPT) and then conduct ISAC transmissions in a time-division manner. In addition to energy supply, the WPT signal also contributes to target sensing, and the localization accuracy is characterized by Cramér-Rao bound (CRB) constraints. Under this setting, we formulate a max-min throughput maximization problem by jointly allocating the WPT duration, ISAC transmission time allocation, and transmit power. Due to the nonconvexity of the resulting problem, a suitable reformulation is developed by exploiting variable substitutions and the monotonicity of logarithmic functions, based on which an efficient successive convex approximation (SCA)-based iterative algorithm is proposed. Simulation results demonstrate convergence and significant performance gains over benchmark schemes, highlighting the importance of coordinated time-power optimization in balancing sensing accuracy and communication performance in wireless powered ISAC systems.

Joint Link Adaptation and Device Scheduling Approach for URLLC Industrial IoT Network: A DRL-based Method with Bayesian Optimization

In this article, we consider an industrial internet of things (IIoT) network supporting multi-device dynamic ultra-reliable low-latency communication (URLLC) while the channel state information (CSI) is imperfect. A joint link adaptation (LA) and device scheduling (including the order) design is provided, aiming at maximizing the total transmission rate under strict block error rate (BLER) constraints. In particular, a Bayesian optimization (BO) driven Twin Delayed Deep Deterministic Policy Gradient (TD3) method is proposed, which determines the device served order sequence and the corresponding modulation and coding scheme (MCS) adaptively based on the imperfect CSI. Note that the imperfection of CSI, error sample imbalance in URLLC networks, as well as the parameter sensitivity nature of the TD3 algorithm likely diminish the algorithm's convergence speed and reliability. To address such an issue, we proposed a BO based training mechanism for the convergence speed improvement, which provides a more reliable learning direction and sample selection method to track the imbalance sample problem. Via extensive simulations, we show that the proposed algorithm achieves faster convergence and higher sum-rate performance compared to existing solutions.

On Signal Peak Power Constraint of Over-the-Air Federated Learning

Federated learning (FL) has been considered a promising privacy preserving distributed edge learning framework. Over-the-air computation (AirComp) technique leveraging analog transmission enables the aggregation of local updates directly over-the-air by exploiting the superposition properties of wireless multiple-access channel, thereby drastically reducing the communication bottleneck issues of FL compared with digital transmission schemes. This work points out that existing AirComp-FL overlooks a key practical constraint, the instantaneous peak-power constraints imposed by the non-linearity of radiofrequency power amplifiers. We present and analyze the effect of the classic method to deal with this issue, amplitude clipping combined with filtering. We investigate the effect of instantaneous peak-power constraints in AirComp-FL for both single-carrier and multi-carrier orthogonal frequency-division multiplexing (OFDM) systems. We highlight the specificity of AirComp-FL: the samples depend on the gradient value distribution, leading to a higher peak-to-average power ratio (PAPR) than that observed for uniformly distributed signals. Simulation results demonstrate that, in practical settings, the instantaneous transmit power regularly exceeds the power-amplifier limit; however, by applying clipping and filtering, the FL performance can be degraded. The degradation becomes pronounced especially in multi-carrier OFDM systems due to the in-band distortions caused by clipping and filtering.

Enhancing CTC-Based Visual Speech Recognition

This paper presents LiteVSR2, an enhanced version of our previously introduced efficient approach to Visual Speech Recognition (VSR). Building upon our knowledge distillation framework from a pre-trained Automatic Speech Recognition (ASR) model, we introduce two key improvements: a stabilized video preprocessing technique and feature normalization in the distillation process. These improvements yield substantial performance gains on the LRS2 and LRS3 benchmarks, positioning LiteVSR2 as the current best CTC-based VSR model without increasing the volume of training data or computational resources utilized. Furthermore, we explore the scalability of our approach by examining performance metrics across varying model complexities and training data volumes. LiteVSR2 maintains the efficiency of its predecessor while significantly enhancing accuracy, thereby demonstrating the potential for resource-efficient advancements in VSR technology.

Parallel Split Learning with Global Sampling

The expansion of IoT devices and the demands of Deep Learning have highlighted significant challenges in Distributed Deep Learning (DDL) systems. Parallel Split Learning (PSL) has emerged as a promising derivative of Split Learning that is well suited for distributed learning on resource-constrained devices. However, PSL faces several obstacles, such as large effective batch sizes, non-IID data distributions, and the straggler effect. We view these issues as a sampling dilemma and propose to address them by orchestrating the mini-batch sampling process on the server side. We introduce the Uniform Global Sampling (UGS) method to decouple the effective batch size from the number of clients and reduce mini-batch deviation in non-IID settings. To address the straggler effect, we introduce the Latent Dirichlet Sampling (LDS) method, which generalizes UGS to balance the trade-off between batch deviation and training time. Our simulations reveal that our proposed methods enhance model accuracy by up to 34.1% in non-IID settings and reduce the training time in the presence of stragglers by up to 62%. In particular, LDS effectively mitigates the straggler effect without compromising model accuracy or adding significant computational overhead compared to UGS. Our results demonstrate the potential of our methods as a promising solution for DDL in real applications.