UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.

State Estimation with 1-Bit Observations and Imperfect Models: Bussgang Meets Kalman in Neural Networks

State estimation from noisy observations is a fundamental problem in many applications of signal processing. Traditional methods, such as the extended Kalman filter, work well under fully-known Gaussian models, while recent hybrid deep learning frameworks, combining model-based and data-driven approaches, can also handle partially known models and non-Gaussian noise. However, existing studies commonly assume the absence of quantization distortion, which is inevitable, especially with non-ideal analog-to-digital converters. In this work, we consider a state estimation problem with 1-bit quantization. 1-bit quantization causes significant quantization distortion and severe information loss, rendering conventional state estimation strategies unsuitable. To address this, inspired by the Bussgang decomposition technique, we first develop the Bussgang-aided Kalman filter by assuming perfectly known models. The proposed method suitably captures quantization distortion into the state estimation process. In addition, we propose a computationally efficient variant, referred to as the reduced Bussgang-aided Kalman filter and, building upon it, introduce a deep learning-based approach for handling partially known models, termed the Bussgang-aided KalmanNet. In particular, the Bussgang-aided KalmanNet jointly uses a dithering technique and a gated recurrent unit (GRU) architecture to effectively mitigate the effects of 1-bit quantization and model mismatch. Through simulations on the Lorenz-Attractor model and the Michigan NCLT dataset, we demonstrate that our proposed methods achieve accurate state estimation performance even under highly nonlinear, mismatched models and 1-bit observations.