Abstract:Vision-Language-Action (VLA) models offer a compelling framework for tackling complex robotic manipulation tasks, but they are often expensive to train. In this paper, we propose a novel VLA approach that leverages the competitive performance of Vision Language Models (VLMs) on 2D images to directly infer robot end-effector poses in image frame coordinates. Unlike prior VLA models that output low-level controls, our model predicts trajectory waypoints, making it both more efficient to train and robot embodiment agnostic. Despite its lightweight design, our next-token prediction architecture effectively learns meaningful and executable robot trajectories. We further explore the underutilized potential of incorporating depth images, inference-time techniques such as decoding strategies, and demonstration-conditioned action generation. Our model is trained on a simulated dataset and exhibits strong sim-to-real transfer capabilities. We evaluate our approach using a combination of simulated and real data, demonstrating its effectiveness on a real robotic system.
Abstract:We propose a real-world dataset of stereoscopic videos for color-mismatch correction. It includes real-world distortions achieved using a beam splitter. Our dataset is larger than any other for this task. We compared eight color-mismatch-correction methods on artificial and real-world datasets and showed that local methods are best suited to artificial distortions and that global methods are best suited to real-world distortions. Our efforts improved on the latest local neural-network method for color-mismatch correction in stereoscopic images, making it work faster and better on both artificial and real-world distortions.