Token Homogenization under Positional Bias

This paper investigates token homogenization - the convergence of token representations toward uniformity across transformer layers and its relationship to positional bias in large language models. We empirically examine whether homogenization occurs and how positional bias amplifies this effect. Through layer-wise similarity analysis and controlled experiments, we demonstrate that tokens systematically lose distinctiveness during processing, particularly when biased toward extremal positions. Our findings confirm both the existence of homogenization and its dependence on positional attention mechanisms.

Experience Report: Towards Moving Things with Types -- Helping Logistics Domain Experts to Control Cyber-Physical Systems with Type-Based Synthesis

One of the ultimate goals of software engineering is to leave virtual spaces and move real things. We take one step toward supporting users with this goal by connecting a type-based synthesis algorithm, (CL)S, and its IDE to a logistics lab environment. The environment is built and used by domain experts, who have little or no training in formal methods, and need to cope with large spaces of software, hardware and problem specific solution variability. It consists of a number of Cyber-Physical Systems (CPS), including wheel-driven robots as well as flying drones, and it has laser-based support to visualize their possible movements. Our work describes results on an experiment integrating the latter with (CL)S. Possibilities and challenges of working in the domain of logistics and in cooperation with its experts are outlined. Future research plans are presented and an invitation is made to join the effort of building better, formally understood, development tools for CPS-enabled industrial environments.