"I May Not Have Articulated Myself Clearly": Diagnosing Dynamic Instability in LLM Reasoning at Inference Time

Reasoning failures in large language models (LLMs) are typically measured only at the end of a generation, yet many failures manifest as a process-level breakdown: the model "loses the thread" mid-reasoning. We study whether such breakdowns are detectable from inference-time observables available in standard APIs (token log probabilities), without any training or fine-tuning. We define a simple instability signal that combines consecutive-step distributional shift (JSD) and uncertainty (entropy), summarize each trace by its peak instability strength, and show that this signal reliably predicts failure. Across GSM8K and HotpotQA, instability strength predicts wrong answers with above-chance AUC and yields monotonic bucket-level accuracy decline at scale across model sizes. Crucially, we show that instability is not uniformly harmful: early instability can reflect subsequent stabilization and a correct final answer (\emph{corrective instability}), whereas late instability is more often followed by failure (\emph{destructive instability}), even at comparable peak magnitudes, indicating that recoverability depends not only on how strongly the distribution changes but also on when such changes occur relative to the remaining decoding horizon. The method is model-agnostic, training-free, and reproducible, and is presented as a diagnostic lens rather than a corrective or control mechanism.

Starfish: A Prototype for Universal Preprocessing and Text-Embedded Programming

We present a novel concept of universal text preprocessing and text-embedded programming (PTEP). Preprocessing and text-embedded programming has been widely used in programming languages and frameworks in a fragmented and mutually isolated way. The PTEP ideas can be found in the implementation of the \TeX\ typesetting system; they are prominent in PHP and similar web languages, and finally they are used in the Jupyter data science framework. This paper presents this area of research and related work in a more unified framework, and we describe the implemented system Starfish that satisfies the following novel principles of PTEP: universality, update and replace modes, flexiblity, configurability, and transparency. We describe the operating model and design of Starfish, which is an open-source system implementing universal preprocessing and text-embedded programming in Perl. The system is transparent and its design allows direct implementation in other programming languages as well.