A Strong Baseline for Temporal Video-Text Alignment

In this paper, we consider the problem of temporally aligning the video and texts from instructional videos, specifically, given a long-term video, and associated text sentences, our goal is to determine their corresponding timestamps in the video. To this end, we establish a simple, yet strong model that adopts a Transformer-based architecture with all texts as queries, iteratively attending to the visual features, to infer the optimal timestamp. We conduct thorough experiments to investigate: (i) the effect of upgrading ASR systems to reduce errors from speech recognition, (ii) the effect of various visual-textual backbones, ranging from CLIP to S3D, to the more recent InternVideo, (iii) the effect of transforming noisy ASR transcripts into descriptive steps by prompting a large language model (LLM), to summarize the core activities within the ASR transcript as a new training dataset. As a result, our proposed simple model demonstrates superior performance on both narration alignment and procedural step grounding tasks, surpassing existing state-of-the-art methods by a significant margin on three public benchmarks, namely, 9.3% on HT-Step, 3.4% on HTM-Align and 4.7% on CrossTask. We believe the proposed model and dataset with descriptive steps can be treated as a strong baseline for future research in temporal video-text alignment. All codes, models, and the resulting dataset will be publicly released to the research community.

Towards Axiomatic, Hierarchical, and Symbolic Explanation for Deep Models

This paper proposes a hierarchical and symbolic And-Or graph (AOG) to objectively explain the internal logic encoded by a well-trained deep model for inference. We first define the objectiveness of an explainer model in game theory, and we develop a rigorous representation of the And-Or logic encoded by the deep model. The objectiveness and trustworthiness of the AOG explainer are both theoretically guaranteed and experimentally verified. Furthermore, we propose several techniques to boost the conciseness of the explanation.

Learning Baseline Values for Shapley Values

This paper aims to formulate the problem of estimating the optimal baseline values for the Shapley value in game theory. The Shapley value measures the attribution of each input variable of a complex model, which is computed as the marginal benefit from the presence of this variable w.r.t.its absence under different contexts. To this end, people usually set the input variable to its baseline value to represent the absence of this variable (i.e.the no-signal state of this variable). Previous studies usually determine the baseline values in an empirical manner, which hurts the trustworthiness of the Shapley value. In this paper, we revisit the feature representation of a deep model from the perspective of game theory, and define the multi-variate interaction patterns of input variables to define the no-signal state of an input variable. Based on the multi-variate interaction, we learn the optimal baseline value of each input variable. Experimental results have demonstrated the effectiveness of our method.