Focus Your Attention: Towards Data-Intuitive Lightweight Vision Transformers

The evolution of Vision Transformers has led to their widespread adaptation to different domains. Despite large-scale success, there remain significant challenges including their reliance on extensive computational and memory resources for pre-training on huge datasets as well as difficulties in task-specific transfer learning. These limitations coupled with energy inefficiencies mainly arise due to the computation-intensive self-attention mechanism. To address these issues, we propose a novel Super-Pixel Based Patch Pooling (SPPP) technique that generates context-aware, semantically rich, patch embeddings to effectively reduce the architectural complexity and improve efficiency. Additionally, we introduce the Light Latent Attention (LLA) module in our pipeline by integrating latent tokens into the attention mechanism allowing cross-attention operations to significantly reduce the time and space complexity of the attention module. By leveraging the data-intuitive patch embeddings coupled with dynamic positional encodings, our approach adaptively modulates the cross-attention process to focus on informative regions while maintaining the global semantic structure. This targeted attention improves training efficiency and accelerates convergence. Notably, the SPPP module is lightweight and can be easily integrated into existing transformer architectures. Extensive experiments demonstrate that our proposed architecture provides significant improvements in terms of computational efficiency while achieving comparable results with the state-of-the-art approaches, highlighting its potential for energy-efficient transformers suitable for edge deployment. (The code is available on our GitHub repository: https://github.com/zser092/Focused-Attention-ViT).

A Simplified Un-Supervised Learning Based Approach for Ink Mismatch Detection in Handwritten Hyper-Spectral Document Images

Hyper-spectral imaging has become the latest trend in the field of optical imaging systems. Among various other applications, hyper-spectral imaging has been widely used for analysis of printed and handwritten documents. This paper proposes an efficient technique for estimating the number of different but visibly similar inks present in a Hyper spectral Document Image. Our approach is based on un-supervised learning and does not require any prior knowledge of the dataset. The algorithm was tested on the iVision HHID dataset and has achieved comparable results with the state of the algorithms present in the literature. This work can prove to be effective when employed during the early stages of forgery detection in Hyper-spectral Document Images.