Get our free extension to see links to code for papers anywhere online!

Chrome logo Add to Chrome

Firefox logo Add to Firefox

"Topic": models, code, and papers

A Survey of Appearance Models in Visual Object Tracking

Mar 20, 2013
Xi Li, Weiming Hu, Chunhua Shen, Zhongfei Zhang, Anthony Dick, Anton van den Hengel

Visual object tracking is a significant computer vision task which can be applied to many domains such as visual surveillance, human computer interaction, and video compression. In the literature, researchers have proposed a variety of 2D appearance models. To help readers swiftly learn the recent advances in 2D appearance models for visual object tracking, we contribute this survey, which provides a detailed review of the existing 2D appearance models. In particular, this survey takes a module-based architecture that enables readers to easily grasp the key points of visual object tracking. In this survey, we first decompose the problem of appearance modeling into two different processing stages: visual representation and statistical modeling. Then, different 2D appearance models are categorized and discussed with respect to their composition modules. Finally, we address several issues of interest as well as the remaining challenges for future research on this topic. The contributions of this survey are four-fold. First, we review the literature of visual representations according to their feature-construction mechanisms (i.e., local and global). Second, the existing statistical modeling schemes for tracking-by-detection are reviewed according to their model-construction mechanisms: generative, discriminative, and hybrid generative-discriminative. Third, each type of visual representations or statistical modeling techniques is analyzed and discussed from a theoretical or practical viewpoint. Fourth, the existing benchmark resources (e.g., source code and video datasets) are examined in this survey.

* Appearing in ACM Transactions on Intelligent Systems and Technology, 2013 

  Access Paper or Ask Questions

Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling

Mar 10, 2022
Tengpeng Li, Hanli Wang, Bin He, Chang Wen Chen

As a technically challenging topic, visual storytelling aims at generating an imaginary and coherent story with narrative multi-sentences from a group of relevant images. Existing methods often generate direct and rigid descriptions of apparent image-based contents, because they are not capable of exploring implicit information beyond images. Hence, these schemes could not capture consistent dependencies from holistic representation, impairing the generation of reasonable and fluent story. To address these problems, a novel knowledge-enriched attention network with group-wise semantic model is proposed. Three main novel components are designed and supported by substantial experiments to reveal practical advantages. First, a knowledge-enriched attention network is designed to extract implicit concepts from external knowledge system, and these concepts are followed by a cascade cross-modal attention mechanism to characterize imaginative and concrete representations. Second, a group-wise semantic module with second-order pooling is developed to explore the globally consistent guidance. Third, a unified one-stage story generation model with encoder-decoder structure is proposed to simultaneously train and infer the knowledge-enriched attention network, group-wise semantic module and multi-modal story generation decoder in an end-to-end fashion. Substantial experiments on the popular Visual Storytelling dataset with both objective and subjective evaluation metrics demonstrate the superior performance of the proposed scheme as compared with other state-of-the-art methods.

  Access Paper or Ask Questions

Mobile Device Association and Resource Allocation in Small-Cell IoT Networks with Mobile Edge Computing and Caching

Feb 26, 2022
Tianqing Zhou, Yali Yue, Dong Qin, Xuefang Nie, Xuan Li, Chunguo Li

To meet the need of computation-sensitive (CS) and high-rate (HR) communications, the framework of mobile edge computing and caching has been widely regarded as a promising solution. When such a framework is implemented in small-cell IoT (Internet of Tings) networks, it is a key and open topic how to assign mobile edge computing and caching servers to mobile devices (MDs) with CS and HR communications. Since these servers are integrated into small base stations (BSs), the assignment of them refers to not only the BS selection (i.e., MD association), but also the selection of computing and caching modes. To mitigate the network interference and thus enhance the system performance, some highly-effective resource partitioning mechanisms are introduced for access and backhaul links firstly. After that a problem with minimizing the sum of MDs' weighted delays is formulated to attain a goal of joint MD association and resource allocation under limited resources. Considering that the MD association and resource allocation parameters are coupling in such a formulated problem, we develop an alternating optimization algorithm according to the coalitional game and convex optimization theorems. To ensure that the designed algorithm begins from a feasible initial solution, we develop an initiation algorithm according to the conventional best channel association, which is used for comparison and the input of coalition game in the simulation. Simulation results show that the algorithm designed for minimizing the sum of MDs' weighted delays may achieve a better performance than the initiation (best channel association) algorithm in general.

  Access Paper or Ask Questions

Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training

Jan 11, 2022
Yehao Li, Jiahao Fan, Yingwei Pan, Ting Yao, Weiyao Lin, Tao Mei

Vision-language pre-training has been an emerging and fast-developing research topic, which transfers multi-modal knowledge from rich-resource pre-training task to limited-resource downstream tasks. Unlike existing works that predominantly learn a single generic encoder, we present a pre-trainable Universal Encoder-DEcoder Network (Uni-EDEN) to facilitate both vision-language perception (e.g., visual question answering) and generation (e.g., image captioning). Uni-EDEN is a two-stream Transformer based structure, consisting of three modules: object and sentence encoders that separately learns the representations of each modality, and sentence decoder that enables both multi-modal reasoning and sentence generation via inter-modal interaction. Considering that the linguistic representations of each image can span different granularities in this hierarchy including, from simple to comprehensive, individual label, a phrase, and a natural sentence, we pre-train Uni-EDEN through multi-granular vision-language proxy tasks: Masked Object Classification (MOC), Masked Region Phrase Generation (MRPG), Image-Sentence Matching (ISM), and Masked Sentence Generation (MSG). In this way, Uni-EDEN is endowed with the power of both multi-modal representation extraction and language modeling. Extensive experiments demonstrate the compelling generalizability of Uni-EDEN by fine-tuning it to four vision-language perception and generation downstream tasks.

* ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 

  Access Paper or Ask Questions

Challenges of Artificial Intelligence -- From Machine Learning and Computer Vision to Emotional Intelligence

Jan 05, 2022
Matti Pietikäinen, Olli Silven

Artificial intelligence (AI) has become a part of everyday conversation and our lives. It is considered as the new electricity that is revolutionizing the world. AI is heavily invested in both industry and academy. However, there is also a lot of hype in the current AI debate. AI based on so-called deep learning has achieved impressive results in many problems, but its limits are already visible. AI has been under research since the 1940s, and the industry has seen many ups and downs due to over-expectations and related disappointments that have followed. The purpose of this book is to give a realistic picture of AI, its history, its potential and limitations. We believe that AI is a helper, not a ruler of humans. We begin by describing what AI is and how it has evolved over the decades. After fundamentals, we explain the importance of massive data for the current mainstream of artificial intelligence. The most common representations for AI, methods, and machine learning are covered. In addition, the main application areas are introduced. Computer vision has been central to the development of AI. The book provides a general introduction to computer vision, and includes an exposure to the results and applications of our own research. Emotions are central to human intelligence, but little use has been made in AI. We present the basics of emotional intelligence and our own research on the topic. We discuss super-intelligence that transcends human understanding, explaining why such achievement seems impossible on the basis of present knowledge,and how AI could be improved. Finally, a summary is made of the current state of AI and what to do in the future. In the appendix, we look at the development of AI education, especially from the perspective of contents at our own university.

* 234 pages. Published as an electronic publication at the University of Oulu, Finland, in December 2021, ISBN: 978-952-62-3199-0 link 

  Access Paper or Ask Questions

Show and Write: Entity-aware News Generation with Image Information

Dec 11, 2021
Zhongping Zhang, Yiwen Gu, Bryan A. Plummer

Automatically writing long articles is a complex and challenging language generation task. Prior work has primarily focused on generating these articles using human-written prompt to provide some topical context and some metadata about the article. That said, for many applications, such as generating news stories, these articles are often paired with images and their captions or alt-text, which in turn are based on real-world events and may reference many different named entities that are difficult to be correctly recognized and predicted by language models. To address these two problems, this paper introduces an Entity-aware News Generation method with Image iNformation, Engin, to incorporate news image information into language models. Engin produces news articles conditioned on both metadata and information such as captions and named entities extracted from images. We also propose an Entity-aware mechanism to help our model better recognize and predict the entity names in news. We perform experiments on two public large-scale news datasets, GoodNews and VisualNews. Quantitative results show that our approach improves article perplexity by 4-5 points over the base models. Qualitative results demonstrate the text generated by Engin is more consistent with news images. We also perform article quality annotation experiment on the generated articles to validate that our model produces higher-quality articles. Finally, we investigate the effect Engin has on methods that automatically detect machine-generated articles.

  Access Paper or Ask Questions

Improving Embedded Knowledge Graph Multi-hop Question Answering by introducing Relational Chain Reasoning

Oct 25, 2021
Weiqiang Jin, Hang Yu, Xi Tao, Ruiping Yin

Knowledge Base Question Answering (KBQA) aims to answer userquestions from a knowledge base (KB) by identifying the reasoningrelations between topic entity and answer. As a complex branchtask of KBQA, multi-hop KGQA requires reasoning over multi-hop relational chains preserved in KG to arrive at the right answer.Despite the successes made in recent years, the existing works onanswering multi-hop complex question face the following challenges: i) suffering from poor performances due to the neglect of explicit relational chain order and its relational types reflected inuser questions; ii) failing to consider implicit relations between thetopic entity and the answer implied in structured KG because oflimited neighborhood size constraints in subgraph retrieval based algorithms. To address these issues in multi-hop KGQA, we proposea novel model in this paper, namely Relational Chain-based Embed-ded KGQA (Rce-KGQA), which simultaneously utilizes the explicitrelational chain described in natural language questions and the implicit relational chain stored in structured KG. Our extensiveempirical study on two open-domain benchmarks proves that ourmethod significantly outperforms the state-of-the-art counterpartslike GraftNet, PullNet and EmbedKGQA. Comprehensive ablation experiments also verify the effectiveness of our method for multi-hop KGQA tasks. We have made our model's source code availableat Github:

* 10 pages, 5 figures; 36 references; This work was carried out during the first author's master time at Shanghai University. This work is also partially supported by an anonymous Natural Research Foundation. We would like to thank Hang Yu for providing helpful discussions and valuable recommendations 

  Access Paper or Ask Questions

Deep Learning Based Person Re-Identification Methods: A Survey and Outlook of Recent Works

Oct 10, 2021
Zhangqiang Ming, Min Zhu, Xiaoyong Wei, Xiangkun Wang, Jiamin Zhu, Junlong Cheng, Yong Yang

In recent years, with the increasing demand for public safety and the rapid development of intelligent surveillance networks, person re-identification (Re-ID) has become one of the hot research topics in the field of computer vision. Its main research goal is to retrieve persons with the same identity from different cameras. However, traditional person Re-ID methods require manual marking of person targets, which consumes a lot of labor costs. With the widespread application of deep neural networks in the field of computer vision, a large number of deep learning-based person Re-ID methods have emerged. To facilitate researchers to better understand the latest research results and future development trends in this field. Firstly, we compare traditional and deep learning-based person Re-ID methods, and present the main contributions of several person Re-ID surveys, and analyze their focused dimensions and shortcomings. Secondly, we focus on the current classic deep learning-based person Re-ID methods, including methods for deep metric learning, local feature learning, generate adversarial networks, sequence feature learning, and graph convolutional networks. Furthermore, we subdivide the above five categories according to their technique types, analyzing and comparing the experimental performance of part subcategories of the method. Finally, we discuss the challenges that remain in the field of person Re-ID field and prospects for future research directions.

* 21 pages, 13 figures 

  Access Paper or Ask Questions

Weight Evolution: Improving Deep Neural Networks Training through Evolving Inferior Weight Values

Oct 09, 2021
Zhenquan Lin, Kailing Guo, Xiaofen Xing, Xiangmin Xu

To obtain good performance, convolutional neural networks are usually over-parameterized. This phenomenon has stimulated two interesting topics: pruning the unimportant weights for compression and reactivating the unimportant weights to make full use of network capability. However, current weight reactivation methods usually reactivate the entire filters, which may not be precise enough. Looking back in history, the prosperity of filter pruning is mainly due to its friendliness to hardware implementation, but pruning at a finer structure level, i.e., weight elements, usually leads to better network performance. We study the problem of weight element reactivation in this paper. Motivated by evolution, we select the unimportant filters and update their unimportant elements by combining them with the important elements of important filters, just like gene crossover to produce better offspring, and the proposed method is called weight evolution (WE). WE is mainly composed of four strategies. We propose a global selection strategy and a local selection strategy and combine them to locate the unimportant filters. A forward matching strategy is proposed to find the matched important filters and a crossover strategy is proposed to utilize the important elements of the important filters for updating unimportant filters. WE is plug-in to existing network architectures. Comprehensive experiments show that WE outperforms the other reactivation methods and plug-in training methods with typical convolutional neural networks, especially lightweight networks. Our code is available at

* This paper is accepted by ACM Multimedia 2021 

  Access Paper or Ask Questions

Unsupervised Monocular Depth Perception: Focusing on Moving Objects

Aug 30, 2021
Hualie Jiang, Laiyan Ding, Zhenglong Sun, Rui Huang

As a flexible passive 3D sensing means, unsupervised learning of depth from monocular videos is becoming an important research topic. It utilizes the photometric errors between the target view and the synthesized views from its adjacent source views as the loss instead of the difference from the ground truth. Occlusion and scene dynamics in real-world scenes still adversely affect the learning, despite significant progress made recently. In this paper, we show that deliberately manipulating photometric errors can efficiently deal with these difficulties better. We first propose an outlier masking technique that considers the occluded or dynamic pixels as statistical outliers in the photometric error map. With the outlier masking, the network learns the depth of objects that move in the opposite direction to the camera more accurately. To the best of our knowledge, such cases have not been seriously considered in the previous works, even though they pose a high risk in applications like autonomous driving. We also propose an efficient weighted multi-scale scheme to reduce the artifacts in the predicted depth maps. Extensive experiments on the KITTI dataset and additional experiments on the Cityscapes dataset have verified the proposed approach's effectiveness on depth or ego-motion estimation. Furthermore, for the first time, we evaluate the predicted depth on the regions of dynamic objects and static background separately for both supervised and unsupervised methods. The evaluation further verifies the effectiveness of our proposed technical approach and provides some interesting observations that might inspire future research in this direction.

* Accepted by IEEE Sensors Journal. arXiv admin note: text overlap with arXiv:2003.01360 

  Access Paper or Ask Questions