AnimalMotionCLIP: Embedding motion in CLIP for Animal Behavior Analysis

Recently, there has been a surge of interest in applying deep learning techniques to animal behavior recognition, particularly leveraging pre-trained visual language models, such as CLIP, due to their remarkable generalization capacity across various downstream tasks. However, adapting these models to the specific domain of animal behavior recognition presents two significant challenges: integrating motion information and devising an effective temporal modeling scheme. In this paper, we propose AnimalMotionCLIP to address these challenges by interleaving video frames and optical flow information in the CLIP framework. Additionally, several temporal modeling schemes using an aggregation of classifiers are proposed and compared: dense, semi dense, and sparse. As a result, fine temporal actions can be correctly recognized, which is of vital importance in animal behavior analysis. Experiments on the Animal Kingdom dataset demonstrate that AnimalMotionCLIP achieves superior performance compared to state-of-the-art approaches.

XLR : a Lightweight Indicator to Measure Video QoE in IP Networks

A novel Key Quality Indicator for video delivery applications, XLR (piXel Loss Rate), is defined, characterized, and evaluated. The proposed indicator is an objective measure that captures the effects of transmission errors in the received video, has a good correlation with subjective Mean Opinion Scores, and provides comparable results with state-of-the-art Full-Reference metrics. Moreover, XLR can be estimated using only a lightweight analysis on the compressed bitstream, thus allowing a No-Reference operational method. Therefore, XLR can be used for measuring the quality of experience without latency at any network location. Thus, it is a relevant tool for network planning, specially in new high-demanding scenarios. The experiments carried out show the outstanding performance of its linear-dimension score and the reliability of the bitstream-based estimation.

Energy- and Quality-Aware Video Request Policy for Wireless Adaptive Streaming Clients

We present a straightforward, non-intrusive adaptive bit rate streaming segment quality selection policy which aims at extending battery lifetime during playback while limiting the impact on the user's quality of experience, thus benefiting consumers of video streaming services. This policy relies on the relationship between the available channel bandwidth and the bit rate of the representations in the quality ladder. It results from the characterization of the energy consumed by smartphones when running adaptive streaming client applications for different network connections (Wifi, 4G, and 5G) and the modeling of the energy consumed as a function of said relationship. Results show that a significant amount of energy can be saved (10 to 30%) by slightly modifying the default policy at the expense of a controlled reduction of video quality.

Perceptually Equivalent Resolution in Handheld Devices for Streaming Bandwidth Saving

We present the description, results, and analysis of the experiments conducted to find the equivalent resolution associated with handheld devices. That is, the resolution from which users stop perceiving quality improvements if better resolutions are presented to them in such devices. Thus, it is the maximum resolution that it is worth considering for generating and delivering video, as long as sequences are not too intensively compressed. Therefore, the detection of the equivalent resolutions allows for notable savings in bandwidth consumption. Subjective assessments have been carried out on fifty subjects using a set of video sequences of very different nature and four handheld devices with a broad range of screen dimensions. The results prove that the equivalent resolution in current handheld devices is 720p as higher resolutions are not valued by users.

Application-Layer FEC Scheme Configuration Optimization via Hybrid Simulated Annealing

An optimization technique based on an adapted combination of simulated annealing (SA) and tabu search (TS) is presented. This method aims at finding near-optimal unequal error protection (UEP) application-layer FEC code configurations. This approach is intended to smartly protect audio and video transmission over IP networks when hard time restrictions apply. The considered code is a UEP version of the widely-used Pro-MPEG COP3 codes enabling the use of several matrices of dissimilar size and thus of unequal recovery capability. Finding the optimal configuration frequently requires the evaluation of a large solution space. So, to fulfill the imposed constraints, SA is adapted to the specifics of the scenario. In particular, the annealing schedule is conditioned by the real-time restrictions. Furthermore, solution neighborhood structures are determined by a proposed definition of distance between protection configurations, which, jointly with TS, conditions the selection of candidate solutions. Experimental results show a significantly improved performance of the optimization process, which invariably fulfills imposed timing constraints, at the expense of a very low distortion increase, when compared to using exhaustive search. These results allow the use of UEP Pro-MPEG COP3 codes for protecting video and audio transmission, which distinctly outperforms the standard code in a wide range of scenarios.

A Video-Aware FEC-Based Unequal Loss Protection System for Video Streaming over RTP

A video-aware unequal loss protection (ULP) system for protecting RTP video streaming in bursty packet loss networks is proposed. Considering the relevance of the frame, the state of the channel, and the bitrate constraints of the protection bitstream, our algorithm selects in real time the most suitable frames to be protected through forward error protection (FEC) techniques. It benefits from a wise RTP encapsulation that allows working at a frame level without requiring any further process than that of parsing RTP headers. This makes our system straightforward and fast, perfectly suitable to be included in commercial video streaming servers. Simulation results show how our technique outperforms other proposed ULP schemes.

Solving the Team Orienteering Problem with Transformers

Route planning for a fleet of vehicles is an important task in applications such as package delivery, surveillance, or transportation. This problem is usually modeled as a Combinatorial Optimization problem named as Team Orienteering Problem. The most popular Team Orienteering Problem solvers are mainly based on either linear programming, which provides accurate solutions by employing a large computation time that grows with the size of the problem, or heuristic methods, which usually find suboptimal solutions in a shorter amount of time. In this paper, a multi-agent route planning system capable of solving the Team Orienteering Problem in a very fast and accurate manner is presented. The proposed system is based on a centralized Transformer neural network that can learn to encode the scenario (modeled as a graph) and the context of the agents to provide fast and accurate solutions. Several experiments have been performed to demonstrate that the presented system can outperform most of the state-of-the-art works in terms of computation speed. In addition, the code is publicly available at http://gti.ssr.upm.es/data.

Soccer line mark segmentation with stochastic watershed transform

Augmented reality applications are beginning to change the way sports are broadcast, providing richer experiences and valuable insights to fans. The first step of augmented reality systems is camera calibration, possibly based on detecting the line markings of the field of play. Most existing proposals for line detection rely on edge detection and Hough transform, but optical distortion and extraneous edges cause inaccurate or spurious detections of line markings. We propose a novel strategy to automatically and accurately segment line markings based on a stochastic watershed transform that is robust to optical distortions, since it makes no assumptions about line straightness, and is unaffected by the presence of players or the ball in the field of play. Firstly, the playing field as a whole is segmented completely eliminating the stands and perimeter boards. Then the line markings are extracted. The strategy has been tested on a new and public database composed by 60 annotated images from matches in five stadiums. The results obtained have proven that the proposed segmentation algorithm allows successful and precise detection of most line mark pixels.

FVV Live: A real-time free-viewpoint video system with consumer electronics hardware

FVV Live is a novel end-to-end free-viewpoint video system, designed for low cost and real-time operation, based on off-the-shelf components. The system has been designed to yield high-quality free-viewpoint video using consumer-grade cameras and hardware, which enables low deployment costs and easy installation for immersive event-broadcasting or videoconferencing. The paper describes the architecture of the system, including acquisition and encoding of multiview plus depth data in several capture servers and virtual view synthesis on an edge server. All the blocks of the system have been designed to overcome the limitations imposed by hardware and network, which impact directly on the accuracy of depth data and thus on the quality of virtual view synthesis. The design of FVV Live allows for an arbitrary number of cameras and capture servers, and the results presented in this paper correspond to an implementation with nine stereo-based depth cameras. FVV Live presents low motion-to-photon and end-to-end delays, which enables seamless free-viewpoint navigation and bilateral immersive communications. Moreover, the visual quality of FVV Live has been assessed through subjective assessment with satisfactory results, and additional comparative tests show that it is preferred over state-of-the-art DIBR alternatives.

FVV Live: Real-Time, Low-Cost, Free Viewpoint Video

FVV Live is a novel real-time, low-latency, end-to-end free viewpoint system including capture, transmission, synthesis on an edge server and visualization and control on a mobile terminal. The system has been specially designed for low-cost and real-time operation, only using off-the-shelf components.