Beyond the Metaverse: XV (eXtended meta/uni/Verse)

We propose the term and concept XV (eXtended meta/omni/uni/Verse) as an alternative to, and generalization of, the shared/social virtual reality widely known as ``metaverse''. XV is shared/social XR. We, and many others, use XR (eXtended Reality) as a broad umbrella term and concept to encompass all the other realities, where X is an ``anything'' variable, like in mathematics, to denote any reality, X $\in$ \{physical, virtual, augmented, \ldots \} reality. Therefore XV inherits this generality from XR. We begin with a very simple organized taxonomy of all these realities in terms of two simple building blocks: (1) physical reality (PR) as made of ``atoms'', and (2) virtual reality (VR) as made of ``bits''. Next we introduce XV as combining all these realities with extended society as a three-dimensional space and taxonomy of (1) ``atoms'' (physical reality), (2) ``bits'' (virtuality), and (3) ``genes'' (sociality). Thus those working in the liminal space between Virtual Reality (VR), Augmented Reality (AR), metaverse, and their various extensions, can describe their work and research as existing in the new field of XV. XV includes the metaverse along with extensions of reality itself like shared seeing in the infrared, ultraviolet, and shared seeing of electromagnetic radio waves, sound waves, and electric currents in motors. For example, workers in a mechanical room can look at a pump and see a superimposed time-varying waveform of the actual rotating magnetic field inside its motor, in real time, while sharing this vision across multiple sites. Presented at IEEE Standards Association, Behind and Beyond the Metaverse: XV (eXtended meta/uni/Verse), Thurs. Dec. 8, 2022, 2:15-3:30pm, EST.

Ghost-free High Dynamic Range Imaging via Hybrid CNN-Transformer and Structure Tensor

Eliminating ghosting artifacts due to moving objects is a challenging problem in high dynamic range (HDR) imaging. In this letter, we present a hybrid model consisting of a convolutional encoder and a Transformer decoder to generate ghost-free HDR images. In the encoder, a context aggregation network and non-local attention block are adopted to optimize multi-scale features and capture both global and local dependencies of multiple low dynamic range (LDR) images. The decoder based on Swin Transformer is utilized to improve the reconstruction capability of the proposed model. Motivated by the phenomenal difference between the presence and absence of artifacts under the field of structure tensor (ST), we integrate the ST information of LDR images as auxiliary inputs of the network and use ST loss to further constrain artifacts. Different from previous approaches, our network is capable of processing an arbitrary number of input LDR images. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed method by comparing it with existing state-of-the-art HDR deghosting models. Codes are available at https://github.com/pandayuanyu/HSTHdr.

Learning to Kindle the Starlight

Capturing highly appreciated star field images is extremely challenging due to light pollution, the requirements of specialized hardware, and the high level of photographic skills needed. Deep learning-based techniques have achieved remarkable results in low-light image enhancement (LLIE) but have not been widely applied to star field image enhancement due to the lack of training data. To address this problem, we construct the first Star Field Image Enhancement Benchmark (SFIEB) that contains 355 real-shot and 854 semi-synthetic star field images, all having the corresponding reference images. Using the presented dataset, we propose the first star field image enhancement approach, namely StarDiffusion, based on conditional denoising diffusion probabilistic models (DDPM). We introduce dynamic stochastic corruptions to the inputs of conditional DDPM to improve the performance and generalization of the network on our small-scale dataset. Experiments show promising results of our method, which outperforms state-of-the-art low-light image enhancement algorithms. The dataset and codes will be open-sourced.

Multimodal Image Fusion based on Hybrid CNN-Transformer and Non-local Cross-modal Attention

The fusion of images taken by heterogeneous sensors helps to enrich the information and improve the quality of imaging. In this article, we present a hybrid model consisting of a convolutional encoder and a Transformer-based decoder to fuse multimodal images. In the encoder, a non-local cross-modal attention block is proposed to capture both local and global dependencies of multiple source images. A branch fusion module is designed to adaptively fuse the features of the two branches. We embed a Transformer module with linear complexity in the decoder to enhance the reconstruction capability of the proposed network. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed method by comparing it with existing state-of-the-art fusion models. The source code of our work is available at https://github.com/pandayuanyu/HCFusion.

Sentence Level Human Translation Quality Estimation with Attention-based Neural Networks

This paper explores the use of Deep Learning methods for automatic estimation of quality of human translations. Automatic estimation can provide useful feedback for translation teaching, examination and quality control. Conventional methods for solving this task rely on manually engineered features and external knowledge. This paper presents an end-to-end neural model without feature engineering, incorporating a cross attention mechanism to detect which parts in sentence pairs are most relevant for assessing quality. Another contribution concerns of prediction of fine-grained scores for measuring different aspects of translation quality. Empirical results on a large human annotated dataset show that the neural model outperforms feature-based methods significantly. The dataset and the tools are available.

Feature-Less End-to-End Nested Term Extraction

In this paper, we proposed a deep learning-based end-to-end method on the domain specified automatic term extraction (ATE), it considers possible term spans within a fixed length in the sentence and predicts them whether they can be conceptual terms. In comparison with current ATE methods, the model supports nested term extraction and does not crucially need extra (extracted) features. Results show that it can achieve high recall and a comparable precision on term extraction task with inputting segmented raw text.