Optimizing Multiple Feature Types for Image Inpainting in the Linear and Nonlinear Setting

Inpainting-based compression stores a carefully optimized subset of the full image data and reconstructs the missing data by inpainting. The quality of these lossy codecs depends decisively on the stored data. So far, these data consist almost exclusively of pixel locations along with their grayscale or color values. In the present paper, we present a general theory and a practical framework that allows to incorporate arbitrary features which can be described by linear or nonlinear equations. This includes e.g. derivatives of arbitrary order or local integrals. Our features can be combined with linear or nonlinear inpainting operators. Moreover, we present an algorithm that automatically optimizes the location and the type of the selected feature. The approach of allowing different types of optimized features turns inpainting-based compression into a more general, versatile and powerful paradigm. Our experiments report a consistent quality gain when increasing the number of feature types from 1 to 5. With the same amount of stored data, the average peak signal-to-noise improvement is 2.76 dB for harmonic (homogeneous diffusion) inpainting, and 1.82 dB for edge-enhancing diffusion inpainting.

Optimising Different Feature Types for Inpainting-based Image Representations

Inpainting-based image compression is a promising alternative to classical transform-based lossy codecs. Typically it stores a carefully selected subset of all pixel locations and their colour values. In the decoding phase the missing information is reconstructed by an inpainting process such as homogeneous diffusion inpainting. Optimising the stored data is the key for achieving good performance. A few heuristic approaches also advocate alternative feature types such as derivative data and construct dedicated inpainting concepts. However, one still lacks a general approach that allows to optimise and inpaint the data simultaneously w.r.t. a collection of different feature types, their locations, and their values. Our paper closes this gap. We introduce a generalised inpainting process that can handle arbitrary features which can be expressed as linear equality constraints. This includes e.g. colour values and derivatives of any order. We propose a fully automatic algorithm that aims at finding the optimal features from a given collection as well as their locations and their function values within a specified total feature density. Its performance is demonstrated with a novel set of features that also includes local averages. Our experiments show that it clearly outperforms the popular inpainting with optimised colour data with the same density.