Speaker
Description
Blind image restoration aims at recovering high-quality images from data deteriorated by artifacts due to noise, low resolution, motion blur, lossy compression or other, in the absence of any knowledge about the degradation function.
In normal settings, restoration relies on pairing an input degraded image with a reference image in good condition to provide prior information for building the restoration model.
However, the lack of access to high quality reference priors limits the practical applicability of blind restoration models, while the limited capacity of learning dictionaries restricts the diversity and richness of restored details.
We propose a blind super-resolution model to address the resolution upscaling scenario when the function for mapping high-resolution and low-resolution images is unknown.
Our solution relies on three training modules with different learning objectives: 1. a degradation-aware network (U-Net) to synthesize the high resolution image, given a low resolution image and the corresponding blur kernel; 2. a pre-trained generative adversarial network (GAN) to be used as prior, bridged to the U-Net by a latent code mapping and several channel-split spatial feature transforms (CS-SFTs); and 3. a rational polynomial image interpolation into deep convolutional neural networks (CNNs) to retain details.
The U-Net is designed to remove complex degradation and extract two kinds of features: i) latent features to map the input image to the closest latent code in the GAN, and ii) multi-resolution spatial features for modulating the GAN features which increase the receptive field for large blur elimination. Part of the GAN features that contribute to image fidelity are spatially modulated by the CS-SFTs, while left-out features that contribute to image realness directly pass through (Figure, top left). These features are subsequently mapped to the intermediate latent codes for improved preservation of the semantic properties. The latent codes are then passed through each convolution layer in the pre-trained GAN, and generate (GAN) features for each resolution scale. The model is trained on synthetic data produced using a known degradation model that approximates realistic low-quality images, and generalize to real images during inference.
We evaluated the performance of this method on degraded 2D 10-micron sections of the BigBrain dataset (https://ftp.bigbrainproject.org/). We observed improved detailed structure in the final restored images. Scores for spatial quality, naturalness, and perception-based image quality evaluation metrics were greatly improved overall for images restored using our approach compared to the original data.
