Prior-Guided Deep Neural Networks for Image Restoration Tasks

Abstract

In recent years, deep learning-based image restoration neural networks have become the methodology of choice, outperforming their traditional counterparts and being gradually adopted in all systems that process, store, or display images. Despite the apparent successes of these networks, rooms for improvements still exist, as demonstrated by this thesis; particularly in terms of restoring sharp and clean high frequency details and textures, which still remains a challenge for existing deep learning-based image restoration methods. To overcome the said weaknesses of existing methods, we study and try to identify the common root causes for the lack of desired sharpness and clarity in the output images of those learned deep restoration models against various types of degradations. Our observations point to the necessity of incorporating into neural restoration models the priors on viewer desired high frequency constructs. In our study, we introduce several novel techniques to investigate and utilize informative high-frequency priors. These techniques include: (i) inducing convolutional neural networks' filters to extract valuable frequency information from images via a pre-designed filter bank, (ii) modifying the loss function of the restoration model during training to prioritize high-frequency textures, (iii) incorporating an auxiliary loss function on the metadata to shape the neural network outputs according to the prior knowledge of the input images, and (iv) integrating the desired priors within the model architecture. In our first work, we propose to put additional hand-crafted constraints on the filters in convolutional neural networks to train them in faster convergence and better performance. We encourage the convolutional neural network kernels to conform to common spatial structures and features of natural images. The proposed regularization technique aims to include structural image priors of traditional filter banks to improve the robustness and generality of convolutional neural network solutions. The usefulness of this approach is not limited to image restoration; it can also be applied to other image processing and computer vision tasks. In our second work, we design a new training strategy to adjust the loss function of image restoration neural networks adaptively. By formulating a classical optimization problem, we are able to pinpoint the complex textures for the image restoration neural network to recover. The resulting textures can be used in the loss function of the neural network during training, which leads to better estimation of the high-frequency textures and details. We have also researched on the problem of improving optical flow estimation. Specifically, we investigate how to increase the accuracy of deep learning based optical flow estimators. We develop a test time adaptive method that efficiently uses motion vector maps provided in H.264 encoders to alleviate the domain shift problem at the inference time. It is critical to handle the out-of-domain inferences as deep learning-based optical flow estimators are mostly trained on synthetic datasets. Finally, we propose a novel asymmetric image compression system with a high throughput real-time encoder and a heavy-duty neural network decoder that is responsible for high rate-distortion performance. The key technical development of the above asymmetric coding system is a special image restoration network that can remove compression artifacts due to the aggressively streamlined encoder.