Information Processing & Algorithm Laboratory

Recent advances have seen a surge of deep learning approaches for image super-resolution. Invariably, a network, e.g. a deep convolutional neural network (CNN) or auto-encoder is trained to learn the relationship between low and hi-resolution image patches. Recognizing that a wavelet transform provides a “coarse” as well as “detail” separation of image content, we design a deep CNN to predict the “missing details” of wavelet coefficients of the low-resolution images to obtain the Super-Resolution (SR) results, which we name Deep Wavelet Super-Resolution (DWSR). Out network is trained in the wavelet domain with four input and output channels respectively. The input comprises of 4 sub-bands of the low resolution wavelet coefficients and outputs are residuals (missing details) of 4 sub-bands of high resolution wavelet coefficients. Wavelet coefficients and wavelet residuals are used as input and outputs of our network to further enhance the sparsity of activation maps. A key benefit of such a design is that it greatly reduces the training burden of learning the network that reconstructs low frequency details. The output prediction is added to the input to form the final SR wavelet coefficients. Then the inverse 2d discrete wavelet transformation is applied to transform the predicted details and generate the SR results. We show that DWSR is computationally simpler and yet produces competitive and often better results than state-of-the-art alternatives.

The python version of DWSR testing code. You can find it in different scales at [x2] [x3] [x4]. And request training code from [here] by providing basic usage information.

Any feedback is welcome.

Testing image 19 and 92 from dataset Urban 100. The assessments are displayed under the SR results from different methods as (PSNR, SSIM). DWSR produces best results with less artifacts.

Co-author of R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang, et al., “Ntire 2017 challenge on single image super-resolution: Methods and results,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017. [CVF]
T. Guo, H. S. Mousavi, T. H. Vu, V. Monga, “Deep Wavelet Coefficients Prediction for Super-resolution,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017. [IEEE Xplore]

J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR Oral), June 2016 [paper].
D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [paper].
C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in Computer Vision–ECCV 2014, pp. 184–199, Springer, 2014. [paper].
C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European Conference on Computer Vision, pp. 391–407, Springer, 2016.[ paper].
J.-B. Huang, A. Singh, and N. Ahuja, “Single image superresolution from transformed self-exemplars,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206, 2015.[paper].
Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep networks for image super-resolution with sparse prior,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 370–378, 2015. [paper].
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.[paper].

Deep Wavelet Coefficients Prediction
for Super-resolution

DWSR

For NTIRE 2017 Competition - CVPR Workshop

ABSTRACT

Code

Network Structure

Evaluations

SR Results

Related Publications

Selected References

Deep Wavelet Coefficients Predictionfor Super-resolution