Adaptive Transform Domain Image SR
Via Orthogonally Regularized Deep Networks

ORDSR


ABSTRACT

Deep learning methods, in particular trained Convolutional Neural Networks (CNNs) have recently been shown to produce compelling state-of-the-art results for single image Super-Resolution (SR). Invariably, a CNN is learned to map the low resolution (LR) image to its corresponding high resolution (HR) version in the spatial domain. Aiming for faster inference and more efficient solutions than solving the SR problem in the spatial domain, we propose a novel network structure for learning the SR mapping function in an image transform domain, specifically the Discrete Cosine Transform (DCT). As a first contribution, we show that DCT can be integrated into the network structure as a Convolutional DCT (CDCT) layer. We further extend the network to allow the CDCT layer to become trainable (i.e. optimizable). Because this layer represents an image transform, we enforce pairwise orthogonality constraints on the individual basis functions/filters. This Orthogonally Regularized Deep SR network (ORDSR) simplifies the SR task by taking advantage of image transform domain while adapting the design of transform basis to the training image set. Experimental results show ORDSR achieves state-of-the-art SR image quality with fewer parameters than most of the deep CNN methods.

Technical Report

Please find details about CDCT layer and ORDSR in our technical report.

Please find step-by-step CDCT layer DCT and IDCT illustration in this slide.

Network Structure

Evaluations

SR Results

Testing image monarch.bmp from dataset Set 14. The assessments are displayed under the SR results from different methods as (PSNR, SSIM). ORDSR produces best results with less artifacts.

Test Code

Theversion of ORDSR testing code. You can find it in [this Github repository].

Any feedback is welcome.

Related Publications

  1. T. Guo, H. S. Mousavi and V. Monga , "Adaptive Transform Domain Image Super-resolution Via Orthogonally Regularized Deep Networks", in IEEE Transactions on Image Processing, vol. 28, no. 9, pp. 4685-4700, Sept. 2019. [IEEE Xplore]

  2. T. Guo, H. S. Mousavi and V. Monga, “Orthogonally Regularized Deep Networks For Image Super-resolution,” in the IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 1463-1467, April 2018, Calgary, AB, Canada. [IEEE Xplore]

Selected References

  1. J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR Oral), June 2016 [paper].

  2. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [paper].

  3. C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in Computer Vision–ECCV 2014, pp. 184–199, Springer, 2014. [paper].

  4. C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European Conference on Computer Vision, pp. 391–407, Springer, 2016.[ paper].

  5. J.-B. Huang, A. Singh, and N. Ahuja, “Single image superresolution from transformed self-exemplars,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206, 2015.[paper].

  6. Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep networks for image super-resolution with sparse prior,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 370–378, 2015. [paper].

  7. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.[paper].

Email
ipal.psu@gmail.com

Address
104 Electrical Engineering East,
University Park, PA 16802, USA

Lab Phone:
814-863-7810
814-867-4564