Multiple hallucinated deep network for image quality assessment

Document Type : Article

Authors

Department of Computer Science and Engineering, Shiraz University, Molla Sadra Ave., Shiraz, Fars, Iran

Abstract

Image Quality Assessment (IQA) refers to quantitatively evaluating the human perception about the quality of a distorted image. Blind IQA (BIQA) is a type of IQA that does not contain any reference or information about the distortion. Since the human brain has no information about the distortion type BIQA is more reliable and compatible with the real world. Traditional methods in this realm used some expert opinion, such as Natural Scene Statistics (NSS), to determine how far the distorted image is from the distribution of pristine samples.

By emerging deep networks, several IQA methods have been proposed to use their capability in automatic feature extraction. The main challenge of available deep models is that they need many annotated samples for training to reach a desirable outcome which is costly. In this paper, inspiring the Human Visual System (HVS), we propose a Generative Adversarial Network (GAN) based approach. To this end, we sample multiple images from a submanifold of pristine data manifold by conditioning the network on the corresponding distorted image. Also, NSS features are used to improve the network training and conduct the training process on the right track.

Keywords

References

References:
1. Kim, J., Zeng, H., Ghadiyaram, D., et al. "Deep convolutional neural models for picture-quality prediction: Challenges and solutions to data-driven image quality assessment", IEEE Signal Processing Magazine, 34(6), pp. 130-141 (2017).
2. Guo, Y., Ding, G., and Han, J. "Robust quantization for general similarity search", IEEE Transactions on Image Processing, 27(2), pp. 949-963 (2017).
3. Dong, C., Loy, C.C., He, K., et al. "Image superresolution using deep convolutional networks", IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), pp. 295-307 (2015).
4. Kim, J. and Lee, S. "Deep learning of human visual sensitivity in image quality assessment framework", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1676-1684 (2017).
5. Golestaneh, S. and Karam, L.J. "Reduced-reference quality assessment based on the entropy of dwt coefficients of locallyweighted gradient magnitudes", IEEE Transactions on Image Processing, 25(11), pp. 5293- 5303 (2016).
6. Ye, P., Kumar, J., Kang, L., et al. "Unsupervised feature learning framework for no-reference image quality assessment", In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1098-1105, IEEE (2012).
7. Lin, K.-Y. and Wang, G. "Hallucinated-iqa: Noreference image quality assessment via adversarial learning", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 732- 741 (2018).
8. Li, L., Zhu, H., Yang, G., et al. "Referenceless measure of blocking artifacts by tchebichef kernel analysis", IEEE Signal Processing Letters, 21(1), pp. 122-125 (2013).
9. Li, L., Lin, W., Wang, X., et al. "No-reference image blur assessment based on discrete orthogonal moments", IEEE Transactions on Cybernetics, 46(1), pp. 39-50 (2015).
10. Liu, H., Klomp, N., and Heynderickx, I. "A noreference metric for perceived ringing artifacts in images", IEEE Transactions on Circuits and Systems for Video Technology, 20(4), pp. 529-539 (2009).
11. Saad, M.A., Bovik, A.C., and Charrier, C. "Blind image quality assessment: A natural scene statistics approach in the DCT domain", IEEE Transactions on Image Processing, 21(8), pp. 3339-3352 (2012a).
12. Mittal, A., Moorthy, A.K., and Bovik, A.C. "Noreference image quality assessment in the spatial domain", IEEE Transactions on Image Processing, 21(12), pp. 4695-4708 (2012a).
13. Xue, W., Zhang, L., and Mou, X. "Learning without human scores for blind image quality assessment", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 995-1002 (2013).
14. Ghadiyaram, D. and Bovik, A.C. "Perceptual quality prediction on authentically distorted images using a bag of features approach", Journal of Vision, 17(1), pp. 32-32 (2017).
15. He, K., Zhang, X., Ren, S., et al. "Deep residual learning for image recognition", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778 (2016).
16. Ding, G., Chen, W., Zhao, S., et al. "Real-time scalable visual tracking via quadrangle kernelized correlation filters", IEEE Transactions on Intelligent Transportation Systems, 19(1), pp. 140-150 (2017).
17. Ding, G., Guo, Y., Chen, K., et al. "DECODE: Deep confidence network for robust image classification", IEEE Transactions on Image Processing, 28(8), pp. 3752-3765 (2019).
18. Kang, L., Ye, P., Li, Y., et al. "Convolutional neural networks for noreference image quality assessment", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1733-1740 (2014).
19. Yan, B., Bare, B., and Tan, W. "Naturalness-aware deep noreference image quality assessment", IEEE Transactions on Multimedia, 21(10), pp. 2603-2615 (2019).
20. Gu, K., Zhai, G., Yang, X., et al. "Using free energy principle for blind image quality assessment", IEEE Transactions on Multimedia, 17(1), pp. 50-63 (2014).
21. Sheikh, H. "LIVE image quality assessment database release 2", http://live. ece. utexas. edu/research/quality (2005).
22. Ponomarenko, N., Jin, L., Ieremeiev, O., et al. "Image database TID2013: Peculiarities, results and perspectives", Signal Processing: Image Communication, 30, pp. 57-77 (2015).
23. Deng, J., Dong, W., Socher, R., et al. "Imagenet: A large-scale hierarchical image database", In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, Ieee (2009).
24. Bianco, S., Celona, L., Napoletano, P., et al. "On the use of deep learning for blind image quality assessment", Signal, Image and Video Processing, 12(2), pp. 355-362 (2018).
25. Talebi, H. and Milanfar, P. "NIMA: Neural image assessment", IEEE Transactions on Image Processing, 27(8), pp. 3998-4011 (2018).
26. Hazrati Fard, S.M. and Hashemi, S. "Proposing a sparse representational based face verification system to run in a shortage of memory", Multimedia Tools and Applications, 79(3), pp. 2965-2985 (2020).
27. Berthelot, D., Milanfar, P., and Goodfellow, I. "Creating high resolution images with a latent adversarial generator", arXiv preprint arXiv:2003.02365 (2020).
28. Gu, S., Bao, J., Chen, D., et al. "GIQA: Generated image quality assessment", In European Conference on Computer Vision, pp. 369-385, Springer (2020).
29. Saad, M.A., Bovik, A.C., and Charrier, C. "Blind image quality assessment: A natural scene statistics approach in the DCT domain", IEEE Transactions on Image Processing, 21(8), pp. 3339-3352 (2012b).
30. Ye, P. and Doermann, D. "No-reference image quality assessment using visual codebooks", IEEE Transactions on Image Processing, 21(7), pp. 3129-3138 (2012).
31. Zhang, P., Zhou, W., Wu, L., et al. "SOM: Semantic obviousness metric for image quality assessment", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2394-2402 (2015a).
32. Moorthy, A.K. and Bovik, A.C. "A two-step framework for constructing blind image quality indices", IEEE Signal Processing Letters, 17(5), pp. 513-516 (2010).
33. Zhang, W., Zhai, K., Zhai, G., et al. "Learning to blindly assess image quality in the laboratory and wild", In 2020 IEEE International Conference on Image Processing (ICIP), pp. 111-115, IEEE (2020).
34. Arjovsky, M. and Chintala, S. "Wasserstein GAN", arXiv preprint arXiv:1701.07875 (2017).
35. Zhang, Y., Tian, Y., Kong, Y., et al. "Residual dense network for image super-resolution", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472-2481 (2018).
36. Liu, D., Wen, B., Fan, Y., et al. "Non-local recurrent network for image restoration", In Advances in Neural Information Processing Systems, pp. 1673-1682 (2018).
37. Dai, T., Cai, J., Zhang, Y., et al. "Second-order attention network for single image super-resolution", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11065- 11074 (2019).
38. Lim, B., Son, S., Kim, H., et al. "Enhanced deep residual networks for single image super-resolution", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136- 144 (2017).
39. Newell, A., Yang, K., and Deng, J. "Stacked hourglass networks for human pose estimation", In European Conference on Computer Vision, pp. 483-499, Springer (2016).
40. Mittal, A., Soundararajan, R., and Bovik, A.C. "Making a completely blind image quality analyzer", IEEE Signal Process. Lett., 20(3), pp. 209-212 (2013).
41. Zhang, L., Zhang, L., and Bovik, A.C. "A feature enriched completely blind image quality evaluator", IEEE Transactions on Image Processing, 24(8), pp. 2579-2591 (2015b).
42. Ponomarenko, N., Ieremeiev, O., Lukin, V., et al. "Color image database TID2013: Peculiarities and preliminary results", In European Workshop on Visual Information Processing (EUVIP), pp. 106-111, IEEE (2013).
43. Ponomarenko, N., Lukin, V., Zelensky, A., et al. "TID2008-a database for evaluation of full-reference visual quality assessment metrics", Advances of Modern Radioelectronics, 10(4), pp. 30-45 (2009).
44. Sheikh, H.R., Sabir, M.F., and Bovik, A.C. "A statistical evaluation of recent full reference image quality assessment algorithms", IEEE Transactions on Image Processing, 15(11), pp. 3440-3451 (2006).
45. Bosse, S., Maniry, D., Muller, K.-R., et al. "Deep neural networks for noreference and full-reference image quality assessment", IEEE Transactions on Image Processing, 27(1), pp. 206-219 (2017).
46. Gulrajani, I., Ahmed, F., Arjovsky, M., et al. "Improved training of wasserstein GANs", arXiv preprint arXiv:1704.00028 (2017).
47. Saad, M.A., Bovik, A.C., and Charrier, C. "A DCT statistics-based blind image quality index", IEEE Signal Processing Letters, 17(6), pp. 583-586 (2010).
48. Mittal, A., Moorthy, A.K., and Bovik, A.C. "Noreference image quality assessment in the spatial domain", IEEE Transactions on Image Processing, 21(12), pp. 4695-4708 (2012b).
Scientia Iranica
Volume 30, Issue 2
Transactions on Computer Science & Engineering and Electrical Engineering (D)
March and April 2023
Pages 492-505

Files

History

  • Receive Date: 13 October 2021
  • Revise Date: 02 June 2022
  • Accept Date: 21 November 2022

Share

How to cite

Statistics