ICTACT Journal on Image and Video Processing

Open Access Open Access  Restricted Access Subscription Access
Open Access Open Access Open Access  Restricted Access Restricted Access Subscription Access

Real-Time Video Scaling Based on Convolution Neural Network Architecture


Affiliations
1 Department of Electronics and Communication Engineering, Sir M. Visvesvaraya Institute of Technology, India
2 Department of Electronics and Communication Engineering, SJB Institute of Technology, India
     

   Subscribe/Renew Journal


In recent years, video super resolution techniques becomes mandatory requirements to get high resolution videos. Many super resolution techniques researched but still video super resolution or scaling is a vital challenge. In this paper, we have presented a real-time video scaling based on convolution neural network architecture to eliminate the blurriness in the images and video frames and to provide better reconstruction quality while scaling of large datasets from lower resolution frames to high resolution frames. We compare our outcomes with multiple exiting algorithms. Our extensive results of proposed technique RemCNN (Reconstruction error minimization Convolution Neural Network) shows that our model outperforms the existing technologies such as bicubic, bilinear, MCResNet and provide better reconstructed motioning images and video frames. The experimental results shows that our average PSNR result is 47.80474 considering upscale-2, 41.70209 for upscale-3 and 36.24503 for upscale-4 for Myanmar dataset which is very high in contrast to other existing techniques. This results proves our proposed model real-time video scaling based on convolution neural network architecture's high efficiency and better performance.

Keywords

Image Scaling, Convolution Neural, Network, Super Resolution.
Subscription Login to verify subscription
User
Notifications
Font Size

  • Wenzhe Shi et.al., “Cardiac Image Super-Resolution with Global Correspondence using Multi-Atlas Patchmatch”, Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 9-16, 2013

  • M.W. Thornton, P.M. Atkinson and D.A. Holland, “Sub-pixel Mapping of Rural Land Cover Objects from Fine Spatial Resolution Satellite Sensor Imagery using Super-Resolution Pixel-Swapping”, International Journal of Remote Sensing, Vol. 27, No. 3, pp. 473-491, 2006.

  • L. Zhang, H. Zhang, H. Shen and P. Li, “A Super-Resolution Reconstruction Algorithm for Surveillance Images”, Signal Processing, Vol. 90, No. 3, pp. 848-859, 2010.

  • T. Goto, T. Fukuoka, F. Nagashima, S. Hirano and M. Sakurai, “Super-Resolution System for 4K-HDTV”, Proceedings of 22nd International Conference on Pattern Recognition, pp. 4453-4458, 2014.

  • B.K. Gunturk, A.U. Batur, Y. Altunbasak, M.H. Hayes and R.M. Mersereau, “Eigenface-Domain Super-Resolution for Face Recognition”, IEEE Transactions on Image Processing, Vol. 12, No. 5, pp. 597-606, 2003.

  • A. Krizhevsky, I. Sutskever, and G.E. Hinton, “Imagenet Classification with Deep Convolutional Neural Networks”, Proceedings of Neural Information Processing Systems, pp. 1097-1105, 2012.

  • C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, “Going Deeper with Convolutions”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2014.

  • J. Deng, W. Dong, R. Socher and L. Li, “A Large-Scale Hierarchical Image Database”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.

  • Harmonic Inc, Avaailable at: http://www.harmonicinc.com/resources/videos/4kvideo-clip-center, Accessed on 2014.

  • Weisheng Dong, Lei Zhang, Guangming Shi and Xiaolin Wu, “Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization”, IEEE Transactions on Image Processing, Vol. 20, No. 7, pp. 1838-1857, 2011.

  • Marshall F. Tappen, Bryan C. Russell and William T. Freeman, “Exploiting the Sparse Derivative Prior for Super-Resolution and Image Demosaicing”, Proceedings of IEEE Workshop on Statistical and Computational Theories of Vision, pp. 1-28, 2003.

  • L. Zhang et al., “FSIM: A Feature Similarity Index for Image Quality Assessment”, IEEE Transactions on Image Processing, Vol. 20, No. 8, pp. 2378-2386, 2011.

  • J. Zhong, B. Yang, Y. Li, F. Zhong and Z. Chen, “Image Fusion and Super-Resolution with Convolutional Neural Network”, Proceedings of Chinese Conference on Pattern Recognition, pp. 78-88, 2016.

  • L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang, L. Zhang, “Image Super-Resolution: The Techniques Applications and Future”, Signal Processing, Vol. 128, pp. 389-408, 2016 [15] Z. Zhao, L. Song, R. Xie and X. Yang, “GPU Accelerated High-Quality Video/Image Super-Resolution”, Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, pp. 1-4, 2016.

  • A. Kappeler, S. Yoo, Q. Dai and A.K. Katsaggelos, “Video Super-Resolution With Convolutional Neural Networks”, IEEE Transactions on Computational Imaging, Vol. 2, No. 2, pp. 109-122, 2016.

  • G.Y. Youm, S.H. Bae and M. Kim, “Image Super-Resolution based on Convolution Neural Networks using Multi-Channel Input”, Proceedings of IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop, pp. 1-5, 2016.

  • Y. Xie, J. Xiao, T. Tillo, Y. Wei and Y. Zhao, “3D Video Super-Resolution using Fully Convolutional Neural Networks”, Proceedings of IEEE International Conference on Multimedia and Expo, pp. 1-6, 2016.

  • W. Shi et al., “Real-Time Single Image and Video Super-Resolution using an Efficient Sub-Pixel Convolutional Neural Network”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874-1883, 2016.

  • Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition”, Proceedings of European Conference on Computer Vision, pp. 346-361, 2014.

  • Alex Krizhevsky, Iiya Sutskever and Geoffrey E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks”, Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.

  • W. Ouyang, P. Luo, X. Zeng, S. Qiu, Y. Tian, H. Li, S. Yang, Z. Wang, Y. Xiong, C. Qian, “Deepid-Net: Multi-Stage and Deformable Deep Convolutional Neural Networks for Object Detection”, Proceedings of Computer Vision and Pattern Recognition, pp. 1-13, 2014.

  • Wanli Ouyang and Xiaogang Wang, “Joint Deep Learning for Pedestrian Detection”, Proceedings of IEEE International Conference on Computer Vision, pp. 20562063, 2013.

  • Yi Sun, Yuheng Chen, Xiaogang Wang and Xiaoou Tang, “Deep Learning Face Representation by Joint Identification-Verification”, Proceedings of Advances in Neural Information Processing Systems, pp. 1988-1996, 2014.

  • Christian Szegedy, Scott Reed, Dumitru Erhan, Dragomir Anguelov and Sergey Ioffe, “Scalable, Highquality Object Detection”, Proceedings of IEEE International Conference on Computer Vision, pp. 1-10, 2014.

  • V. Nair and G.E. Hinton, “Rectified Linear Units Improve Restricted Boltzmann Machines”, Proceedings of International Conference on Machine Learning, pp. 807814, 2010.

  • Matthew D. Zeiler and Rob Fergus, “Visualizing and Understanding Convolutional Networks”, Proceedings of European Conference on Computer Vision, pp. 818-833, 2014.

  • Byung Cheol Song, Shin-Cheol Jeong and Yanglim Choi, “Video Super-Resolution algorithm using Bi-Directional Overlapped Block Motion Compensation and on-the-Fly Dictionary Training”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 21, No. 3, pp. 274-285, 2011.

  • Edson Mintsu Hung, Ricardo L. de Queiroz, Fernanda Brandi, Karen França de Oliveira and Debargha Mukherjee, “Video Super-Resolution using Codebooks Derived from Key-Frames”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 22, No. 9, pp. 1321-1331, 2012.

  • Zhengdong Zhang and Vivienne Sze, “Fast: Free Adaptive Super-Resolution via Transfer for Compressed Videos”, Proceedings of Computer Vision and Pattern Recognition, pp. 1-17, 2016.

  • Jing Zhang, Yang Cao, Zheng-Jun Zha, Zhigang Zheng, Chang Wen Chen and Zengfu Wang, “A Unified Scheme for Super-Resolution and Depth Estimation from Asymmetric Stereoscopic Video”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 26, No. 3, pp. 479-493, 2016.

  • Zhi Jin, Tammam Tillo, Chao Yao, Jimin Xiao and Yao Zhao, “Virtual-View-Assisted Video Super-Resolution and Enhancement”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 26, No. 3, pp. 467-478, 2016.

  • Kamal Nasrollahi and Thomas B. Moeslund, “Super-Resolution: A Comprehensive Survey”, Machine Vision and Applications, Vol. 25, No. 6, pp. 1423-1468, 2014.

  • S. Farsiu, M.D. Robinson, M. Elad and P. Milanfar, “Fast and Robust Multiframe Super Resolution”, IEEE Transactions on Image Processing, Vol. 13, No. 10, pp. 1327-1344, 2004.

  • M. Protter, M. Elad, H. Takeda and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction”, IEEE Transactions on Image Processing, Vol. 18, No. 1, pp. 36-51, 2009.

  • S. Baker and T. Kanade, “Limits on Super-Resolution and How to Break Them”, IEEE Transaction on Pattern Analysis Machine Intelligence, Vol. 24, No. 9, pp. 11671183, 2002.

  • Zhouchen Lin and Heung-Yeung Shum, “Fundamental Limits of Reconstruction based Super Resolution Algorithms under Local Translation”, IEEE Transaction on Pattern Analysis Machine Intelligence, Vol. 26, No. 1, pp. 83-97, 2004.

  • Matthew D. Zeiler and Rob Fergus, “Visualizing and Understanding Convolutional Networks”, Proceedings of European Conference on Computer Vision, pp. 818-833, 2014.

  • W.T. Freeman, T.R. Jones and E.C. Pasztor, “Example-based Super Resolution” , IEEE Computer Graphics and Applications, Vol. 22, No. 2, pp. 56-65, 2002.

  • C. Liu and D. Sun, “On Bayesian Adaptive Video Super Resolution”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 36, No. 2, pp. 346-360, 2014.

  • Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han and Thomas Huang, “Deep Networks for Image Super-Resolution with Sparse Prior”, Proceedings of IEEE International Conference on Computer Vision, pp. 370-378, 2015.

  • Video enhancer, Available: http://www.infognition.com/videoenhancer/

  • Z. Ma, R. Liao, X. Tao, L. Xu, J. Jia, and E. Wu, “Handling Motion Blur In Multi-Frame Super-Resolution”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5224-5232, 2015.

  • Renjie Liao, Xin Tao, Ruiyu Li, Ziyang Ma and Jiaya Jia, “Video Super-Resolution via Deep Draft-Ensemble Learning”, Proceedings of IEEE International Conference on Computer Vision, pp. 531-539, 2015.

  • J. Yang, Z. Wang, Z. Lin, X. Shu, and T. Huang, “Bilevel Sparse Coding for Coupled Feature Spaces” , Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2360-2367, 2012.

  • M. Bevilacqua, A. Roumy, C. Guillemot, and M.L. Alberi-Morel, “Lowcomplexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding”, Proceedings of 23rd British Machine Vision Conference, pp. 1350113510, 2012.

  • H. Chang, D.Y. Yeung and Y. Xiong, “Super-Resolution through Neighbor Embedding”, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 1-6, 2004.

  • R. Timofte, V. De and L. Van Gool, “Anchored Neighborhood Regression for Fast Example-based Super-Resolution”, Proceedings of IEEE International Conference on Computer Vision, pp. 1920-1927, 2013.

  • C. Dong, C.C. Loy, K. He and X. Tang, “Learning A Deep Convolutional Network for Image Super-Resolution” , Proceedings of European Conference on Computer Vision, pp. 184-199, 2014.

  • Q. Dai, S. Yoo, A. Kappeler and A.K. Katsaggelos, “Sparse Representation-Based Multiple Frame Video Super-Resolution”, IEEE Transactions on Image Processing, Vol. 26, No. 2, pp. 765-781, 2017.

  • Dingyi Li and Zengfu Wang, “Video Super-Resolution via Motion Compensation and Deep Residual Learning”, IEEE Transactions on Computational Imaging, Vol. PP, No. 99, pp. 1-15, 2017.


Abstract Views: 473

PDF Views: 6




  • Real-Time Video Scaling Based on Convolution Neural Network Architecture

Abstract Views: 473  |  PDF Views: 6

Authors

Safinaz S.
Department of Electronics and Communication Engineering, Sir M. Visvesvaraya Institute of Technology, India
Ravi Kumar A. V.
Department of Electronics and Communication Engineering, SJB Institute of Technology, India

Abstract


In recent years, video super resolution techniques becomes mandatory requirements to get high resolution videos. Many super resolution techniques researched but still video super resolution or scaling is a vital challenge. In this paper, we have presented a real-time video scaling based on convolution neural network architecture to eliminate the blurriness in the images and video frames and to provide better reconstruction quality while scaling of large datasets from lower resolution frames to high resolution frames. We compare our outcomes with multiple exiting algorithms. Our extensive results of proposed technique RemCNN (Reconstruction error minimization Convolution Neural Network) shows that our model outperforms the existing technologies such as bicubic, bilinear, MCResNet and provide better reconstructed motioning images and video frames. The experimental results shows that our average PSNR result is 47.80474 considering upscale-2, 41.70209 for upscale-3 and 36.24503 for upscale-4 for Myanmar dataset which is very high in contrast to other existing techniques. This results proves our proposed model real-time video scaling based on convolution neural network architecture's high efficiency and better performance.

Keywords


Image Scaling, Convolution Neural, Network, Super Resolution.

References