Real-time single-pixel video imaging based on deep learning

Ziran Wei; Jianlin Zhang; Wei Du; Zhiruo Wang

doi:10.62677/IJETAA.2412130

Authors

Ziran Wei Chengdu Aircraft Industrial (Group) Co. Ltd., Aviation Industry Corporation of China (AVIC), Chengdu, China Author
Jianlin Zhang Institute of Optics and Electronics, Chinese Academy of Sciences, Chengdu, China Author
Wei Du Chengdu Aircraft Industrial (Group) Co. Ltd., Aviation Industry Corporation of China (AVIC), Chengdu, China Author
Zhiruo Wang Xidian University, Xi'an, China Author

DOI:

https://doi.org/10.62677/IJETAA.2412130

Keywords:

Compressed sensing, Real-time single-pixel imaging, Deep Learning

Abstract

The emergence of compressed sensing (CS) theory has enabled the development of single-pixel cameras that achieve high-resolution imaging using a single photodetector. However, traditional CS reconstruction algorithms require significant computational time and face an inherent trade-off between imaging resolution and frame rate, limiting current single-pixel cameras to static scene imaging. A key challenge lies in achieving real-time single-pixel imaging with both high frame rate and high resolution. This paper proposes a real-time single-pixel imaging technology based on deep learning. We design a deep convolutional neural network architecture incorporating residual networks to simulate the measurement and reconstruction process of CS-based single-pixel imaging. The network is trained on an image dataset and subsequently deployed for single-pixel imaging. The trained network can complete image reconstruction at a low sampling rate with minimal latency, enabling real-time single-pixel video capture at 128×128 resolution with 33 frames per second (fps) at a 4\% sampling rate. Furthermore, we implement a four-channel parallel signal processing method to achieve real-time single-pixel imaging video at 256×256 resolution at 33 fps.

Downloads

Download data is not yet available.

References

D. L. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol. 52,no. 4, pp. 1289–1306, Apr. 2006.

E. Candes, "Compressive sampling," in Proc. Int. Congress Math., Madrid, Spain, 2006, vol. 3, pp. 1433–1452.

H. Li, J. Xu, H. Song, and Y. Wang, "Deep unfolding for sparse aperture ISAR imaging via compressed sensing: CIST+," in IET Int. Radar Conf., Chongqing, China, 2023, pp. 3483–3488.

D. Milioris, "Classification in twitter via compressive sensing," in Proc. IEEE Conf. Comput. Commun. Workshops, Hong Kong, China, 2015, pp. 95–96.

Z. Xue et al., "DARCS: Memory-efficient deep compressed sensing reconstruction for acceleration of 3D whole-heart coronary MR angiography," IEEE Trans. Med. Imaging, Jan. 2024.

B. Xia and Y. Tang, "Compressive sensing based channel estimation method for MIMO-OFDM power line communication," in Int. Conf.Energy, Power and Grid, Guangzhou, China, 2024, pp. 1658–1661.

M. Duarte et al., "Single-pixel imaging via compressive sampling," IEEE Signal Process. Mag., vol. 25, no. 2, pp. 83–91, 2008.

W. Dong, G. Shi, X. Li, Y. Ma, and F. Huang, "Compressive sensing via nonlocal low-rank regularization," IEEE Trans. Image Process., vol.23, no. 8, pp. 3618–3632, 2014.

C. A. Metzler, A. Maleki, and R. G. Baraniuk, "From denoising to compressed sensing," IEEE Trans. Inf. Theory, vol. 62, no. 9, pp. 51175144, 2016.

A. Sinha, J. Lee, S. Li, and G. Barbastathis, "Lensless computational imaging through deep learning," Optica, vol. 4, pp. 1117–1125, 2017.

X. He, P. Wang, H. Liu, and X. Yu, "Passive non-line-of-sight imaging reconstruction based on dual input U-Net," in Int. Conf. Neural Netw., Inf. Commun. Eng., Guangzhou, China, 2023, pp. 337–341.

Y. Hualong and G. Daidou, "Research on double encryption of ghost imaging by SegNet deep neural network," IEEE Photon. Technol. Lett., vol. 36, no. 10, pp. 669–672, May 2024.

N. Mishra and S. Mishra, "Impact of deep learning applications in medical hyperspectral imaging," in Healthcare Big Data Analytics: Computational Optimization and Cohesive Approaches. De Gruyter,2024, pp. 179–206.

F. Hashimoto et al., "Compressed-sensing magnetic resonance image reconstruction using an iterative convolutional neural network approach," Appl. Sci., vol. 10, no. 6, pp. 1902, 2020.

K. Haris et al., "Free-breathing fetal cardiac MRI with doppler ultrasound gating, compressed sensing, and motion compensation," J. Magn. Reson. Imaging, vol. 51, no. 1, pp. 260–272, 2020.

G. R. Matcuk, J. S. Gross, and J. Fritz, "Compressed sensing MRI," Adv. Clin. Radiol., vol. 2, pp. 257–271, 2020.

V. B. Manur and L. Ali, "Compressed sensing channel estimation for STBC-SM based hybrid MIMO-OFDM system for visible light communication," Int. J. Commun. Syst., vol. 33, no. 11, pp. e4403,2020.

Y. Chen et al., "Hyperspectral image compressive sensing reconstruction using subspace-based nonlocal tensor ring decomposition," IEEE Trans.Image Process., vol. 29, pp. 6813–6828, 2020.

Z. R. Wei, J. L. Zhang, Z. Y. Xu et al., "Improving the signal-to-noise ratio of super resolution imaging based on single pixel camera," IEEE Photon. J., pp. 1–16, 2019.

J. E. Fowler, S. Mun, and E. W. Tramel, "Block-based compressed sensing of images and video," Found. Trends Signal Process., vol. 4, no. 4, pp. 297–416, 2010.

K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.

W. Shi et al., "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network," in Proc. IEEE Conf.Comput. Vis. Pattern Recognit., 2016, pp. 1874–1883.

A. Paszke et al., "Automatic differentiation in PyTorch," in NIPS Workshop Autodiff, 2017.

E. Agustsson and R. Timofte, "NTIRE 2017 challenge on single image super-resolution: Methods and results," in IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2017, pp. 1110–1121.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, "Loss functions for neural networks for image processing," arXiv preprint arXiv:1511.08861, 2015.