光学神经网络智能处理：从分立到集成，由二维卷积到高维张量

doi:10.7527/S1000-6893.2024.30439

Abstract

Abstract: Convolutional Neural Network (CNN) has shown a wide range of applications in the fields of face recognition, image classification, machine vision, medical imaging, and aerospace due to its excellent feature extraction capabilities. However, traditional electrical intelligent processing chips are restricted by Moore's Law, which is difficult to meet the continuous growth of CNN computing power demand. With its characteristics of ultra-large broadband and ultra-low loss, light wave is a disruptive technology that supports the high computing power demand of the next generation of artificial intelligence, with optical or electrical high-dimensional control structure as the basic unit, and realizes computing through the controlled propagation of light. In this paper, we summarize the research progress and technological breakthroughs of optical convolutional neural networks, summarize the overall trend of their development, discuss the technical problems that need to be solved in the future, and look forward to the application prospects of optical convolutional neural networks.

Key words: convolution, tensor, neural networks, artificial intelligence, deep learning

CLC Number:

V247
TN256

References

[1] FUKUSHIMA K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[J]. Biological Cybernetics, 1980, 36(4): 193-202.
[2] LECUN Y, BOSER B, DENKER J, et al. Handwritten digit recognition with a backpropagation network[J]. Advances in Neural Information Processing Systems, 1989, 1 (4): 541–551.
[3] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[J]. Advances in Neural Information Processing Systems, 2012, 25(2): 1097–1105.
[4] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint: 1409.1556, 2015.
[5] SZEGEDY C, LIU W, SERMANET P, et al. Going deeper with convolutions[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1-9.
[6] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[7] HOWARD A G, ZHU M L, CHEN B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint: 1704.04861, 2017.
[8] CHANG J L, SITZMANN V, DUN X, et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification[J]. Scientific Reports, 2018, 8(1): 1-10.
[9] AEEATIA A, SEPULVEDA E. Convolutional neural networks, image recognition and financial time series forecasting[C]. Mining Data for Financial Applications: 4th ECML PKDD Workshop, MIDAS 2019, Würzburg, Germany, September 16, 2019, Revised Selected Papers 4. Springer International Publishing, 2020: 60-69.
[10] SZEGEDY C, TOSHEV A, ERHAN D. Deep neural networks for object detection[J]. Advances in Neural Information Processing Systems, 2013, 26.
[11] HE S F, LAU R W H, LIU W X, et al. Supercnn: A superpixelwise convolutional neural network for salient object detection[J]. International Journal of Computer Vision, 2015, 115: 330-344.
[12] LI Q, CAI W, WANG X, et al. Medical image classification with convolutional neural network[C]. 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV). IEEE, 2014: 844-848.
[13] LU L, ZHENG Y F, CARNEIRO G, et al. Deep learning and convolutional neural networks for medical image computing[J]. Advances in Computer Vision and Pattern Recognition, 2017, 10: 978-981.
[14] KUMAR A, KIM J, LYNDON D, et al. An ensemble of fine-tuned convolutional neural networks for medical image classification[J]. IEEE Journal of Biomedical and Health Informatics, 2016, 21(1): 31-40.
[15] YAMASHITA R, NISHIO M, DO R K G, et al. Convolutional neural networks: an overview and application in radiology[J]. Insights into Imaging, 2018, 9: 611-629.
[16] ZHOU C, XU D M, CHEN L, et al. Evaluation of fish feeding intensity in aquaculture using a convolutional neural network and machine vision[J]. Aquaculture, 2019, 507: 457-465.
[17] WILLIAMS H A M, JONES M H, NEJATI M, et al. Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms[J]. Biosystems Engineering, 2019, 181: 140-156.
[18] LIN X, RIVENSON Y, YARDIMCI N T, et al. All-optical machine learning using diffractive deep neural networks[J]. Science, 2018, 361(6406): 1004-1008.
[19] SHAHVERDY M, FATHY M, BERANGI R, et al. Driver behavior detection and classification using deep convolutional neural networks[J]. Expert Systems with Applications, 2020, 149: 113240.
[20] CHEN L, LIN S B, LU X K, et al. Deep neural network based vehicle and pedestrian detection for autonomous driving: A survey[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(6): 3234-3246.
[21] SIROHI D, KUMAR N, RANA P S. Convolutional neural networks for 5G-enabled intelligent transportation system: A systematic review[J]. Computer Communications, 2020, 153: 459-498.
[22] XU X Y, TAN M X, CORCORAN B, et al. 11 TOPS photonic convolutional accelerator for optical neural networks[J]. Nature, 2021, 589(7840): 44-51.
[23] MENG X Y, SHI N N, LI G Z, et al. Optical Convolutional Neural Networks: Methodology and Advances[J]. Applied Sciences, 2023, 13(13): 7523.
[24] SLUDDS A, BANDYOPADHYAY S, CHEN Z, et al. Delocalized Photonic Deep Learning on the Internet’s Edge[J]. Science, 2022, 378: 270–276.
[25] COLBURN S, CHU Y, SHILZERMAN E, et al. Optical frontend for a convolutional neural network[J]. Applied Optics, 2019, 58(12): 3179-3186.
[26] OKUMA N, KAWABATA K, SHIOZAKI K, et al. Topological origin of non-Hermitian skin effects[J]. Physical Review Letters, 2020, 124(8): 086801.
[27] COLBURN S, CHU Y, SHILZERMAN E, et al. Optical frontend for a convolutional neural network[J]. Applied Optics, 2019, 58(12): 3179-3186.
[28] TASI F, O’BRIEN C, NOVAK S P, et al. Analysis of optical channel cross talk for free-space optical interconnects in the presence of higher-order transverse modes[J] Applied Optics, 2005, 44(30): 6380-6387.
[29] HU W H, LI X J, YANG J K, et al. Crosstalk analysis of aligned and misaligned free-space optical interconnect systems[J]. Journal of the Optical Society of America A, 2010, 27(2): 200–205.
[30] ASHTIANI F, GEERS A J, AFLATOUNI F. An on-chip photonic deep neural network for image classification[J]. Nature, 2022, 606(7914): 501-506.
[31] XU S F, WANG J, WANG R, et al. High-accuracy optical convolution unit architecture for convolutional neural networks by cascaded acousto-optical modulator arrays[J]. Optics Express, 2019, 27(14): 19778-19787.
[32] XU S F, WANG J, SHU H W, et al. Optical coherent dot-product chip for sophisticated deep learning regression[J]. Light: Science & Applications, 2021, 10(1): 221.
[33] QU Y R, ZHU H Z, SHEN Y C, et al. Inverse design of an integrated-nanophotonics optical neural network[J]. Science Bulletin, 2020, 65(14): 1177-1183.
[34] KHORAM E, CHEN A, LIU D, et al. Nanophotonic media for artificial neural inference[J]. Photonics Research, 2019, 7(8): 823-827.
[35] ATHALE R A, COLLINS W C. Optical matrix–matrix multiplier based on outer product decomposition[J]. Applied Optics, 1982, 21(12): 2089-2090.
[36] RECK M, ZEILINGER A, BERNSTEIN H J, et al. Experimental realization of any discrete unitary operator[J]. Physical Review Letters, 1994, 73(1): 58.
[37] ZHU W W, ZHANG L, LU Y Y, et al. Design and experimental verification for optical module of optical vector–matrix multiplier[J]. Applied Optics, 2013, 52(18): 4412-4418.
[38] CLEMENTS W R, HUMPHREYS P C, METCALF B J, et al. Optimal design for universal multiport interferometers[J]. Optica, 2016, 3(12): 1460-1465.
[39] CHENG J W, ZHOU H L, DONG J J. Photonic matrix computing: from fundamentals to applications[J]. Nanomaterials, 2021, 11(7): 1683.
[40] XU X Y, TAN M X, CORCORAN B, et al. Photonic perceptron based on a Kerr Microcomb for high-speed, scalable, optical neural networks[J]. Laser & Photonics Reviews, 2020, 14(10): 2000070.
[41] XU X Y, TAN M X, CORCORAN B, et al. 11 TOPS photonic convolutional accelerator for optical neural networks[J]. Nature, 2021, 589(7840): 44-51.
[42] ZANG Y B, Chen M H, Yang S G, et al. Optoelectronic convolutional neural networks based on time-stretch method[J]. Science China Information Sciences, 2021, 64: 1-12.
[43] MENG X Y, Shi N N, Shi D F, et al. Photonics-enabled spiking timing-dependent convolutional neural network for real-time image classification[J]. Optics Express, 2022, 30(10): 16217-16228.
[44] MENG X Y, Shi N N, Li G Y, et al. On-demand reconfigurable incoherent optical matrix operator for real-time video image display[J]. Journal of Lightwave Technology, 2022, 41(6): 1637-1648.
[45] HUANG L, YAO J P. Optical processor for a binarized neural network[J]. Optics Letters, 2022, 47(15): 3892-3895.
[46] XU Z Z, TANG K F, JI X, et al. Experimental demonstration of a photonic convolutional accelerator based on a monolithically integrated multi-wavelength distributed feedback laser[J]. Optics Letters, 2022, 47(22): 5977-5980.
[47] HUANG Y Y, ZHANG W J, YANG F, et al. Programmable matrix operation with reconfigurable time-wavelength plane manipulation and dispersed time delay[J]. Optics Express, 2019, 27(15): 20456-20467.
[48] CHANG J, SITZMANN V, DUN X, et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification[J]. Scientific Reports, 2018, 8(1): 12324.
[49] LIN X, RIVENSON Y, YARDIMCI N T, et al. All-optical machine learning using diffractive deep neural networks[J]. Science, 2018, 361(6406): 1004-1008.
[50] LIU Z B, GAO S C, LAI Z Y, et al. Broadband, Low-Crosstalk, and Massive-Channels OAM Modes De/Multiplexing Based on Optical Diffraction Neural Network[J]. Laser & Photonics Reviews, 2023, 17(4): 2200536.
[51] ZHANG J Q, YE Z Y, YIN J H, et al. Polarized deep diffractive neural network for sorting, generation, multiplexing, and de-multiplexing of orbital angular momentum modes[J]. Optics Express, 2022, 30(15): 26728-26741.
[52] MENGU D, LUO Y, RIVENSON Y, et al. Analysis of diffractive optical neural networks and their integration with electronic neural networks[J]. IEEE Journal of Selected Topics in Quantum Electronics, 2019, 26(1): 1-14.
[53] YAN T, WU J M, ZHOU T K, et al. Fourier-space diffractive deep neural network[J]. Physical Review Letters, 2019, 123(2): 023901.
[54] CHEN Y S, ZHU J F. An optical diffractive deep neural network with multiple frequency-channels[J]. arXiv preprint: 1912.10730, 2019.
[55] LUO Y, MENGU D, YARDIMCI N T, et al. Design of task-specific optical systems using broadband diffractive neural networks[J]. Light: Science & Applications, 2019, 8(1): 112.
[56] SHEN Y C, HARRIS N C, SKIRLO S, et al. Deep learning with coherent nanophotonic circuits[J]. Nature Photonics, 2017, 11(7): 441-446.
[57] BAGHERIAN H, SKIRLO S, SHEN Y C, et al. On-chip optical convolutional neural networks[J]. arXiv preprint: 1808.03303, 2018.
[58] XU X, ZHU L, ZHUANG W, et al. A Convolution Neural Network Implemented by Three 3×3 Photonic Integrated Reconfigurable Linear Processors[J] Photonics, 2022, 9(2): 80.
[59] DE MARINIS L, COCOCCIONI M, LIBOIRON-Ladouceur O, et al. Photonic integrated reconfigurable linear processors as neural network accelerators[J]. Applied Sciences, 2021, 11(13): 6232.
[60] XU X F, ZHU L Q, ZHUANG W, et al. Photoelectric hybrid convolution neural network with coherent nanophotonic circuits[J]. Optical Engineering, 2021, 60(11): 117106-117106.
[61] YANG Z, TAN W M, ZHANG T J, et al. MXene‐based broadband ultrafast nonlinear activator for optical computing[J]. Advanced Optical Materials, 2022, 10(17): 2200714.
[62] SOLDANO L B, PENNINGS E C M. Optical multi-mode interference devices based on self-imaging: principles and applications[J]. Journal of Lightwave Technology, 1995, 13(4): 615-627.
[63] MENG X, ZHANG G, SHI N, et al. Compact optical convolution processing unit based on multimode interference[J]. Nature Communications, 2023, 14(1): 3000.
[64] LIAO K, CHEN Y, Yu Z, et al. All-optical computing based on convolutional neural networks[J]. Opto-Electronic Advances, 2021, 4(11): 200060.
[65] XU X F, ZHU L Q, ZHUANG W. Convolutional neural networks with coherent nanophotonic circuits[J] 10th International Symposium on Advanced Optical Manufacturing and Testing Technologies: Intelligent Sensing Technologies and Applications. SPIE, 2021, 12075: 207-213.
[66] JIANG Y, ZHANG W, YANG F, et al. Photonic convolution neural network based on interleaved time-wavelength modulation[J]. Journal of Lightwave Technology, 2021, 39(14): 4592-4600.
[67] BANGARI V, MARQUEZ B A, MILLER H, et al. Digital electronics and analog photonics for convolutional neural networks (DEAP-CNNs)[J]. IEEE Journal of Selected Topics in Quantum Electronics, 2019, 26(1): 1-13.
[68] XU S F, WANG J, Zou W W. Optical convolutional neural network with WDM-based optical patching and microring weighting banks[J]. IEEE Photonics Technology Letters, 2020, 33(2): 89-92.
[69] BAI B W, YANG Q P, SHU H W, et al. Microcomb-based integrated photonic processing unit[J]. Nature Communications, 2023, 14(1): 66.
[70] XU S F, WANG J, ZOU W W. Optical patching scheme for optical convolutional neural networks based on wavelength-division multiplexing and optical delay lines[J]. Optics Letters, 2020, 45(13): 3689-3692.
[71] XU S F, WANG J, Zou W W. High-energy-efficiency integrated photonic convolutional neural networks[J]. arXiv preprint:1910.12635, 2019.
[72] OHNO S, TANG R, TOPRASERTPONG K, et al. Si microring resonator crossbar array for on-chip inference and training of the optical neural network[J]. Acs Photonics, 2022, 9(8): 2614-2622.
[73] TAIT A N, DE LIMA T F, NAHMIAS M A, et al. Silicon photonic modulator neuron[J]. Physical Review Applied, 2019, 11(6): 064043.
[74] ONG J R, OOI C C, ANG T Y L, et al. Photonic convolutional neural networks using integrated diffractive optics[J]. IEEE Journal of Selected Topics in Quantum Electronics, 2020, 26(5): 1-8.
[75] FU T, ZANG Y, HUANG Y, et al. Photonic machine learning with on-chip diffractive optics[J]. Nature Communications, 2023, 14(1): 70.
[76] CHEN Z J, SLUDDS A, DAVIS III R, et al. Deep learning with coherent VCSEL neural networks[J]. Nature Photonics, 2023, 17(8): 723-730.
[77] LI S, YANG H, WONG C W, et al. Photofourier: A photonic joint transform correlator-based neural network accelerator[C]. 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2023: 15-28.
[78] ZHU H H, ZOU J, ZHANG H, et al. Space-efficient optical computing with an integrated chip diffractive neural network[J]. Nature Communications, 2022, 13(1): 1044.
[79] XU S, WANG J, YI S, et al. High-order tensor flow processing using integrated photonic circuits[J]. Nature Communications, 2022, 13(1): 7970.
[80] FELDMANN J, YOUNGBIOOD N, KARPOV M, et al. Parallel convolutional processing using an integrated photonic tensor core[J]. Nature, 2021, 589(7840): 52-58.
[81] HUANG Y Y, Fu T Z, Huang H H, et al. Sophisticated deep learning with on-chip optical diffractive tensor processing[J]. Photonics Research, 2023, 11(6): 1125-1138.
[82] DONG B W, AGGARWAL S, ZHOU W, et al. Higher-dimensional processing using a photonic tensor core with continuous-time data[J]. Nature Photonics, 2023, 17(12): 1080-1088.
[83] YIN R Y, XIAO H F, JIANG Y H, et al. Optical mode division multiplexing inspired photonic neural network accelerator[J]. Research Square Platform LLC, 2023.
[84] CHEN Y T, NAZHAMAITI M, XU H, et al. All-analog photoelectronic chip for high-speed vision tasks[J]. Nature, 2023, 623(7985): 48-57.
[85] GU Z Y, GAO Y S, LIU X Z. Optronic convolutional neural networks of multi-layers with different functions executed in optics for image classification. Optics Express, 2021, 29(4): 5877-5889.
[86] NAHMIAS M A, TAIT A N, TOLIAS L, et al. An integrated analog O/E/O link for multi-channel laser neurons[J]. Applied Physics Letters, 2016, 108(15): 151106.
[87] MESARITAKIS C, KAPSALIS A, SYVRIDIS D. All-optical reservoir computing system based on InGaAsP ring resonators for high-speed identification and optical routing in optical networks[J]. Quantum Sensing and Nanophotonic Devices XII. SPIE, 2015, 9370: 608-614.
[88] DUPORT F, SCHNEIDER B, SMERIERI A, et al. All-optical reservoir computing[J]. Optics Express, 2012, 20(20): 22783-22795.
[89] CHENG Z, TSANG H K, WANG X, et al. In-plane optical absorption and free carrier absorption in graphene-on-silicon waveguides[J]. IEEE Journal of Selected Topics in Quantum Electronics, 2013, 20(1): 43-48.
[90] MISCUGLIO M, MEHRABIAN A, HU Z, et al. All-optical nonlinear activation function for photonic neural networks[J]. Optical Materials Express, 2018, 8(12): 3851-3863.
[91] ZHANG H, THOMPSON J, GU M, et al. Efficient on-chip training of optical neural networks using genetic algorithm[J]. Acs Photonics, 2021, 8(6): 1662-1672.

Intelligent processing of optical neural networks: from discrete to integrated and from two-dimensional convolution to higher dimensional tensor

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Xudong LUO, Yiquan WU, Jinlin CHEN. Research progress on deep learning methods for object detection and semantic segmentation in UAV aerial images [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 28822-028822.
[2]	Shengzhe SHAN, Weiwei ZHANG. Air combat intelligent decision-making method based on self-play and deep reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328723-328723.
[3]	Haiqiao LIU, Meng LIU, Zichao GONG, Jing DONG. A review of image matching methods based on deep learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(3): 28796-028796.
[4]	Liqun CHEN, Xu ZOU, Lei ZHANG, Yingpan ZHU, Gang WANG, Jinyong CHEN. On⁃board intelligent target detection technology based on domestic commercial components [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 728860-728860.
[5]	Xin SU, Runcheng GUAN, Qiao WANG, Weizheng YUAN, Xianglian LYU, Yang HE. Ice area and thickness detection method based on deep learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729283-729283.
[6]	Pengyu LIU, Xueyao ZHU. Semantic parsing technology of air traffic control instruction in fusion airspace based on deep learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S1): 727592-727592.
[7]	Zhongzhi LI, Jinyi MA, Jianliang AI, Yiqun DONG. Fault detection and classification of aerospace sensors using deep neural networks finetuned from VGG16 [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S1): 727615-727615.
[8]	Lei HE, Weiqi QIAN, Kangsheng DONG, Xian YI, Congcong CHAI. Aerodynamic characteristics modeling of iced airfoil based on convolution neural networks [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(5): 126434-126434.
[9]	Peng DING, Yafei SONG. A cost-sensitive method for aerial target intention recognition [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(24): 328551-328551.
[10]	Linkun HE, Wenchao XUE, Ran ZHANG, Huifeng LI. Guidance and control for powered descent and landing of launch vehicles: Overview and outlook [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(23): 628462-628462.
[11]	Xiaohang LI, Jianjiang ZHOU. Multi⁃scale modality fusion network based on adaptive memory length [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(22): 628977-628977.
[12]	Haowen LUO, Shaoming HE, Tianyu JIN, Zichao LIU. Impact-angle-constrained with time-minimum guidance algorithm based on transfer learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(19): 328400-328400.
[13]	Yunhe ZHAO, Shengnan WANG. Solution to stress intensity factor by weight function method based on deep learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(19): 228367-228367.
[14]	Penghui JI, Shiqi XING, Dahai DAI, Bo Pang, Dejun FENG. A SAR scene deceptive jamming method based on range convolution and azimuth multiplication modulation [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(18): 328235-328235.
[15]	Shuyi GAO, Defu LIN, Duo ZHENG, Xinyu HU. Intelligent cooperative interception strategy of aircraft against cluster attack [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(18): 328301-328301.