光学神经网络智能处理：技术演变与未来展望

doi:10.7527/S1000-6893.2024.30439

Abstract

Abstract:

Convolutional Neural Network （CNN） has shown a wide range of applications in the fields of face recognition， image classification， machine vision， medical imaging， and aerospace due to its excellent feature extraction capabilities. However， traditional electrical intelligent processing chips are restricted by Moore’‍s law， and is difficult to meet the continuous growth of CNN computing power demand. With its characteristics of ultra-large broadband and ultra-low loss， light wave is a disruptive technology that supports the high computing power demand of the next generation of artificial intelligence. With optical or electrical high-dimensional control structure as the basic unit， it can realize computing through controlled propagation of light. In this paper， the research progress and technological breakthroughs of optical convolutional neural networks are reviewed. The overall trend of their development and the technical problems that need to be solved in the future are summarized. The prospects of optical convolutional neural networks for application are also discussed.

Key words: convolution, tensor, neural networks, artificial intelligence, deep learning

CLC Number:

V247

Junshen LI, Xiangyan MENG, Nuannuan SHI, Wei LI, Ninghua ZHU, Ming LI. Intelligent processing of optical neural networks: Technological evolution and future prospects[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(20): 630439.

Figures/Tables 8

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

References 93

1	FUKUSHIMA K. Neocognitron： a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position［J］. Biological Cybernetics， 1980， 36（4）： 193-202.
2	LECUN Y， BOSER B， DENKER J， et al. Handwritten digit recognition with a backpropagation network［J］. Advances in Neural Information Processing Systems， 1989， 1 （4）： 541-551.
3	KRIZHEVSKY A， SUTSKEVER I， HINTON G E. Imagenet classification with deep convolutional neural networks［J］. Advances in Neural Information Processing Systems， 2012， 25（2）： 1097-1105.
4	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［DB/OL］. arXiv preprint： 1409.1556， 2014.
5	SZEGEDY C， LIU W， JIA Y Q， et al. Going deeper with convolutions［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 1-9.
6	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］∥ 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2016： 770-778.
7	HOWARD A G， ZHU M L， CHEN B， et al. MobileNets： efficient convolutional neural networks for mobile vision applications［DB/OL］. arXiv preprint： 1704.04861， 2017.
8	CHANG J L， SITZMANN V， DUN X， et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification［J］. Scientific Reports， 2018， 8： 12324.
9	ARRATIA A， SEPÚLVEDA E. Convolutional neural networks， image recognition and financial time series forecasting［C］∥Workshop on Mining Data for Financial Applications. Cham： Springer， 2020： 60-69.
10	SZEGEDY C， TOSHEV A， ERHAN D. Deep neural networks for object detection［J］. Advances in Neural Information Processing Systems， 2013， 26.
11	HE S F， LAU R W H， LIU W X， et al. SuperCNN： a superpixelwise convolutional neural network for salient object detection［J］. International Journal of Computer Vision， 2015， 115（3）： 330-344.
12	LI Q， CAI W D， WANG X G， et al. Medical image classification with convolutional neural network［C］∥ 2014 13th International Conference on Control Automation Robotics & Vision （ICARCV）. Piscataway： IEEE Press， 2014： 844-848.
13	LU L， ZHENG Y F， CARNEIRO G， et al. Deep learning and convolutional neural networks for medical image computing［J］. Advances in Computer Vision and Pattern Recognition， 2017， 10： 978-981.
14	KUMAR A， KIM J， LYNDON D， et al. An ensemble of fine-tuned convolutional neural networks for medical image classification［J］. IEEE Journal of Biomedical and Health Informatics， 2017， 21（1）： 31-40.
15	YAMASHITA R， NISHIO M， DO R K G， et al. Convolutional neural networks： an overview and application in radiology［J］. Insights into Imaging， 2018， 9（4）： 611-629.
16	ZHOU C， XU D M， CHEN L， et al. Evaluation of fish feeding intensity in aquaculture using a convolutional neural network and machine vision［J］. Aquaculture， 2019， 507： 457-465.
17	WILLIAMS H A M， JONES M H， NEJATI M， et al. Robotic kiwifruit harvesting using machine vision， convolutional neural networks， and robotic arms［J］. Biosystems Engineering， 2019， 181： 140-156.
18	LIN X， RIVENSON Y， YARDIMCI N T， et al. All-optical machine learning using diffractive deep neural networks［J］. Science， 2018， 361（6406）： 1004-1008.
19	SHAHVERDY M， FATHY M， BERANGI R， et al. Driver behavior detection and classification using deep convolutional neural networks［J］. Expert Systems with Applications， 2020， 149： 113240.
20	CHEN L， LIN S B， LU X K， et al. Deep neural network based vehicle and pedestrian detection for autonomous driving： a survey［J］. IEEE Transactions on Intelligent Transportation Systems， 2021， 22（6）： 3234-3246.
21	SIROHI D， KUMAR N， RANA P S. Convolutional neural networks for 5G-enabled intelligent transportation system： a systematic review［J］. Computer Communications， 2020， 153： 459-498.
22	KASTER J， PATRICK J， CLOUSE H S. Convolutional neural networks on small unmanned aerial systems［C］∥ 2017 IEEE National Aerospace and Electronics Conference （NAECON）. Piscataway： IEEE Press， 2017： 149-154.
23	ZHOU L M， YAN H X， SHAN Y Z， et al. Aircraft detection for remote sensing images based on deep convolutional neural networks［J］. Journal of Electrical and Computer Engineering， 2021： 4685644.
24	DING P， ZHANG Y， DENG W J， et al. A light and faster regional convolutional neural network for object detection in optical remote sensing images［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2018， 141： 208-218.
25	RADOVIC M， ADARKWA O， WANG Q S. Object recognition in aerial images using convolutional neural networks［J］. Journal of Imaging， 2017， 3（2）： 21.
26	GALLEGO A J， PERTUSA A， GIL P. Automatic ship classification from optical aerial images with convolutional neural networks［J］. Remote Sensing， 2018， 10（4）： 511.
27	XU X Y， TAN M X， CORCORAN B， et al. 11 TOPS photonic convolutional accelerator for optical neural networks［J］. Nature， 2021， 589： 44-51.
28	MENG X Y， SHI N N， LI G Y， et al. Optical convolutional neural networks： methodology and advances （invited）［J］. Applied Sciences， 2023， 13（13）： 7523.
29	SLUDDS A， BANDYOPADHYAY S， CHEN Z J， et al. Delocalized photonic deep learning on the internet’s edge［J］. Science， 2022， 378（6617）： 270-276.
30	COLBURN S， CHU Y， SHILZERMAN E， et al. Optical frontend for a convolutional neural network［J］. Applied Optics， 2019， 58（12）： 3179-3186.
31	OKUMA N， KAWABATA K， SHIOZAKI K， et al. Topological origin of non-hermitian skin effects［J］. Physical Review Letters， 2020， 124（8）： 086801.
32	TSAI F C F， O’BRIEN C J， PETROVIĆ N S， et al. Analysis of optical channel cross talk for free-space optical interconnects in the presence of higher-order transverse modes［J］. Applied Optics， 2005， 44（30）： 6380-6387.
33	HU W H， LI X J， YANG J K， et al. Crosstalk analysis of aligned and misaligned free-space optical interconnect systems［J］. Journal of the Optical Society of America A， Optics， Image Science， and Vision， 2010， 27（2）： 200-205.
34	ASHTIANI F， GEERS A J， AFLATOUNI F. An on-chip photonic deep neural network for image classification［J］. Nature， 2022， 606： 501-506.
35	XU S F， WANG J， WANG R， et al. High-accuracy optical convolution unit architecture for convolutional neural networks by cascaded acousto-optical modulator arrays［J］. Optics Express， 2019， 27（14）： 19778-19787.
36	XU S F， WANG J， SHU H W， et al. Optical coherent dot-product chip for sophisticated deep learning regression［J］. Light： Science & Applications， 2021， 10： 221.
37	QU Y R， ZHU H Z， SHEN Y C， et al. Inverse design of an integrated-nanophotonics optical neural network［J］. Science Bulletin， 2020， 65（14）： 1177-1183.
38	KHORAM E， CHEN A， LIU D J， et al. Nanophotonic media for artificial neural inference［J］. Photonics Research， 2019， 7（8）： 823.
39	ATHALE R A， COLLINS W C. Optical matrix-matrix multiplier based on outer product decomposition［J］. Applied Optics， 1982， 21（12）： 2089-2090.
40	RECK M， ZEILINGER A， BERNSTEIN H J， et al. Experimental realization of any discrete unitary operator［J］. Physical Review Letters， 1994， 73（1）： 58-61.
41	ZHU W W， ZHANG L， LU Y Y， et al. Design and experimental verification for optical module of optical vector-matrix multiplier［J］. Applied Optics， 2013， 52（18）： 4412-4418.
42	CLEMENTS W R， HUMPHREYS P C， METCALF B J， et al. Optimal design for universal multiport interferometers［J］. Optica， 2016， 3（12）： 1460.
43	CHENG J W， ZHOU H L， DONG J J. Photonic matrix computing： from fundamentals to applications［J］. Nanomaterials， 2021， 11（7）： 1683.
44	XU X Y， TAN M X， WU J Y， et al. Photonic perceptron based on a Kerr microcomb for high-speed， scalable， optical neural networks［J］ Laser & Photonics Reviews， 2020， 14（10）： 2000070.
45	ZANG Y B， CHEN M H， YANG S G， et al. Optoelectronic convolutional neural networks based on time-stretch method［J］. Science China Information Sciences， 2021， 64（2）： 122401.
46	MENG X Y， SHI N N， SHI D F， et al. Photonics-enabled spiking timing-dependent convolutional neural network for real-time image classification［J］. Optics Express， 2022， 30（10）： 16217-16228.
47	MENG X Y， SHI N N， LI G Y， et al. On-demand reconfigurable incoherent optical matrix operator for real-time video image display［J］. Journal of Lightwave Technology， 2023， 41（6）： 1637-1648.
48	HUANG L， YAO J P. Optical processor for a binarized neural network［J］. Optics Letters， 2022， 47（15）： 3892-3895.
49	XU Z Z， TANG K F， JI X， et al. Experimental demonstration of a photonic convolutional accelerator based on a monolithically integrated multi-wavelength distributed feedback laser［J］. Optics Letters， 2022， 47（22）： 5977-5980.
50	HUANG Y Y， ZHANG W J， YANG F， et al. Programmable matrix operation with reconfigurable time-wavelength plane manipulation and dispersed time delay［J］. Optics Express， 2019， 27（15）： 20456-20467.
51	LIU Z B， GAO S C， LAI Z Y， et al. Broadband， low-crosstalk， and massive-channels OAM modes de/multiplexing based on optical diffraction neural network［J］. Laser & Photonics Reviews， 2023， 17（4）： 2200536.
52	ZHANG J Q， YE Z Y， YIN J H， et al. Polarized deep diffractive neural network for sorting， generation， multiplexing， and de-multiplexing of orbital angular momentum modes［J］. Optics Express， 2022， 30（15）： 26728-26741.
53	MENGU D， LUO Y， RIVENSON Y， et al. Analysis of diffractive optical neural networks and their integration with electronic neural networks［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2020， 26（1）： 3700114.
54	YAN T， WU J M， ZHOU T K， et al. Fourier-space diffractive deep neural network［J］. Physical Review Letters， 2019， 123（2）： 023901.
55	CHEN Y S， ZHU J F. An optical diffractive deep neural network with multiple frequency-channels［DB/OL］. arXiv preprint： 1912.10730， 2019.
56	LUO Y， MENGU D， YARDIMCI N T， et al. Design of task-specific optical systems using broadband diffractive neural networks［J］. Light： Science & Applications， 2019， 8： 112.
57	SHEN Y C， HARRIS N C， SKIRLO S， et al. Deep learning with coherent nanophotonic circuits［J］. Nature Photonics， 2017， 11： 441-446.
58	BAGHERIAN H， SKIRLO S， SHEN Y C， et al. On-chip optical convolutional neural networks［DB/OL］. arXiv preprint： 1808.03303， 2018.
59	XU X F， ZHU L Q， ZHUANG W， et al. A convolution neural network implemented by three 3 × 3 photonic integrated reconfigurable linear processors［J］. Photonics， 2022， 9（2）： 80.
60	DE MARINIS L， COCOCCIONI M， LIBOIRON-LADOUCEUR O， et al. Photonic integrated reconfigurable linear processors as neural network accelerators［J］. Applied Sciences， 2021， 11（13）： 6232.
61	XU X F， ZHU L Q， ZHUANG W， et al. Photoelectric hybrid convolution neural network with coherent nanophotonic circuits［J］. Optical Engineering， 2021， 60： 117106.
62	YANG Z， TAN W M， ZHANG T J， et al. MXene-based broadband ultrafast nonlinear activator for optical computing［J］. Advanced Optical Materials， 2022， 10（17）： 2200714.
63	SOLDANO L B， PENNINGS E C M. Optical multi-mode interference devices based on self-imaging： principles and applications［J］. Journal of Lightwave Technology， 1995， 13（4）： 615-627.
64	MENG X Y， ZHANG G J， SHI N N， et al. Compact optical convolution processing unit based on multimode interference［J］. Nature Communications， 2023， 14： 3000.
65	LIAO K， CHEN Y， YU Z C， et al. All-optical computing based on convolutional neural networks［J］. Opto-Electronic Advances， 2021， 4（11）： 200060.
66	XU X F， ZHU L Q， ZHUANG W. Convolutional neural networks with coherent nanophotonic circuits［C］∥ 10th International Symposium on Advanced Optical Manufacturing and Testing Technologies： Intelligent Sensing Technologies and Applications. SPIE， 2021： 207-213.
67	JIANG Y， ZHANG W J， YANG F， et al. Photonic convolution neural network based on interleaved time-wavelength modulation［J］. Journal of Lightwave Technology， 2021， 39（14）： 4592-4600.
68	BANGARI V， MARQUEZ B A， MILLER H， et al. Digital electronics and analog photonics for convolutional neural networks （DEAP-CNNs）［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2020， 26（1）： 7701213.
69	XU S F， WANG J， ZOU W W. Optical convolutional neural network with WDM-based optical patching and microring weighting banks［J］. IEEE Photonics Technology Letters， 2021， 33（2）： 89-92.
70	BAI B W， YANG Q P， SHU H W， et al. Microcomb-based integrated photonic processing unit［J］. Nature Communications， 2023， 14： 66.
71	XU S F， WANG J， ZOU W W. Optical patching scheme for optical convolutional neural networks based on wavelength-division multiplexing and optical delay lines［J］. Optics Letters， 2020， 45（13）： 3689-3692.
72	XU S F， WANG J， ZOU W W. High-energy-efficiency integrated photonic convolutional neural networks［DB/OL］. arXiv preprint： 1910.12635， 2019.
73	OHNO S， TANG R， TOPRASERTPONG K， et al. Si microring resonator crossbar array for on-chip inference and training of the optical neural network［J］. ACS Photonics， 2022， 9（8）： 2614-2622.
74	TAIT A N， FERREIRA DE LIMA T， NAHMIAS M A， et al. Silicon photonic modulator neuron［J］. Physical Review Applied， 2019， 11（6）： 064043.
75	ONG J R， OOI C C， ANG T Y L， et al. Photonic convolutional neural networks using integrated diffractive optics［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2020， 26（5）： 7702108.
76	FU T Z， ZANG Y B， HUANG H H， et al. On-chip photonic diffractive optical neural network based on a spatial domain electromagnetic propagation model［J］. Optics Express， 2021， 29（20）： 31924-31940.
77	FU T Z， ZANG Y B， HUANG Y Y， et al. Photonic machine learning with on-chip diffractive optics［J］. Nature Communications， 2023， 14： 70.
78	CHEN Z J， SLUDDS A， DAVIS R， et al. Deep learning with coherent VCSEL neural networks［J］. Nature Photonics， 2023， 17： 723-730.
79	LI S R， YANG H B， WONG C W， et al. PhotoFourier： a photonic joint transform correlator-based neural network accelerator［C］∥ 2023 IEEE International Symposium on High-Performance Computer Architecture （HPCA）. Piscataway： IEEE Press， 2023： 15-28.
80	ZHU H H， ZOU J， ZHANG H， et al. Space-efficient optical computing with an integrated chip diffractive neural network［J］. Nature Communications， 2022， 13： 1044.
81	FELDMANN J， YOUNGBLOOD N， KARPOV M， et al. Parallel convolutional processing using an integrated photonic tensor core［J］. Nature， 2021， 589： 52-58.
82	XU S F， WANG J， YI S C， et al. High-order tensor flow processing using integrated photonic circuits［J］. Nature Communications， 2022， 13： 7970.
83	HUANG Y Y， FU T Z， HUANG H H， et al. Sophisticated deep learning with on-chip optical diffractive tensor processing［J］. Photonics Research， 2023， 11（6）： 1125.
84	DONG B W， AGGARWAL S， ZHOU W， et al. Higher-dimensional processing using a photonic tensor core with continuous-time data［J］. Nature Photonics， 2023， 17： 1080-1088.
85	YIN R Y， XIAO H F， JIANG Y H， et al. Optical mode division multiplexing inspired photonic neural network accelerator［DB/OL］. Research Square preprint， 2023.
86	CHEN Y T， NAZHAMAITI M， XU H， et al. All-analog photoelectronic chip for high-speed vision tasks［J］. Nature， 2023， 623： 48-57.
87	GU Z Y， GAO Y S， LIU X Z. Optronic convolutional neural networks of multi-layers with different functions executed in optics for image classification［J］. Optics Express， 2021， 29（4）： 5877-5889.
88	NAHMIAS M A， TAIT A N， TOLIAS L， et al. An integrated analog O/E/O link for multi-channel laser neurons［J］. Applied Physics Letters， 2016， 108（15）： 151106.
89	MESARITAKIS C， KAPSALIS A， SYVRIDIS D. All-optical reservoir computing system based on InGaAsP ring resonators for high-speed identification and optical routing in optical networks［C］∥ Quantum Sensing and Nanophotonic Devices XII. SPIE， 2015， 9370： 608-614.
90	DUPORT F， SCHNEIDER B， SMERIERI A， et al. All-optical reservoir computing［J］. Optics Express， 2012， 20（20）： 22783.
91	CHENG Z Z， TSANG H K， WANG X M， et al. In-plane optical absorption and free carrier absorption in graphene-on-silicon waveguides［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2014， 20（1）： 4400106.
92	MISCUGLIO M， MEHRABIAN A， HU Z B， et al. All-optical nonlinear activation function for photonic neural networks［J］. Optical Materials Express， 2018， 8（12）： 3851.
93	ZHANG H， THOMPSON J， GU M L， et al. Efficient on-chip training of optical neural networks using genetic algorithm［J］. ACS Photonics， 2021， 8（6）： 1662-1672.

[1]	Jianyu XU, Li ZHOU, Zhanxue WANG, Jie SHI, Hao SHI. Calculation method for hypersonic plume infrared radiation based on a fast line-by-line calculation model [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 630778-630778.
[2]	Lingjie MENG, Hongguang LI, Xinjun LI. SAR image simulation method guided by geomorphic category information [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(7): 331003-331003.
[3]	Mou CHEN, Zhengguo HUANG, Yaohua SHEN, Fan LIU. Overview of composite anti-disturbance control technology of advanced vehicles [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(6): 531303-531303.
[4]	Guanghui WU, Jing WANG, Hairun XIE, Tuliang MA, Qiang MIAO, Jixin XIANG, Miao ZHANG. Data and knowledge-enabled intelligent aerodynamic design for civil aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(5): 531485-531485.
[5]	Zhihao ZHAO, Zhaohua YANG, Yun WU, Yuanjin YU. Single-photon counting imaging denoising method based on deep learning in low-light environment [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(3): 630531-630531.
[6]	Yiquan WU, Kang TONG. Research advances on deep learning-based small object detection in UAV aerial images [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(3): 30848-030848.
[7]	Yinxuan ZHANG, Qi ZHANG, Zhenyong XU, Linshu MENG. Predicting method of aircraft mechanical response based on residual neural networks [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(19): 531295-531295.
[8]	Yugang ZHANG, Zhe YANG, Senpeng HE, Wenqing YANG. Aircraft attitude prediction model based on physical information neural networks [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(19): 531850-531850.
[9]	Lin LIN, Shiwei SUO, Dan LIU, Yinxuan ZHANG, Lingyu YUE, Sihao ZHANG, Yikun LIU, Song FU. A deep feature fusion network based on multi-scale kernel construction for filling wing stress field data [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(19): 532343-532343.
[10]	Xiaowei JIANG, Yiquan WU. Research progress of UAV aerial image mosaic methods [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(17): 331799-331799.
[11]	Lin CHEN, Xiwen GU, Zhiying CHEN, Zhuo ZHANG, Xiaoliang SUN. High-precision monocular vision pose measurement for large distance span in carrier landing guidance [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(15): 331568-331568.
[12]	Wei CHEN, Lulu LI, Dong CHEN, Shaohui ZHANG, Yafei LI, Ke WANG, Yuanyuan JIN, Mingliang XU. Multi-aircraft cooperative decision-making methods driven by differentiated support demands for carrier-based aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531274-531274.
[13]	Ming YAN, Jiaxing WANG, Heqi LI, Kai LIU. Active disturbance rejection control of carrier-based aircraft based on offline network/online identification [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531317-531317.
[14]	Yifeng WANG, Yiming PENG, Long LI, Xiaohui WEI, Hong NIE. DQN-based active arrest and recovery technique for UAVs [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(12): 231448-231448.
[15]	Bin SUN, Hang YOU, Wenbo LI, Xiangrui LIU, Jiayi MA. Dual-band payload image fusion and its applications in low-altitude remote sensing [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(11): 531343-531343.

Intelligent processing of optical neural networks: Technological evolution and future prospects

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 93

Related Articles 15

Recommended Articles

Metrics

Comments