ACTA AERONAUTICAET ASTRONAUTICA SINICA >
Intelligent processing of optical neural networks: Technological evolution and future prospects
Received date: 2024-03-25
Revised date: 2024-04-19
Accepted date: 2024-05-23
Online published: 2024-05-29
Supported by
Chinese Academy of Sciences Youth Innovation Promotion Association(2022111);National Natural Science Foundation of China(62235011)
Convolutional Neural Network （CNN） has shown a wide range of applications in the fields of face recognition， image classification， machine vision， medical imaging， and aerospace due to its excellent feature extraction capabilities. However， traditional electrical intelligent processing chips are restricted by Moore’s law， and is difficult to meet the continuous growth of CNN computing power demand. With its characteristics of ultra-large broadband and ultra-low loss， light wave is a disruptive technology that supports the high computing power demand of the next generation of artificial intelligence. With optical or electrical high-dimensional control structure as the basic unit， it can realize computing through controlled propagation of light. In this paper， the research progress and technological breakthroughs of optical convolutional neural networks are reviewed. The overall trend of their development and the technical problems that need to be solved in the future are summarized. The prospects of optical convolutional neural networks for application are also discussed.
Key words： convolution; tensor; neural networks; artificial intelligence; deep learning
Junshen LI , Xiangyan MENG , Nuannuan SHI , Wei LI , Ninghua ZHU , Ming LI . Intelligent processing of optical neural networks: Technological evolution and future prospects[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2024 , 45(20) : 630439 -630439 . DOI: 10.7527/S1000-6893.2024.30439
| 1 | FUKUSHIMA K. Neocognitron： a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position［J］. Biological Cybernetics， 1980， 36（4）： 193-202. |
| 2 | LECUN Y， BOSER B， DENKER J， et al. Handwritten digit recognition with a backpropagation network［J］. Advances in Neural Information Processing Systems， 1989， 1 （4）： 541-551. |
| 3 | KRIZHEVSKY A， SUTSKEVER I， HINTON G E. Imagenet classification with deep convolutional neural networks［J］. Advances in Neural Information Processing Systems， 2012， 25（2）： 1097-1105. |
| 4 | SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［DB/OL］. arXiv preprint： 1409.1556， 2014. |
| 5 | SZEGEDY C， LIU W， JIA Y Q， et al. Going deeper with convolutions［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 1-9. |
| 6 | HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］∥ 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2016： 770-778. |
| 7 | HOWARD A G， ZHU M L， CHEN B， et al. MobileNets： efficient convolutional neural networks for mobile vision applications［DB/OL］. arXiv preprint： 1704.04861， 2017. |
| 8 | CHANG J L， SITZMANN V， DUN X， et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification［J］. Scientific Reports， 2018， 8： 12324. |
| 9 | ARRATIA A， SEPúLVEDA E. Convolutional neural networks， image recognition and financial time series forecasting［C］∥Workshop on Mining Data for Financial Applications. Cham： Springer， 2020： 60-69. |
| 10 | SZEGEDY C， TOSHEV A， ERHAN D. Deep neural networks for object detection［J］. Advances in Neural Information Processing Systems， 2013， 26. |
| 11 | HE S F， LAU R W H， LIU W X， et al. SuperCNN： a superpixelwise convolutional neural network for salient object detection［J］. International Journal of Computer Vision， 2015， 115（3）： 330-344. |
| 12 | LI Q， CAI W D， WANG X G， et al. Medical image classification with convolutional neural network［C］∥ 2014 13th International Conference on Control Automation Robotics & Vision （ICARCV）. Piscataway： IEEE Press， 2014： 844-848. |
| 13 | LU L， ZHENG Y F， CARNEIRO G， et al. Deep learning and convolutional neural networks for medical image computing［J］. Advances in Computer Vision and Pattern Recognition， 2017， 10： 978-981. |
| 14 | KUMAR A， KIM J， LYNDON D， et al. An ensemble of fine-tuned convolutional neural networks for medical image classification［J］. IEEE Journal of Biomedical and Health Informatics， 2017， 21（1）： 31-40. |
| 15 | YAMASHITA R， NISHIO M， DO R K G， et al. Convolutional neural networks： an overview and application in radiology［J］. Insights into Imaging， 2018， 9（4）： 611-629. |
| 16 | ZHOU C， XU D M， CHEN L， et al. Evaluation of fish feeding intensity in aquaculture using a convolutional neural network and machine vision［J］. Aquaculture， 2019， 507： 457-465. |
| 17 | WILLIAMS H A M， JONES M H， NEJATI M， et al. Robotic kiwifruit harvesting using machine vision， convolutional neural networks， and robotic arms［J］. Biosystems Engineering， 2019， 181： 140-156. |
| 18 | LIN X， RIVENSON Y， YARDIMCI N T， et al. All-optical machine learning using diffractive deep neural networks［J］. Science， 2018， 361（6406）： 1004-1008. |
| 19 | SHAHVERDY M， FATHY M， BERANGI R， et al. Driver behavior detection and classification using deep convolutional neural networks［J］. Expert Systems with Applications， 2020， 149： 113240. |
| 20 | CHEN L， LIN S B， LU X K， et al. Deep neural network based vehicle and pedestrian detection for autonomous driving： a survey［J］. IEEE Transactions on Intelligent Transportation Systems， 2021， 22（6）： 3234-3246. |
| 21 | SIROHI D， KUMAR N， RANA P S. Convolutional neural networks for 5G-enabled intelligent transportation system： a systematic review［J］. Computer Communications， 2020， 153： 459-498. |
| 22 | KASTER J， PATRICK J， CLOUSE H S. Convolutional neural networks on small unmanned aerial systems［C］∥ 2017 IEEE National Aerospace and Electronics Conference （NAECON）. Piscataway： IEEE Press， 2017： 149-154. |
| 23 | ZHOU L M， YAN H X， SHAN Y Z， et al. Aircraft detection for remote sensing images based on deep convolutional neural networks［J］. Journal of Electrical and Computer Engineering， 2021： 4685644. |
| 24 | DING P， ZHANG Y， DENG W J， et al. A light and faster regional convolutional neural network for object detection in optical remote sensing images［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2018， 141： 208-218. |
| 25 | RADOVIC M， ADARKWA O， WANG Q S. Object recognition in aerial images using convolutional neural networks［J］. Journal of Imaging， 2017， 3（2）： 21. |
| 26 | GALLEGO A J， PERTUSA A， GIL P. Automatic ship classification from optical aerial images with convolutional neural networks［J］. Remote Sensing， 2018， 10（4）： 511. |
| 27 | XU X Y， TAN M X， CORCORAN B， et al. 11 TOPS photonic convolutional accelerator for optical neural networks［J］. Nature， 2021， 589： 44-51. |
| 28 | MENG X Y， SHI N N， LI G Y， et al. Optical convolutional neural networks： methodology and advances （invited）［J］. Applied Sciences， 2023， 13（13）： 7523. |
| 29 | SLUDDS A， BANDYOPADHYAY S， CHEN Z J， et al. Delocalized photonic deep learning on the internet’s edge［J］. Science， 2022， 378（6617）： 270-276. |
| 30 | COLBURN S， CHU Y， SHILZERMAN E， et al. Optical frontend for a convolutional neural network［J］. Applied Optics， 2019， 58（12）： 3179-3186. |
| 31 | OKUMA N， KAWABATA K， SHIOZAKI K， et al. Topological origin of non-hermitian skin effects［J］. Physical Review Letters， 2020， 124（8）： 086801. |
| 32 | TSAI F C F， O’BRIEN C J， PETROVI? N S， et al. Analysis of optical channel cross talk for free-space optical interconnects in the presence of higher-order transverse modes［J］. Applied Optics， 2005， 44（30）： 6380-6387. |
| 33 | HU W H， LI X J， YANG J K， et al. Crosstalk analysis of aligned and misaligned free-space optical interconnect systems［J］. Journal of the Optical Society of America A， Optics， Image Science， and Vision， 2010， 27（2）： 200-205. |
| 34 | ASHTIANI F， GEERS A J， AFLATOUNI F. An on-chip photonic deep neural network for image classification［J］. Nature， 2022， 606： 501-506. |
| 35 | XU S F， WANG J， WANG R， et al. High-accuracy optical convolution unit architecture for convolutional neural networks by cascaded acousto-optical modulator arrays［J］. Optics Express， 2019， 27（14）： 19778-19787. |
| 36 | XU S F， WANG J， SHU H W， et al. Optical coherent dot-product chip for sophisticated deep learning regression［J］. Light： Science & Applications， 2021， 10： 221. |
| 37 | QU Y R， ZHU H Z， SHEN Y C， et al. Inverse design of an integrated-nanophotonics optical neural network［J］. Science Bulletin， 2020， 65（14）： 1177-1183. |
| 38 | KHORAM E， CHEN A， LIU D J， et al. Nanophotonic media for artificial neural inference［J］. Photonics Research， 2019， 7（8）： 823. |
| 39 | ATHALE R A， COLLINS W C. Optical matrix-matrix multiplier based on outer product decomposition［J］. Applied Optics， 1982， 21（12）： 2089-2090. |
| 40 | RECK M， ZEILINGER A， BERNSTEIN H J， et al. Experimental realization of any discrete unitary operator［J］. Physical Review Letters， 1994， 73（1）： 58-61. |
| 41 | ZHU W W， ZHANG L， LU Y Y， et al. Design and experimental verification for optical module of optical vector-matrix multiplier［J］. Applied Optics， 2013， 52（18）： 4412-4418. |
| 42 | CLEMENTS W R， HUMPHREYS P C， METCALF B J， et al. Optimal design for universal multiport interferometers［J］. Optica， 2016， 3（12）： 1460. |
| 43 | CHENG J W， ZHOU H L， DONG J J. Photonic matrix computing： from fundamentals to applications［J］. Nanomaterials， 2021， 11（7）： 1683. |
| 44 | XU X Y， TAN M X， WU J Y， et al. Photonic perceptron based on a Kerr microcomb for high-speed， scalable， optical neural networks［J］ Laser & Photonics Reviews， 2020， 14（10）： 2000070. |
| 45 | ZANG Y B， CHEN M H， YANG S G， et al. Optoelectronic convolutional neural networks based on time-stretch method［J］. Science China Information Sciences， 2021， 64（2）： 122401. |
| 46 | MENG X Y， SHI N N， SHI D F， et al. Photonics-enabled spiking timing-dependent convolutional neural network for real-time image classification［J］. Optics Express， 2022， 30（10）： 16217-16228. |
| 47 | MENG X Y， SHI N N， LI G Y， et al. On-demand reconfigurable incoherent optical matrix operator for real-time video image display［J］. Journal of Lightwave Technology， 2023， 41（6）： 1637-1648. |
| 48 | HUANG L， YAO J P. Optical processor for a binarized neural network［J］. Optics Letters， 2022， 47（15）： 3892-3895. |
| 49 | XU Z Z， TANG K F， JI X， et al. Experimental demonstration of a photonic convolutional accelerator based on a monolithically integrated multi-wavelength distributed feedback laser［J］. Optics Letters， 2022， 47（22）： 5977-5980. |
| 50 | HUANG Y Y， ZHANG W J， YANG F， et al. Programmable matrix operation with reconfigurable time-wavelength plane manipulation and dispersed time delay［J］. Optics Express， 2019， 27（15）： 20456-20467. |
| 51 | LIU Z B， GAO S C， LAI Z Y， et al. Broadband， low-crosstalk， and massive-channels OAM modes de/multiplexing based on optical diffraction neural network［J］. Laser & Photonics Reviews， 2023， 17（4）： 2200536. |
| 52 | ZHANG J Q， YE Z Y， YIN J H， et al. Polarized deep diffractive neural network for sorting， generation， multiplexing， and de-multiplexing of orbital angular momentum modes［J］. Optics Express， 2022， 30（15）： 26728-26741. |
| 53 | MENGU D， LUO Y， RIVENSON Y， et al. Analysis of diffractive optical neural networks and their integration with electronic neural networks［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2020， 26（1）： 3700114. |
| 54 | YAN T， WU J M， ZHOU T K， et al. Fourier-space diffractive deep neural network［J］. Physical Review Letters， 2019， 123（2）： 023901. |
| 55 | CHEN Y S， ZHU J F. An optical diffractive deep neural network with multiple frequency-channels［DB/OL］. arXiv preprint： 1912.10730， 2019. |
| 56 | LUO Y， MENGU D， YARDIMCI N T， et al. Design of task-specific optical systems using broadband diffractive neural networks［J］. Light： Science & Applications， 2019， 8： 112. |
| 57 | SHEN Y C， HARRIS N C， SKIRLO S， et al. Deep learning with coherent nanophotonic circuits［J］. Nature Photonics， 2017， 11： 441-446. |
| 58 | BAGHERIAN H， SKIRLO S， SHEN Y C， et al. On-chip optical convolutional neural networks［DB/OL］. arXiv preprint： 1808.03303， 2018. |
| 59 | XU X F， ZHU L Q， ZHUANG W， et al. A convolution neural network implemented by three 3 × 3 photonic integrated reconfigurable linear processors［J］. Photonics， 2022， 9（2）： 80. |
| 60 | DE MARINIS L， COCOCCIONI M， LIBOIRON-LADOUCEUR O， et al. Photonic integrated reconfigurable linear processors as neural network accelerators［J］. Applied Sciences， 2021， 11（13）： 6232. |
| 61 | XU X F， ZHU L Q， ZHUANG W， et al. Photoelectric hybrid convolution neural network with coherent nanophotonic circuits［J］. Optical Engineering， 2021， 60： 117106. |
| 62 | YANG Z， TAN W M， ZHANG T J， et al. MXene-based broadband ultrafast nonlinear activator for optical computing［J］. Advanced Optical Materials， 2022， 10（17）： 2200714. |
| 63 | SOLDANO L B， PENNINGS E C M. Optical multi-mode interference devices based on self-imaging： principles and applications［J］. Journal of Lightwave Technology， 1995， 13（4）： 615-627. |
| 64 | MENG X Y， ZHANG G J， SHI N N， et al. Compact optical convolution processing unit based on multimode interference［J］. Nature Communications， 2023， 14： 3000. |
| 65 | LIAO K， CHEN Y， YU Z C， et al. All-optical computing based on convolutional neural networks［J］. Opto-Electronic Advances， 2021， 4（11）： 200060. |
| 66 | XU X F， ZHU L Q， ZHUANG W. Convolutional neural networks with coherent nanophotonic circuits［C］∥ 10th International Symposium on Advanced Optical Manufacturing and Testing Technologies： Intelligent Sensing Technologies and Applications. SPIE， 2021： 207-213. |
| 67 | JIANG Y， ZHANG W J， YANG F， et al. Photonic convolution neural network based on interleaved time-wavelength modulation［J］. Journal of Lightwave Technology， 2021， 39（14）： 4592-4600. |
| 68 | BANGARI V， MARQUEZ B A， MILLER H， et al. Digital electronics and analog photonics for convolutional neural networks （DEAP-CNNs）［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2020， 26（1）： 7701213. |
| 69 | XU S F， WANG J， ZOU W W. Optical convolutional neural network with WDM-based optical patching and microring weighting banks［J］. IEEE Photonics Technology Letters， 2021， 33（2）： 89-92. |
| 70 | BAI B W， YANG Q P， SHU H W， et al. Microcomb-based integrated photonic processing unit［J］. Nature Communications， 2023， 14： 66. |
| 71 | XU S F， WANG J， ZOU W W. Optical patching scheme for optical convolutional neural networks based on wavelength-division multiplexing and optical delay lines［J］. Optics Letters， 2020， 45（13）： 3689-3692. |
| 72 | XU S F， WANG J， ZOU W W. High-energy-efficiency integrated photonic convolutional neural networks［DB/OL］. arXiv preprint： 1910.12635， 2019. |
| 73 | OHNO S， TANG R， TOPRASERTPONG K， et al. Si microring resonator crossbar array for on-chip inference and training of the optical neural network［J］. ACS Photonics， 2022， 9（8）： 2614-2622. |
| 74 | TAIT A N， FERREIRA DE LIMA T， NAHMIAS M A， et al. Silicon photonic modulator neuron［J］. Physical Review Applied， 2019， 11（6）： 064043. |
| 75 | ONG J R， OOI C C， ANG T Y L， et al. Photonic convolutional neural networks using integrated diffractive optics［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2020， 26（5）： 7702108. |
| 76 | FU T Z， ZANG Y B， HUANG H H， et al. On-chip photonic diffractive optical neural network based on a spatial domain electromagnetic propagation model［J］. Optics Express， 2021， 29（20）： 31924-31940. |
| 77 | FU T Z， ZANG Y B， HUANG Y Y， et al. Photonic machine learning with on-chip diffractive optics［J］. Nature Communications， 2023， 14： 70. |
| 78 | CHEN Z J， SLUDDS A， DAVIS R， et al. Deep learning with coherent VCSEL neural networks［J］. Nature Photonics， 2023， 17： 723-730. |
| 79 | LI S R， YANG H B， WONG C W， et al. PhotoFourier： a photonic joint transform correlator-based neural network accelerator［C］∥ 2023 IEEE International Symposium on High-Performance Computer Architecture （HPCA）. Piscataway： IEEE Press， 2023： 15-28. |
| 80 | ZHU H H， ZOU J， ZHANG H， et al. Space-efficient optical computing with an integrated chip diffractive neural network［J］. Nature Communications， 2022， 13： 1044. |
| 81 | FELDMANN J， YOUNGBLOOD N， KARPOV M， et al. Parallel convolutional processing using an integrated photonic tensor core［J］. Nature， 2021， 589： 52-58. |
| 82 | XU S F， WANG J， YI S C， et al. High-order tensor flow processing using integrated photonic circuits［J］. Nature Communications， 2022， 13： 7970. |
| 83 | HUANG Y Y， FU T Z， HUANG H H， et al. Sophisticated deep learning with on-chip optical diffractive tensor processing［J］. Photonics Research， 2023， 11（6）： 1125. |
| 84 | DONG B W， AGGARWAL S， ZHOU W， et al. Higher-dimensional processing using a photonic tensor core with continuous-time data［J］. Nature Photonics， 2023， 17： 1080-1088. |
| 85 | YIN R Y， XIAO H F， JIANG Y H， et al. Optical mode division multiplexing inspired photonic neural network accelerator［DB/OL］. Research Square preprint， 2023. |
| 86 | CHEN Y T， NAZHAMAITI M， XU H， et al. All-analog photoelectronic chip for high-speed vision tasks［J］. Nature， 2023， 623： 48-57. |
| 87 | GU Z Y， GAO Y S， LIU X Z. Optronic convolutional neural networks of multi-layers with different functions executed in optics for image classification［J］. Optics Express， 2021， 29（4）： 5877-5889. |
| 88 | NAHMIAS M A， TAIT A N， TOLIAS L， et al. An integrated analog O/E/O link for multi-channel laser neurons［J］. Applied Physics Letters， 2016， 108（15）： 151106. |
| 89 | MESARITAKIS C， KAPSALIS A， SYVRIDIS D. All-optical reservoir computing system based on InGaAsP ring resonators for high-speed identification and optical routing in optical networks［C］∥ Quantum Sensing and Nanophotonic Devices XII. SPIE， 2015， 9370： 608-614. |
| 90 | DUPORT F， SCHNEIDER B， SMERIERI A， et al. All-optical reservoir computing［J］. Optics Express， 2012， 20（20）： 22783. |
| 91 | CHENG Z Z， TSANG H K， WANG X M， et al. In-plane optical absorption and free carrier absorption in graphene-on-silicon waveguides［J］. IEEE Journal of Selected Topics in Quantum Electronics， 2014， 20（1）： 4400106. |
| 92 | MISCUGLIO M， MEHRABIAN A， HU Z B， et al. All-optical nonlinear activation function for photonic neural networks［J］. Optical Materials Express， 2018， 8（12）： 3851. |
| 93 | ZHANG H， THOMPSON J， GU M L， et al. Efficient on-chip training of optical neural networks using genetic algorithm［J］. ACS Photonics， 2021， 8（6）： 1662-1672. |
/
| 〈 |
|
〉 |
CJA