Research development of small object traching based on deep learning

LI Hongguang; YU Ruonan; DING Wenrui

doi:10.7527/S1000-6893.2020.24691

ACTA AERONAUTICAET ASTRONAUTICA SINICA >

2021 , Vol. 42 >Issue 7: 24691 - 024691

DOI: https://doi.org/10.7527/S1000-6893.2020.24691

Review

Research development of small object traching based on deep learning

LI Hongguang ,
YU Ruonan ,
DING Wenrui

Expand

1. Research Institute of Unmanned System, Beihang University, Beijing 100191, China;
2. School of Electronics and Information Engineering, Beihang University, Beijing 100191, China

Received date: 2020-09-01

Revised date: 2020-09-15

Online published: 2020-10-23

Supported by

Equipment Pre-research Field Fundation(61403110404); National Natural Science Foundation of China (62076019)

Fold

Abstract

Object detection, one of the most fundamental and challenging tasks in computer vision, has achieved remarkable breakthroughs with the rapid development of deep learning methods. However, most of the current state-of-the-art algorithms are designed for medium or large objects with regular sizes or proportions. The performance of detecting small objects is still far from satisfactory due to the small size and weak features of target objects. In recent years, Small Object Traching (SOT), which is widely applied in outdoor remote shooting and aerospace remote sensing scenarios, has drawn significant attention and substantial approaches have emerged. However, few comprehensive reviews on this issue have been conducted. This paper summarizes the research progress of deep learning-based SOT methods in terms of problem definition, algorithm analysis, application introduction, and future prospects. We first present the definition of the SOT problem and illustrate its technical difficulties and challenges faced in practical applications. The main reasons for the low accuracy of small object detectors and the corresponding improvement methods from eight different perspectives are then analyzed, and the SOT research work in various technical aspects summarized in detail. The representative applications of SOT algorithms in several specific scenarios are introduced. Finally, we prospect the development trends and promising directions for future research. Hopefully, this survey can provide reference for the research work in this field.

Key words： object detection; deep learning; small object traching; computer vision; convolutional neural networks

Cite this article

LI Hongguang , YU Ruonan , DING Wenrui . Research development of small object traching based on deep learning[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021 , 42(7) : 24691 -024691 . DOI: 10.7527/S1000-6893.2020.24691

References

[1] NAJIBI M, SAMANGOUEI P, CHELLAPPA R, et al. SSH:Single stage headless face detector[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017:4885-4894.
[2] ZHANG L L, LIN L, LIANG X D, et al. Is faster R-CNN doing well for pedestrian detection?[C]//Computer Vision-ECCV 2016, 2016.
[3] CHEN Q, TANG S H, YANG Q, et al. Cooper:Cooperative perception for connected autonomous vehicles based on 3D point clouds[C]//2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). Piscataway:IEEE Press, 2019:514-524.
[4] WANG Z Y, FU H, WANG L, et al. SCNet:Subdivision coding network for object detection based on 3D point cloud[J]. IEEE Access, 2019, 7:120449-120462.
[5] RAGHUNANDAN A, MOHANA, RAGHAV P, et al. Object detection algorithms for video surveillance applications[C]//2018 International Conference on Communication and Signal Processing (ICCSP). Piscataway:IEEE Press, 2018:0563-0568.
[6] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2014:580-587.
[7] GIRSHICK R. Fast R-CNN[C]//2015 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2015:1440-1448.
[8] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN:Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149.
[9] LIU W, ANGUELOV D, ERHAN D, et al. SSD:Single shot MultiBox detector[M]//Computer Vision-ECCV 2016. Cham:Springer International Publishing, 2016:21-37.
[10] REDMON J, FARHADI A. YOLO9000:Better, faster, stronger[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017:6517-6525.
[11] REDMON J, FARHADI A. YOLOv3:An incremental improvement[DB/OL]. arXiv preprint:1804.02767, 2018.
[12] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:770-778.
[13] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017:2261-2269.
[14] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017:936-944.
[15] FU C Y, LIU W, RANGA A, et al. DSSD:Deconvolutional single shot detector[DB/OL]. arXiv preprint:1701.06659, 2017
[16] BELL S, ZITNICK C L, BALA K, et al. Inside-outside net:Detecting objects in context with skip pooling and recurrent neural networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:2874-2883.
[17] CHEN C Y, LIU M Y, TUZEL O, et al. R-CNN for small object detection[M]//Computer Vision-ACCV 2016. Cham:Springer International Publishing, 2017:214-230.
[18] CAI Z W, FAN Q F, FERIS R S, et al. A unified multi-scale deep convolutional neural network for fast object detection[C]//Computer Vision-ECCV 2016, 2016.
[19] KRISHNA H, JAWAHAR C V. Improving small object detection[C]//2017 4th IAPR Asian Conference on Pattern Recognition (ACPR). Piscataway:IEEE Press, 2017:340-345.
[20] WANG J Q, CHEN K, YANG S, et al. Region proposal by guided anchoring[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019:2960-2969.
[21] SINGH B, DAVIS L S. An analysis of scale invariance in object detection-SNIP[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018:3578-3587.
[22] HU X W, XU X M, XIAO Y J, et al. SINet:A scale-insensitive convolutional neural network for fast vehicle detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(3):1010-1019.
[23] HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017:2980-2988.
[24] KONG T, YAO A B, CHEN Y R, et al. HyperNet:towards accurate region proposal generation and joint object detection[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:845-853.
[25] GHODRATI A, DIBA A, PEDERSOLI M, et al. DeepProposal:Hunting objects by cascading deep convolutional layers[C]//2015 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2015:2578-2586.
[26] CAI Z W, VASCONCELOS N. Cascade R-CNN:Delving into high quality object detection[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018:6154-6162.
[27] CUI L S, MA R, LV P, et al. MDSSD:Multi-scale deconvolutional single shot detector for small objects[J]. Science China Information Sciences, 2020, 63(2):1-3.
[28] CAO G M, XIE X M, YANG W Z, et al. Feature-fused SSD:Fast detection for small objects[C]//International Conference on Graphic and Image Processing (ICGIP),2017.
[29] LI Z, ZHOU F. FSSD:Feature fusion single shot Multibox detector[DB/OL]. arXiv preprint:1712.00960, 2018.
[30] LI J N, LIANG X D, WEI Y C, et al. Perceptual generative adversarial networks for small object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017:1951-1959.
[31] BAI Y C, ZHANG Y Q, DING M L, et al. SOD-MTGAN:Small object detection via multi-task generative adversarial network[C]//Computer Vision-ECCV 2018, 2018.
[32] NOH J, BAE W, LEE W, et al. Better to follow, follow to be better:Towards precise supervision of feature super-resolution for small object detection[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019:9724-9733.
[33] KISANTAL M, WOJNA Z, MURAWSKI J, et al. Augmentation for small object detection[C]//9th International Conference on Advances in Computing and Information Technology (ACITY 2019), 2019.
[34] CHEN C R, ZHANG Y, LV Q, et al. RRNet:A hybrid detector for object detection in drone-captured images[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Piscataway:IEEE Press, 2019:100-108.
[35] CHEN Y, ZHANG P, LI Z, et al. Stitcher:Feedback-driven data provider for object detection[DB/OL]. arXiv preprint:004.12432, 2020.
[36] BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4:Optimal speed and accuracy of object detection[DB/OL]. arXiv preprint:2004.10934, 2020
[37] 刘颖, 刘红燕, 范九伦, 等. 基于深度学习的小目标检测研究与应用综述[J]. 电子学报, 2020, 48(3):590-601. LIU Y, LIU H Y, FAN J L, et al. A survey of research and application of small object detection based on deep learning[J]. Acta Electronica Sinica, 2020, 48(3):590-601(in Chinese).
[38] TONG K, WU Y Q, ZHOU F. Recent advances in small object detection based on deep learning:A review[J]. Image and Vision Computing, 2020, 97:103910.
[39] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017:2999-3007.
[40] SINGH B, NAJIBI M, DAVIS L. SNIPER:Efficient multi-scale training[DB/OL]. arXiv preprint:1805.09300, 2018.
[41] LI Z M, PENG C, YU G, et al. DetNet:Design backbone for object detection[M]//Computer Vision-ECCV 2018. Cham:Springer International Publishing, 2018:339-354.
[42] KIM Y, KANG B N, KIM D. SAN:Learning relationship between convolutional features for multi-scale object detection[M]//Computer Vision-ECCV 2018. Cham:Springer International Publishing, 2018:328-343.
[43] GIDARIS S, KOMODAKIS N. Object detection via a multi-region and semantic segmentation-aware CNN model[C]//2015 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2015:1134-1142.
[44] ZHU Y S, ZHAO C Y, WANG J Q, et al. CoupleNet:Coupling global structure with local parts for object detection[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017:4146-4154.
[45] GUAN L T, WU Y, ZHAO J Q. SCAN:Semantic context aware network for accurate small object detection[J]. International Journal of Computational Intelligence Systems, 2018, 11(1):936.
[46] YU X H, GONG Y Q, JIANG N, et al. Scale match for tiny person detection[C]//2020 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway:IEEE Press, 2020:1246-1254.
[47] ZHU R, ZHANG S F, WANG X B, et al. ScratchDet:Training single-shot object detectors from scratch[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019:2263-2272.
[48] LU Y X, JAVIDI T, LAZEBNIK S. Adaptive object detection using adjacency and zoom prediction[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:2351-2359.
[49] BELLVER M, NIETO X G, ACOSTA F M, et al. Hierarchical object detection with deep reinforcement learning[C]//Neural Information Processing Systems, 2016:137-163.
[50] GAO M F, YU R C, LI A, et al. Dynamic zoom-in network for fast object detection in large images[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018:6926-6935.
[51] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO:Common objects in context[M]//Computer Vision-ECCV 2014. Cham:Springer International Publishing, 2014:740-755.
[52] WOO S, HWANG S, KWEON I S. StairNet:Top-down semantic aggregation for accurate one shot detection[C]//2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway:IEEE Press, 2018:1093-1102.
[53] ZHANG S F, WEN L Y, BIAN X, et al. Single-shot refinement neural network for object detection[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018:4203-4212.
[54] ZHAO Q J, SHENG T, WANG Y T, et al. M2Det:A single-shot object detector based on multi-level feature pyramid network[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019:9259-9266.
[55] LI H C, XIONG P F, AN J, et al. Pyramid attention network for semantic segmentation[DB/OL]. arXiv preprint:1805.10180V1, 2018.
[56] SHRIVASTAVA A, SUKTHANKAR R, MALIK J, et al. Beyond skip connections:Top-down modulation for object detection[DB/OL]. arXiv preprint:1612.06851, 2016.
[57] LI J, WANG Y B, WANG C G, et al. DSFD:Dual shot face detector[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019:5055-5064.
[58] LI Y H, CHEN Y T, WANG N Y, et al. Scale-aware trident networks for object detection[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019:6053-6062.
[59] ZHANG Y J, SHEN T. Small object detection with multiple receptive fields[J]. IOP Conference Series:Earth and Environmental Science, 2020, 440:032093.
[60] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8):2011-2023.
[61] JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks[C]//Neural Information Processing Systems, 2015:2017-2025.
[62] WOO S, PARK J, LEE J Y, et al. CBAM:Convolutional block attention module[M]//Computer Vision-ECCV 2018. Cham:Springer International Publishing, 2018:3-19.
[63] ZHU Y S, ZHAO C Y, GUO H Y, et al. Attention CoupleNet:Fully convolutional attention coupling network for object detection[J]. IEEE Transactions on Image Processing, 2019, 28(1):113-126.
[64] LI W, LI H L, WU Q B, et al. Simultaneously detecting and counting dense vehicles from drone images[J]. IEEE Transactions on Industrial Electronics, 2019, 66(12):9651-9662.
[65] 李红艳, 李春庚, 安居白, 等. 注意力机制改进卷积神经网络的遥感图像目标检测[J]. 中国图象图形学报, 2019, 24(8):1400-1408. LI H Y, LI C G, AN J B, et al. Attention mechanism improves CNN remote sensing image object detection[J]. Journal of Image and Graphics, 2019, 24(8):1400-1408(in Chinese).
[66] YING X, WANG Q, LI X W, et al. Multi-attention object detection model in remote sensing images based on multi-scale[J]. IEEE Access, 2019, 7:94508-94519.
[67] ZHANG S F, ZHU X Y, LEI Z, et al. S³FD:Single shot scale-invariant face detector[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017:192-201.
[68] LAW H, DENG J. CornerNet:Detecting objects as paired keypoints[C]//Computer Vision-ECCV 2018, 2018.
[69] ZHOU X Y, ZHUO J C, KRÄHENBÜHL P. Bottom-up object detection by grouping extreme and center points[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019:850-859.
[70] ZHOU X Y, WANG D Q, KRÄHENBÜHL P. Objects as points[DB/OL]. arXiv preprint:1904.07850, 2019
[71] WANG C, SHI J, YANG X Q, et al. Geospatial object detection via deconvolutional region proposal network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(8):3014-3027.
[72] LI Y Y, PEI X, HUANG Q, et al. Anchor-free single stage detector in remote sensing images based on multiscale dense path aggregation feature pyramid network[J]. IEEE Access, 2020, 8:63121-63133.
[73] JIANG B R, LUO R X, MAO J Y, et al. Acquisition of localization confidence for accurate object detection[C]//Computer Vision-ECCV 2018, 2018.
[74] YU J H, JIANG Y N, WANG Z Y, et al. UnitBox:An advanced object detection network[C]//Proceedings of the 24th ACM International Conference on Multimedia. New York:ACM, 2016:516-520.
[75] REZATOFIGHI H, TSOI N, GWAK J, et al. Generalized intersection over union:a metric and a loss for bounding box regression[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019:658-666.
[76] ZHENG Z H, WANG P, LIU W, et al. Distance-IoU loss:Faster and better learning for bounding box regression[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020:12993-13000.
[77] XIA G S, BAI X, DING J, et al. DOTA:A large-scale dataset for object detection in aerial images[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018:3974-3983.
[78] CHENG G, HAN J W, ZHOU P C, et al. Multi-class geospatial object detection and geographic image classification based on collection of part detectors[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 98:119-132.
[79] ZHU H G, CHEN X G, DAI W Q, et al. Orientation robust object detection in aerial images using deep convolutional neural network[C]//2015 IEEE International Conference on Image Processing (ICIP). Piscataway:IEEE Press, 2015:3735-3739.
[80] RAZAKARIVONY S, JURIE F. Vehicle detection in aerial imagery:A small target detection benchmark[J]. Journal of Visual Communication and Image Representation, 2016, 34:187-203.
[81] JAIN V, LEARNED-MILLER E. Fddb:A benchmark for face detection in unconstrained settings[R]. Boston:University of Massachusetts, 2010
[82] YANG S, LUO P, LOY C C, et al. WIDER FACE:A face detection benchmark[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:5525-5533.
[83] YANG B, YAN J J, LEI Z, et al. Fine-grained evaluation on face detection in the wild[C]//2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). Piscataway:IEEE Press, 2015:1-7.
[84] KEMELMACHER-SHLIZERMAN I, SEITZ S M, MILLER D, et al. The MegaFace benchmark:1 million faces for recognition at scale[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:4873-4882.
[85] DOLLAR P, WOJEK C, SCHIELE B, et al. Pedestrian detection:An evaluation of the state of the art[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(4):743-761.
[86] MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark[C]//2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). Piscataway:IEEE Press, 2017:3226-3229.
[87] ZHANG S S, BENENSON R, SCHIELE B. CityPersons:A diverse dataset for pedestrian detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017:4457-4465.
[88] CORDTS M, OMRAN M, RAMOS S, et al. The cityscapes dataset for semantic urban scene understanding[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:3213-3223.
[89] ZHU Z, LIANG D, ZHANG S H, et al. Traffic-sign detection and classification in the wild[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016:2110-2118.
[90] ZHANG J M, HUANG M T, JIN X K, et al. A real-time Chinese traffic sign detection algorithm based on modified YOLOv2[J]. Algorithms, 2017, 10(4):127.
[91] BEHRENDT K, NOVAK L, BOTROS R. A deep learning approach to traffic lights:Detection, tracking, and classification[C]//2017 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2017:1370-1377.
[92] LONG Y, GONG Y P, XIAO Z F, et al. Accurate object localization in remote sensing images based on convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5):2486-2498.
[93] PANG J M, LI C, SHI J P, et al. R²-CNN:Fast tiny object detection in large-scale remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8):5512-5524.
[94] HU P, RAMANAN D. Find tiny face[DB/OL]. arXiv preprint:1612.04402, 2016.
[95] TANG X, DU D K, HE Z Q, et al. PyramidBox:A context-assisted single shot face detector[C]//Computer Vision-ECCV 2018, 2018.
[96] MAO J Y, XIAO T T, JIANG Y N, et al. What can help pedestrian detection?[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017:6034-6043.
[97] LIU W, LIAO S C, HU W D, et al. Learning efficient single-stage pedestrian detectors by asymptotic localization fitting[C]//Computer Vision-ECCV 2018, 2018.
[98] CHENG P, LIU W, ZHANG Y F, et al. LOCO:Local context based faster R-CNN for small traffic sign detection[C]//MultiMedia Modeling, 2018.
[99] LU Y F, LU J M, ZHANG S H, et al. Traffic signal detection and classification in street views using an attention model[J]. Computational Visual Media, 2018, 4(3):253-266.

Options

Outlines

模态框（Modal）标题

Abstract

Cite this article

References