ACTA AERONAUTICAET ASTRONAUTICA SINICA >
Hyperspectral-LiDAR joint classification method based on vision-language pre-trained models
Received date: 2025-07-28
Revised date: 2025-09-08
Accepted date: 2025-09-30
Online published: 2025-10-24
Supported by
General Program of National Natural Science Foundation of China(62571387);the Fundamental Research Funds for the Central Universities(YJSJ25014)
To address the challenge of inaccurate land-cover classification caused by differences in spatial resolution， data heterogeneity， and limited labeled samples in multimodal remote sensing data， we investigate the joint classification of Hyperspectral Imagery （HSI） and LiDAR data. We propose a Semantic-aware Cross-modal Fusion Network （SCF-Net）. First， a lightweight patch encoder transforms the input data into RGB-compatible feature maps， which are then fed into a Contrastive Language-Image Pre-training （CLIP）-based visual encoder enhanced with learnable prompts. To efficiently integrate multimodal information， an adaptive cross-modal fusion architecture is employed， featuring grouped linear projection and a relation-aware interaction module that enables dynamic spatial feature exchange at low computational cost. For semantic discrimination， attribute-category textual prompts are generated， and classification is performed by computing the cosine similarity between visual and textual embeddings， followed by a TopK attribute averaging strategy. Experiments on the Houston 2013， MUUFL， and Trento datasets demonstrate that SCF-Net outperforms eight state-of-the-art fusion methods， achieving improvements of over 2.88% in overall accuracy， 2.69% in average accuracy， and 3.02% in Kappa coefficient， while maintaining high parameter efficiency. Ablation studies further validate the effectiveness of each component. This network offers a novel paradigm for integrating multimodal remote sensing data with large-scale vision-language pre-trained models in complex classification tasks.
Xu TANG , Feng GU , Jingjing MA , Xiangrong ZHANG . Hyperspectral-LiDAR joint classification method based on vision-language pre-trained models[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2026 , 47(10) : 632630 -632630 . DOI: 10.7527/S1000-6893.2025.32630
| [1] | XU K， WANG B J， ZHU Z， et al. A contrastive learning enhanced adaptive multimodal fusion network for hyperspectral and LiDAR data classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 4700319. |
| [2] | CHANG H H， BI H X， LI F， et al. Deep symmetric fusion transformer for multimodal remote sensing data classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5644115. |
| [3] | 李晓航， 周建江. 基于自适应记忆长度的多尺度模态融合网络［J］. 航空学报， 2023， 44（22）： 628977. |
| LI X H， ZHOU J J. Multi-scale modality fusion network based on adaptive memory length［J］. Acta Aeronautica et Astronautica Sinica， 2023， 44（22）： 628977 （in Chinese）. | |
| [4] | ZHU F， SHI C P， SHI K J， et al. Joint classification of hyperspectral and LiDAR data using hierarchical multimodal feature aggregation-based multihead axial attention transformer［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 5503817. |
| [5] | ZHANG M M， ZHAO Y Y， CHEN R J， et al. Remote sensing collaborative classification using multimodal adaptive modulation network［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5529512. |
| [6] | MENG X C， ZHANG S F， LIU Q， et al. Uncertain category-aware fusion network for hyperspectral and LiDAR joint classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5523015. |
| [7] | LIAO D L， WANG Q S， LAI T， et al. Joint classification of hyperspectral and LiDAR data based on mamba［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5530915. |
| [8] | HU W S， LI W， LI H C， et al. Global clue-guided cross-memory quaternion transformer network for multisource remote sensing data classification［J］. IEEE Transactions on Neural Networks and Learning Systems， 2025， 36（4）： 7357-7371. |
| [9] | CAI Y M， ZHANG Z J， LIU X B， et al. Learning unified anchor graph for joint clustering of hyperspectral and LiDAR data［J］. IEEE Transactions on Neural Networks and Learning Systems， 2025， 36（4）： 6341-6354. |
| [10] | 冯燕， 贾应彪， 曹宇明， 等. 高光谱图像压缩感知投影与复合正则重构［J］. 航空学报， 2012， 33（8）： 1466-1473. |
| FENG Y， JIA Y B， CAO Y M， et al. Compressed sensing projection and compound regularizer reconstruction for hyperspectral images［J］. Acta Aeronautica et Astronautica Sinica， 2012， 33（8）： 1466-1473 （in Chinese）. | |
| [11] | YU W B， HUANG H， SHEN Y， et al. IamCSC： Intuitive assimilation modality driven crossmodal subspace clustering for land-cover identification and hyperspectral-LiDAR fusion［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 5505013. |
| [12] | HU Y T， HU Q W， LI J Y. CMINet： A unified cross-modal integration framework for crop classification from satellite image time series［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 4402213. |
| [13] | WANG L F， MEI S H， WANG Y， et al. CAMCFormer： Cross-attention and multicorrelation aided transformer for few-shot object detection in optical remote sensing images［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 5613316. |
| [14] | FU X Y， ZHOU X， FU Y W， et al. Progressive semantic enhancement network for hyperspectral and LiDAR classification［J］. IEEE Transactions on Neural Networks and Learning Systems， 2025， 36（6）： 11327-11339. |
| [15] | GUO F M， LI Z W， REN G B， et al. Instance-wise domain generalization for cross-scene wetland classification with hyperspectral and LiDAR data［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 5501212. |
| [16] | DONG W Q， ZHANG T， QU J H， et al. Multibranch feature fusion network with self-and cross-guided attention for hyperspectral and LiDAR classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5530612. |
| [17] | FENG Y N， JIN J R， YIN Y， et al. MCFT： Multimodal contrastive fusion transformer for classification of hyperspectral image and LiDAR data［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5537017. |
| [18] | KONG Y， YU S C， CHENG Y H， et al. Joint classification of hyperspectral images and LiDAR data based on candidate pseudo labels pruning and dual mixture of experts［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 5505912. |
| [19] | YANG C， ZHAO M H， BRUZZONE L， et al. Cross-modality adaptive feature fusion for multitype and multiscale impact craters identification on Mars［J］. IEEE Transactions on Geoscience and Remote Sensing， 2025， 63： 5613013. |
| [20] | WANG X H， SONG L Y， FENG Y N， et al. S3F2Net： Spatial-spectral-structural feature fusion network for hyperspectral image and LiDAR data classification［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2025， 35（5）： 4801-4815. |
| [21] | JIN X D， GU Y F， LIU T Z， et al. Supervoxel-based intrinsic scene properties from hyperspectral images and LiDAR［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5510613. |
| [22] | ZHU B E， TANG K H， SUN Q R， et al. Generalized logit adjustment： Calibrating fine-tuned models by removing label bias in foundation models［C］∥37th International Conference on Neural Information Processing Systems. 2023. |
| [23] | MALLADI S， GAO T Y， NICHANI E， et al. Fine-tuning language models with just forward passes［C］∥37th International Conference on Neural Information Processing Systems. 2023. |
| [24] | HU E J， SHEN Y L， WALLIS P， et al. LoRA： Low-rank adaptation of large language models［C］∥10th International Conference on Learning Representations. 2022. |
| [25] | JIA M L， TANG L M， CHEN B C， et al. Visual prompt tuning［C］∥European Conference on Computer Vision 2022. 2022： 709-727. |
| [26] | HOULSBY N， GIURGIU A， JASTRZEBSKI S， et al. Parameter-efficient transfer learning for NLP［C］∥International Conference on Machine Learning. 2019. |
| [27] | QU J H， YANG Y B， DONG W Q， et al. LDS2AE： Local diffusion shared-specific autoencoder for multimodal remote sensing image classification with arbitrary missing modalities［C］∥AAAI Conference on Artificial Intelligence. 2024： 14731-14739. |
| [28] | PEDERGNANA M， MARPU P R， DALLA MURA M， et al. Classification of remote sensing optical and LiDAR data using extended attribute profiles［J］. IEEE Journal of Selected Topics in Signal Processing， 2012， 6（7）： 856-865. |
| [29] | CHINI M， PIERDICCA N， EMERY W J. Exploiting SAR and VHR optical images to quantify damage caused by the 2003 bam earthquake［J］. IEEE Transactions on Geoscience and Remote Sensing， 2009， 47（1）： 145-152. |
| [30] | RASTI B， GHAMISI P. Remote sensing image classification using subspace sensor fusion［J］. Information Fusion， 2020， 64： 121-130. |
| [31] | RASTI B， ULFARSSON M O， SVEINSSON J R. Hyperspectral feature extraction using total variation component analysis［J］. IEEE Transactions on Geoscience and Remote Sensing， 2016， 54（12）： 6976-6985. |
| [32] | ROY S K， SUKUL A， JAMALI A， et al. Cross hyperspectral and LiDAR attention transformer： An extended self-attention for land use and land cover classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5512815. |
| [33] | DING K X， LU T， FU W， et al. Global-local transformer network for HSI and LiDAR data joint classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5541213. |
| [34] | ROY S K， DERIA A， HONG D F， et al. Hyperspectral and LiDAR data classification using joint CNNs and morphological feature learning［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5530416. |
| [35] | HANG R L， LI Z， GHAMISI P， et al. Classification of hyperspectral and LiDAR data using coupled CNNs［J］. IEEE Transactions on Geoscience and Remote Sensing， 2020， 58（7）： 4939-4950. |
| [36] | JIA S， ZHOU X， JIANG S G， et al. Collaborative contrastive learning for hyperspectral and LiDAR classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2023， 61： 5507714. |
| [37] | XIA S X， ZHANG X H， MENG H Y， et al. Ternary modality contrastive learning for hyperspectral and LiDAR data classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5522017. |
| [38] | AGIZA A， NESEEM M， REDA S. MTLoRA： A low-rank adaptation approach for efficient multi-task learning［C］∥2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2024： 16196-16205. |
| [39] | WU J L， HU X， WANG Y Q， et al. Omni-SMoLA： Boosting generalist multimodal models with soft mixture of low-rank experts［C］∥2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2024： 14205-14215. |
| [40] | IMTIAZ HOSSAIN M R， SIAM M， SIGAL L， et al. Visual prompting for generalized few-shot segmentation： A multi-scale approach［C］∥2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2024： 23470-23480. |
| [41] | KIM B， YU J， HWANG S J. ECLIPSE： Efficient continual learning in panoptic segmentation with visual prompt tuning［C］∥2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2024： 3346-3356. |
| [42] | KONG Y， CHENG Y H， CHEN Y， et al. Joint classification of hyperspectral image and LiDAR data based on spectral prompt tuning［J］. IEEE Transactions on Geoscience and Remote Sensing. Piscataway： IEEE Press， 2024， 62： 5521312. |
| [43] | ZHU P F， SUN Y， CAO B， et al. Task-customized mixture of adapters for general image fusion［C］∥2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2024： 7099-7108. |
| [44] | YIN D S， YANG Y R， WANG Z C， et al. 1% VS 100%： Parameter-efficient low rank adapter for dense predictions［C］∥2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2023： 20116-20126. |
| [45] | GE C R， DU Q， SUN W W， et al. Deep residual network-based fusion framework for hyperspectral and LiDAR data［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2021， 14： 2458-2472. |
| [46] | DEBES C， MERENTITIS A， HEREMANS R， et al. Hyperspectral and LiDAR data fusion： Outcome of the 2013 GRSS data fusion contest［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2014， 7（6）： 2405-2418. |
| [47] | LU T， DING K X， FU W， et al. Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data［J］. Information Fusion， 2023， 93： 118-131. |
| [48] | Congalton R G. A review of assessing the accuracy of classifications of remotely sensed data［J］. Remote Sensing of Environment， 1991， 37（1）： 35-46. |
| [49] | ZHANG M M， LI W， TAO R， et al. Information fusion for classification of hyperspectral and LiDAR data using IP-CNN［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5506812. |
| [50] | ZHAO X D， TAO R， LI W， et al. Joint classification of hyperspectral and LiDAR data using hierarchical random walk and deep CNN architecture［J］. IEEE Transactions on Geoscience and Remote Sensing， 2020， 58（10）： 7355-7370. |
| [51] | LI J J， LIU Y Z， SONG R， et al. HyperMLP： Superpixel prior and feature aggregated perceptron networks for hyperspectral and LiDAR hybrid classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 5505614. |
| [52] | SUN L， WANG X Y， ZHENG Y H， et al. Multiscale 3-D-2-D mixed CNN and lightweight attention-free transformer for hyperspectral and LiDAR classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024， 62： 2100116. |
| [53] | Gao H， Feng H， Zhang Y， et al. Interactive enhanced network based on multihead self-attention and graph convolution for classification of hyperspectral and lidar data［J］. IEEE Transactions on Geoscience and Remote Sensing， 2024. |
| [54] | Arora S， Hu W， Kothari P K， An analysis of the t-SNE algorithm for data visualization［C］∥31st Annual Conference on Learning Theory. 2018. |
/
| 〈 |
|
〉 |
CJA