Please wait a minute...
img

官方微信

遥感技术与应用  2022, Vol. 37 Issue (4): 800-810    DOI: 10.11873/j.issn.1004-0323.2022.4.0800
深度学习专栏     
面向云覆盖的遥感影像时空融合深度学习方法及其应用
隋冰清1(),殷志祥1,吴鹏海1,2(),吴艳兰1,2
1.安徽大学 资源与环境工程学院,安徽 合肥 230601
2.安徽大学 物质科学与信息技术研究院,安徽 合肥 230601
Method and Application of Spatial-temporal Fusion for Cloud Coverage of Satellite Images on Deep Learning
Bingqing Sui1(),Zhixiang Yin1,Penghai Wu1,2(),Yan lan Wu1,2
1.School of Resources and Environmental Engineering,Anhui University,Hefei 230601,China
2.Institute of Physical Science and Information Technology,Anhui University,Hefei 230601,China
 全文: PDF(5253 KB)   HTML
摘要:

遥感影像时空融合是一种获取高时空分辨率数据的有效手段,但现有方法在选定基础数据对时要求预测时间低分辨率数据无云覆盖影响,这极大地限制了其应用潜力。为此,提出一种面向云覆盖的遥感影像时空融合方法,即在深度学习框架下,构建重建子网络恢复预测时刻云下缺失信息,将重建后的低分辨率影像与前后相邻时刻高、低分辨率数据对构建时空融合子网络,得到最终的融合影像。以安徽淮南采煤沉陷区Landsat和MODIS反射率数据为例,对预测时刻MODIS数据模拟不同缺失率的云污染;利用所提方法进行时空融合实验,进而比较深度学习与传统方法融合数据对水体信息的提取效果。结果表明:该方法融合结果各波段的RMSE和SSIM均取得较好的定量评价效果,且总体优于传统方法;沉陷区水体提取实验表明本方法水体提取结果更加接近真实观测影像。因此,该方法降低了时空融合对数据的限制要求,且具有更高的融合精度和更有效的应用性。

关键词: 云覆盖遥感时空融合深度学习重建水体提取    
Abstract:

Spatio-temporal fusion of remote sensing images is considered as an effective way to obtain high spatio-temporal resolution data. However, the existing methods require that the low-resolution data at the predicted time is not affected by cloud cover when the basic data pairs is selected, which greatly limits the application potential of the spatio-temporal fusion method. Thus, this article proposes a spatio-temporal fusion method based cloud-covered remote sensing image. Under the deep learning framework, there are two types of remote sensing data featured by high spatial resolution but low temporal resolution (HSLT) and the other type by low spatial resolution but high temporal resolution (LSHT). The reconstruction subnetwork is constructed to repair the missing information under the cloud coveraged area of LSHT at the prediction dates, and the reconstructed LSHT image and two prior HSLT images are integrated to obtain the final fusion result on the prediction date by the constructed spatiotemporal fusion subnetwork.We take the Landsat (HSLT) and MODIS (LSHT) reflectance data in the coal mining subsidence area of Huainan City, Anhui Province as an example, simulate cloud pollution with different missing rates on the MODIS data at the prediction time, Spatial-temporal fusion experiments are conducted with the proposed method, and then compare water information extraction effects of deep learning fusion data and traditional method fusion data.The results show that the proposed method achieves a good quantitative evaluation effect on the root mean square error and the structural similarity index of the fusion results in each band, and that the fusion results are generally superior to the traditional classical method. The experiment of water extraction in subsidence area clearly shows that the water body extraction result of the proposed method is generally closer to the real observation image. Therefore, the proposed method reduces the data limitation requirements of spatio-temporal fusion, and has higher fusion accuracy and more effective application than the classic traditional method.

Key words: Cloud cover    Remote sensing spatial-temporal fusion    Deep learning    Reconstruction    Water body extraction
收稿日期: 2021-08-11 出版日期: 2022-09-28
:  TP751  
基金资助: 安徽省科技重大专项(201903a07020014);安徽大学物质科学与信息技术研究院学科建设开放基金资助
通讯作者: 吴鹏海     E-mail: wuph@ahu.edu.cn
作者简介: 隋冰清(1992-),男,安徽阜阳人,硕士研究生,主要从事遥感时空融合方面的研究。E?mail:wuph@ahu.edu.cn
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
RSS
作者相关文章  
隋冰清
殷志祥
吴鹏海
吴艳兰

引用本文:

隋冰清,殷志祥,吴鹏海,吴艳兰. 面向云覆盖的遥感影像时空融合深度学习方法及其应用[J]. 遥感技术与应用, 2022, 37(4): 800-810.

Bingqing Sui,Zhixiang Yin,Penghai Wu,Yan lan Wu. Method and Application of Spatial-temporal Fusion for Cloud Coverage of Satellite Images on Deep Learning. Remote Sensing Technology and Application, 2022, 37(4): 800-810.

链接本文:

http://www.rsta.ac.cn/CN/10.11873/j.issn.1004-0323.2022.4.0800        http://www.rsta.ac.cn/CN/Y2022/V37/I4/800

图1  面向云覆盖的遥感影像时空融合示意图
图2  面向云覆盖的遥感影像时空融合深度学习模型总体网络结构
日期Landsat数据MODIS数据
2014-12-02OLIMOD09GA
2014-12-18OLIMOD09GA
2015-01-11ETM+MOD09GA
2015-01-19OLIMOD09GA
2015-02-12ETM+MOD09GA
2015-04-25rOLIMOD09GA
2015-05-19ETM+MOD09GA
2015-07-30OLIMOD09GA
2015-10-02OLIMOD09GA
2015-10-10ETM+MOD09GA
表1  2014—2015年淮南矿区Landsat和MODIS无云数据对

波段

名称

Landsat-7 ETM+Landsat-8 OLIMODIS(MOD09GA)
波段编号带宽/μm分辨率/m波段编号带宽/μm分辨率/m波段编号带宽/μm分辨率/m
绿色20.52—0.603030.53—0.593040.545—0.565500
红色30.63—0.693040.64—0.673010.620—0.670250
近红外40.77—0.903050.85—0.883020.841—0.876250
中红外51.55—1.753061.57—1.653061.628—1.652500
表2  Landsat和MODIS的光谱范围
图3  三对连续时刻“中红外、近红外、红色”三波段组合的Landsat和MODIS影像和不同掩膜类型
实验数据类型t1t2t3

第一组实验

训练集201412022014121820150111
201412182015011120150212
201501112015021220150425
201402122015042520150519
测试集201501112015011920150212

第二组实验

训练集201501192015021220150425
201502122015042520150730
201504252015073020151002
201507302015100220151010
测试集201504252015051920150730
表3  训练数据和预测数据分配情况
缺失率类型波段2015年1月19日2015年5月19日
ESTARFM本文方法ESTARFM本文方法
RMSESSIMRMSESSIMRMSESSIMRMSESSIM
33%绿色0.048 590.991 870.034 010.992 030.032 010.988 700.020 930.997 01
红色0.046 510.990 170.032 370.987 890.039 280.985 190.038 030.993 13
近红外0.071 190.985 570.057 200.988 500.075 050.978 280.069 600.982 32
中红外0.040 150.991 420.021 540.993 750.041 580.988 120.039 320.988 77
平均值0.051 610.991 470.037 050.989 150.046 980.985 070.041 950.990 31
59%绿色0.047 670.992 020.033 780.992 050.031 250.989 110.020 880.997 01
红色0.045 340.990 350.032 580.987 840.039 110.985 060.038 390.993 07
近红外0.072 910.984 700.057 900.985 140.078 430.978 110.067 100.982 60
中红外0.042 710.990 540.022 350.993 590.043 550.987 770.036 470.988 58
平均值0.052 160.988 530.036 650.990 260.048 090.985 010.040 780.990 32
67%绿色0.050 850.991 470.033 780.992 050.032 510.989 050.020 950.997 01
红色0.052 050.989 230.032 580.987 840.038 090.986 200.038 140.993 11
近红外0.073 600.985 740.062 380.983 990.083 700.978 400.065 400.982 70
中红外0.053 000.990 520.022 780.993 530.038 990.988 870.037 270.988 69
平均值0.062 230.989 240.037 880.991 140.048 320.985 630.040 440.990 38
表4  2015年1月19日和5月19日两种方法融合结果的定量对比(黑体加粗结果更好)
图4  不同缺失率下两种时空融合结果视觉比较
图5  不同缺失率下的两种时空融合结果的水体提取对比
指标

2015年1月19日

(缺失率33%)

2015年5月19日

(缺失率59%)

ESTARFM本文方法ESTARFM本文方法
用户精度/%76767380
制图精度/%74938594
总体精度/%92959295
Kappa系数0.710.830.730.83
表5  两种时空融合结果水体分类图的精度评价
1 Zhang Xiaochuan, Wang Jie. The analysis of eco-hydrological structure of Shengjin lake wetland based on spatial and temporal fusion technology of remote sensing[J]. Remote Sensing Technology and Application,2020, 35(5): 1109-1117.
1 张晓川, 王杰. 基于遥感时空融合的升金湖湿地生态水文结构分析[J]. 遥感技术与应用, 2020, 35(5): 1109-1117.
2 Dong Wenquan, Meng Jihua. Review of spatiotemporal fusion model of remote sensing data[J]. Remote Sensing of Land and Resources, 2018, 30(2): 1-11.
2 董文全, 蒙继华. 遥感数据时空融合研究进展及展望[J]. 国土资源遥感, 2018, 30(2): 1-11.
3 Zhu X L, Cai F Y, Tian J Q, et al. Spatiotemporal fusion of multisource remote sensing data:Literature survey,taxonomy, principles,applications,and future directions[J]. Remote Sensing, 2018, 204: 31-42. DOI: 10.3390/rs10040527 .
doi: 10.3390/rs10040527
4 Zhukov B, Oertel D, Lanzl F, et al. Unmixing-based multisensor multiresolution image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 1999, 37(3): 1212-1226. DOI: 10.1109/36.763276 .
doi: 10.1109/36.763276
5 Zurita-Milla R, Kaiser G, Clevers J, et al. Downscaling time series of MERIS full resolution data to monitor vegetation seasonal dynamics[J]. Remote Sensing of Environment, 2009, 113(9): 1874-1885. DOI: 10.1016/j.rse.2009.04.011 .
doi: 10.1016/j.rse.2009.04.011
6 Wu Mingquan, Wang Jie, Niu Zheng, et al. A model for spatial and temporal data fusion[J]. Journal of Infrared and Millimeter Waves, 2012,31(1): 82-86.
6 邬明权, 王洁, 牛铮, 等. 融合MODIS与Landsat数据生成高时间分辨率Landsat数据[J]. 红外与毫米波学报, 2012,31(1): 82-86.
7 Zhang W, Li A N, Jin H A, et al. An enhanced spatial and temporal data fusion model for fusing Landsat and MODIS surface reflectance to generate high temporal Landsat-like data[J].Remote Sensing,2013,5(10):5346-5368. DOI:10.3390/rs 5105346 .
doi: 10.3390/rs 5105346
8 Gao F, Masek J, Schwaller M, et al. On the blending of the landsat and MODIS surface reflectance:Predicting daily Landsat surface reflectance[J]. IEEE Transactions on Geoscience and Remote Sensing,2006,44(8):2207-2218. DOI: 10.1109/ TGRS.2006.872081 .
doi: 10.1109/ TGRS.2006.872081
9 Zhu X L, Chen J, Gao F, et al. An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions[J]. Remote Sensing of Environment, 2010, 114(11): 2610-2623. DOI: 10.1016/j.rse.2010.05.032 .
doi: 10.1016/j.rse.2010.05.032
10 Fu D J, Chen B Z, Wang J, et al. An improved image fusion approach based on enhanced spatial and temporal the adaptive reflectance fusion model[J]. Remote Sensing, 2013, 5(12): 6346-6360. DOI: 10.3390/rs5126346 .
doi: 10.3390/rs5126346
11 Wang Q M, Zhang Y H, Onojeghuo A O, et al. Enhancing spatio-temporal fusion of MODIS and Landsat data by incorporating 250 m MODIS Data[J].IEEE Journal of Selected To-pics in Applied Earth Observations & Remote Sensing, 2017,10(9):4116-4123. DOI:10.1109/JSTARS.2017.2701643 .
doi: 10.1109/JSTARS.2017.2701643
12 Shen H, Wu P H, Liu L Y, et al. A spatial and temporal reflectance fusion model considering sensor observation differences[J]. International Journal of Remote Sensing, 2013, 34(12): 4367-4383. DOI: 10.1080/01431161.2013.777488 .
doi: 10.1080/01431161.2013.777488
13 Wang J, Huang B. A Rigorously-weighted spatiotemporal fusion model with uncertainty analysis[J]. Remote Sensing, 2017, 9(10): 990. DOI: 10.3390/rs9100900 .
doi: 10.3390/rs9100900
14 Wang Q, Atkinson P M. Spatio-temporal fusion for daily Sentinel-2 images[J]. Remote Sensing of Environment, 2018, 204: 31-42. DOI: 10.1016/j.rse.2017.10.046 .
doi: 10.1016/j.rse.2017.10.046
15 Zhu X L, Helmer E, Gao F, et al. A flexible spatiotemporal method for fusing satellite images with different resolutions[J]. Remote Sensing of Environment, 2016, 172(11): 165-177. DOI: 10.1016/j.rse.2015.11.016 .
doi: 10.1016/j.rse.2015.11.016
16 Zhao Y Q, Huang B, Song H H. A robust adaptive spatial and temporal image fusion model for complex land surface changes[J]. Remote Sensing of Environment, 2018, 208(2): 42-62. DOI: 10.1016/j.rse.2018.02.009 .
doi: 10.1016/j.rse.2018.02.009
17 Song H H, Huang B. Spatiotemporal satellite image fusion through one-pair image learning[J]. IEEE Transactions on Geoscience and Remote Sensing,2013, 51(4): 1883-1896. DOI: 10.1109/TGRS.2012.2213095 .
doi: 10.1109/TGRS.2012.2213095
18 Wu B, Huang B, Zhang L P. An drror-bound-regularized sparse coding for spatiotemporal reflectance fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(12): 6791-6803. DOI: 10.1109/TGRS.2015.2448100 .
doi: 10.1109/TGRS.2015.2448100
19 Wei J B, Wang L Z, Liu P,et al. Spatiotemporal fusion of MODIS and Landsat-7 reflectance images via compressed sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017,55(12):7126-7139. DOI:10.1109/TGRS. 2017. 2742529 .
doi: 10.1109/TGRS. 2017. 2742529
20 Song H H, Liu Q S, Wang G J, et al. Spatiotemporal satellite image fusion using deep convolutional neural networks[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3): 821-829. DOI: 10.1109/JSTARS.2018.2797894 .
doi: 10.1109/JSTARS.2018.2797894
21 Tan Z Y, Yue P, Di L P, et al. Deriving high spatiotemporal remote sensing images using deep convolutional network[J]. Remote Sensing,2018,10(7):1066. DOI:10.3390/rs10071066 .
doi: 10.3390/rs10071066
22 Liu X, Deng C W, Chanussot J, et al. StfNet: A two-stream convolutional neural network for spatiotemporal image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019,57(9):6552-6564. DOI:10.1109/TGRS.2019.2907310 .
doi: 10.1109/TGRS.2019.2907310
23 Yao Kaixuan, Cao Feilong. Spatial-temporal fusion algorithm for remote sensing images based on multi-input dense connected neural network[J]. Pattern Recognition and Artificial Intelligence, 2019,32(5):429-435.
23 姚凯旋, 曹飞龙. 基于多输入密集连接神经网络的遥感影像时空融合方法[J]. 模式识别与人工智能, 2019,32(5):429-435.
24 Zhang H Y, Song Y Y, Han C, et al. Remote sensing image spatiotemporal fusion using a generative adversarial network[J].IEEE Transactions on Geoscience and Remote Sensing,2020,59(5):4273-4286. DOI:10.1109/TGRS.2020.3010530 .
doi: 10.1109/TGRS.2020.3010530
25 Wu P H, Yin Z X, Zeng C, et al. Spatially continuous and high-resolution land surface temperature product generation: A review of reconstruction and spatiotemporal fusion techniques[J]. IEEE Geoscience and Remote Sensing Magazine, 2021,9(3):112-137. DOI: 10.1109/MGRS.2021.3050782 .
doi: 10.1109/MGRS.2021.3050782
26 Liu Q, Meng Meng X C, Shao F, et al. PSTAF-GAN: Progressive spatio-temporal attention fusion method based on generative adversarial network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-13. DOI: 10.1109/TGRS.2022.3161563 .
doi: 10.1109/TGRS.2022.3161563
27 Shen H F, Li X H, Cheng Q, et al. Missing information reconstruction of remote sensing data: A technical review[J]. IEEE Geoscience & Remote Sensing Magazine, 2015, 3(3): 61-85. DOI: 10.1109/MGRS.2015.2441912 .
doi: 10.1109/MGRS.2015.2441912
28 Chen Zihan, Wang Feng, Xu Ning, et al. Cloud removal by fusing optical and SAR images based on improved PCNN in NSST domain[J]. Remote Sensing Technology and Application, 2021, 36(4): 810-819.
28 陈子涵, 王峰, 许宁, 等. 基于改进NSST-PCNN的光学与SAR图像融合去云方法[J]. 遥感技术与应用, 2021, 36(4): 810-819.
29 Yin Z X, Wu P H, Foody G M, et al. Spatiotemporal fusion of land surface temperature based on a convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing,2020,59(2):1808-1822. DOI: 10.1109/TGRS. 2020.2999943 .
doi: 10.1109/TGRS. 2020.2999943
30 Wu P H, Yin Z X, Yang H, et al. Reconstructing geostationary satellite land surface temperature imagery based on a multiscale feature connected convolutional neural network[J]. Remote Sensing,2019, 11(3): 300. DOI: 10.3390/rs11030300 .
doi: 10.3390/rs11030300
31 Zhang Q, Yuan Q Q, Li J, et al. Thick cloud and cloud shadow removal in multitemporal imagery using progressively spatio-temporal patch group deep learning[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 162: 148-160. DOI: 10.1016/j.isprsjprs.2020.02008 .
doi: 10.1016/j.isprsjprs.2020.02008
32 Xia M, Jia K. Reconstructing missing information of remote sensing data contaminated by large and thick clouds based on an improved multitemporal dictionary learning method[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-14. DOI: 10.1109/TGRS.2021.3095067 .
doi: 10.1109/TGRS.2021.3095067
33 Xu Hanqiu. A study on information extraction of water body with the Modified Normalized Difference Water Index(MNDWI)[J]. National Remote Sensing Bulletin,2005,9(5):589-595.
33 徐涵秋. 利用改进的归一化差异水体指数(MNDWI)提取水体信息的研究[J]. 遥感学报, 2005, 9(5): 589-595.
[1] 冯崎,王琦,黄海兰,王征强. 基于Sentinel-1双极化数据改进水体提取的Otsu算法[J]. 遥感技术与应用, 2022, 37(4): 1003-1011.
[2] 于枫世,隋毅,王常颖,初佳兰. 基于深度学习的高分辨率卫星遥感影像围填海检测识别[J]. 遥感技术与应用, 2022, 37(4): 789-799.
[3] 史佳睿,申茜,彭红春,李利伟,姚月,汪明秀,王茹. 基于三种高分辨率影像样本迁移学习的城市水体提取研究[J]. 遥感技术与应用, 2022, 37(3): 731-738.
[4] 付涵,范湘涛,严珍珍,杜小平. 基于深度学习的遥感图像目标检测技术研究进展[J]. 遥感技术与应用, 2022, 37(2): 290-305.
[5] 吕冬梅,马玥,李华朋. 基于CNN的吉林一号卫星城市土地覆被制图潜力评估[J]. 遥感技术与应用, 2022, 37(2): 368-378.
[6] 沈洁,辛晓平,张景,苗晨,王旭,丁蕾,沈贝贝. 基于Cubist的中国植被区域叶绿素荧光数据重建[J]. 遥感技术与应用, 2022, 37(1): 244-252.
[7] 吴川虎,陶于祥,罗小波. 基于Google Earth Engine的重庆市植被指数长时间序列S-G滤波方法的改进与实现[J]. 遥感技术与应用, 2021, 36(5): 1189-1198.
[8] 路春燕,雷依凡,苏颖,黄雨菲,刘明月,贾明明. 基于面向对象—深度学习的闽东南低海拔海岸带地区湿地动态遥感分析[J]. 遥感技术与应用, 2021, 36(4): 713-727.
[9] 郑磊,何直蒙,丁海勇. 基于ENVINet5的高分辨率遥感影像稀疏塑料大棚提取研究[J]. 遥感技术与应用, 2021, 36(4): 908-915.
[10] 胡乃勋,陈涛,甄娜,牛瑞卿. 基于卷积神经网络的面向对象露天采场提取[J]. 遥感技术与应用, 2021, 36(2): 265-274.
[11] 林娜,冯丽蓉,张小青. 基于优化Faster-RCNN的遥感影像飞机检测[J]. 遥感技术与应用, 2021, 36(2): 275-284.
[12] 李庆,陈俊杰,李庆亭,李柏鹏,卢凯旋,昝露洋,陈正超. 基于SSD模型的京津冀地区尾矿库检测[J]. 遥感技术与应用, 2021, 36(2): 293-303.
[13] 徐瑾昊,冯敏,王建邦,冉有华,祁元,杨联安,李新. 基于高分遥感数据和深度学习的石冰川自动提取研究[J]. 遥感技术与应用, 2020, 35(6): 1329-1336.
[14] 张永宏,曹海啸,阚希. 基于FY-4A/AGRI时空特征融合的新疆地区积雪判识[J]. 遥感技术与应用, 2020, 35(6): 1337-1347.
[15] 杨瑞,祁元,苏阳. 深度学习U-Net方法及其在高分辨卫星影像分类中的应用[J]. 遥感技术与应用, 2020, 35(4): 767-774.