Due to technological constraints, very high resolution Earth observation satellites (e.g. WorldView-3 by Maxar Technologies) typically provide a single panchromatic band at the highest spatial resolution (e.g., 30 cm) coupled with a lower-resolution (e.g., 4 times lower: 1.2m) multispectral image (e.g., having 8 narrower spectral bands). An image fusion process which combines the low-resolution multispectral image with the corresponding panchromatic band, aimed to provide a higher (e.g., 30cm) resolution multispectral image is called pansharpening.
Although pansharpening is a longstanding problem, deep learning-based solutions have gained large popularity since 2016, when our popular method (Pansharpening by convolutional neural networks - PNN) appeared in literature as first example of convolutional neural network (CNN) specifically designed for pansharpening. As of today, PNN is a well-established baseline CNN method for comparison. Since 2016 the GRIP team has continuously worked on pansharpening producing more and more competitive solutions. Besides, the pansharpening quality assessment problem has also been put under investigation, giving raise to some recent valuable contributions.
In recent years, deep learning techniques for pansharpening multiresolution images have gained increasing interest. Due to the lack of ground truth data, most deep learning solutions rely on synthetic reduced-resolution data for supervised training. This approach has limitations due to the statistical mismatch between real full-resolution and synthetic reduced-resolution data, which affects the models’ generalization capacity. Consequently, there has been a shift towards unsupervised learning frameworks for pansharpening deep learning-based techniques. Unsupervised schemes require defining sophisticated loss functions with at least two components: one for spectral quality, ensuring consistency between the pansharpened image and the input multispectral component, and another for spatial quality, ensuring consistency between the output and the panchromatic input. Despite promising results, there has been limited investigation into the interaction and balance of these loss terms to ensure stability and accuracy. This work explores how unsupervised spatial and spectral consistency losses can be reliably combined preserving the outocome quality. By examining these interactions, we propose a general rule for balancing the two loss components to enhance the stability and performance of unsupervised pansharpening models. Experiments on three state-of-the-art algorithms using WorldView-3 images demonstrate that methods trained with the proposed framework achieve good performance in terms of visual quality and numerical indexes.
Hyperspectral (HS) pansharpening has received a growing interest in the last few years as testified by a large number of research papers and challenges. It consists in a pixel-level fusion between a lower resolution HS datacube and a higher resolution single-band image, the panchromatic (PAN) image, with the goal of providing an HS datacube at PAN resolution. Due to their powerful representational capabilities, deep learning models have succeeded to provide unprecedented results on many general-purpose image processing tasks. However, when moving to domain-specific problems, as in this case, the advantages with respect to traditional model-based approaches are much lesser clear-cut due to several contextual reasons. Scarcity of training data, lack of ground truth (GT), and data shape variability are some such factors that limit the generalization capacity of the state-of-the-art deep learning networks for HS pansharpening. To cope with these limitations, in this work, we propose a new deep learning method, which inherits a simple single-band unsupervised pansharpening model nested in a sequential band-wise adaptive scheme, where each band is pansharpened refining the model tuned on the preceding one. By doing so, a simple model is propagated along the wavelength dimension, adaptively and flexibly, with no need to have a fixed number of spectral bands, and, with no need to dispose of large, expensive, and labeled training datasets. The proposed method achieves very good results on our datasets, outperforming both traditional and deep learning reference methods.
This work proposes a simple yet effective method to adapt unsupervised convolutional neural networks (CNNs) from multispectral (MS) to hyperspectral (HS) pansharpening. Thus, it focuses on the fusion of a single high-resolution panchromatic (PAN) band with a low-resolution HS data cube. This is achieved by means of a decorrelation transform, following the principal component analysis (PCA) approach, which enables the compression of a significant portion of the HS image energy into a few bands. Afterward, a suitably adapted pansharpening network designed for four spectral bands is used to super-resolve only the principal components (PCs). Experiments demonstrate high performance in both quantitative and qualitative evaluations, favorably comparing against state-of-the-art methods.
In latest years, deep learning (DL) has gained a leading role in the pansharpening of multiresolution images. Given the lack of ground truth data, most DL-based methods carry out supervised training in a reduced-resolution domain. However, models trained on downsized images tend to perform poorly on high-resolution target images. For this reason, several research groups are now turning to unsupervised training in the full-resolution domain, through the definition of appropriate loss functions and training paradigms. In this context, we have recently proposed a full-resolution training framework that can be applied to many existing architectures. Here, we propose a new DL-based pansharpening model that fully exploits the potential of this approach and provides cutting-edge performance. Besides architectural improvements with respect to previous work, such as the use of residual attention modules, the proposed model features a novel loss function that jointly promotes the spectral and spatial quality of the pansharpened data. In addition, thanks to a new fine-tuning strategy, it improves inference-time adaptation to target images. Experiments on a large variety of test images, performed in challenging scenarios, demonstrate that the proposed method compares favorably with the state-of-the-art both in terms of numerical results and visual output.
In the last few years, there has been a renewed interest in data fusion techniques, and, in particular, in pansharpening due to a paradigm shift from model-based to data-driven approaches, supported by the recent advances in deep learning. Although a plethora of convolutional neural networks (CNN) for pansharpening have been devised, some fundamental issues still wait for answers. Among these, cross-scale and cross-datasets generalization capabilities are probably the most urgent ones since most of the current networks are trained at a different scale (reduced-resolution), and, in general, they are well-fitted on some datasets but fail on others. A recent attempt to address both these issues leverages on a target-adaptive inference scheme operating with a suitable full-resolution loss. On the downside, such an approach pays an additional computational overhead due to the adaptation phase. In this work, we propose a variant of this method with an effective target-adaptation scheme that allows for the reduction in inference time by a factor of ten, on average, without accuracy loss. A wide set of experiments carried out on three different datasets, GeoEye-1, WorldView-2 and WorldView-3, prove the computational gain obtained while keeping top accuracy scores compared to state-of-the-art methods, both model-based and deep-learning ones. The generality of the proposed solution has also been validated, applying the new adaptation framework to different CNN models.
In this paper, we propose a full-resolution training framework for deep learning-based pansharpening. The framework is fully general and can be used for any deep learning-based pansharpening model. Training takes place in the high-resolution domain, relying only on the original data, thus avoiding any loss of information. To ensure spectral and spatial fidelity, a suitable two-component loss is defined.
A reliable quality assessment procedure for pansharpening methods is of critical importance for the development of the related solutions. Unfortunately, the lack of ground truths to be used as guidance for an objective evaluation has pushed the community to resort to no-reference quality indicators. In this work, we propose an alternative no-reference full-resolution assessment framework. On one side, we introduce a protocol, namely the reprojection protocol, to take care of the spectral consistency issue. On the other side, a new index of the spatial consistency between the pansharpened image and the panchromatic band at full resolution is also proposed.
In the last 30 years, several approaches to deal with the pansharpening issue have been proposed. However, the reproducibility of these methods is often limited, making the comparison with the state of the art hard to achieve. Thus, to fill this gap, we propose a new benchmark consisting of recent advances in MS pansharpening. All of the implementations (of methods and accuracy indicators) have been collected in a MATLAB toolbox that is made available to the community.
In this work we introduce a cross-scale learning strategy for CNN pansharpening models. Early CNN approaches resort to a resolution downgrading process to produce suitable training samples. As a consequence, the actual performance at the target resolution of the models trained at a reduced scale is an open issue. To cope with this shortcoming we propose a more complex loss computation that involves simultaneously reduced and full resolution training samples.
We recently proposed a convolutional neural network (CNN) for remote sensing image pansharpening obtaining a significant performance gain over the state of the art. In this paper, we explore a number of architectural and training variations to this baseline, achieving further performance gains with a lightweight network that trains very fast. Leveraging on this latter property, we propose a target-adaptive usage modality that ensures a very good performance also in the presence of a mismatch with respect to the training set and even across different sensors. The proposed method, published online as an off-the-shelf software tool, allows users to perform fast and high-quality CNN-based pansharpening of their own target images on general-purpose hardware.
A new pansharpening method is proposed, based on convolutional neural networks. A simple and effective three-layer architecture recently proposed for super-resolution is suitably redesigned to address the pansharpening problem. To the best of our knowledge this is the first convolutional neural network specifically designed for pansharpening. This work ranks among the top-20 most cited papers of the Remote Sensing journal.
