/

Processing data from storage

How to process data from your User Storage in a workflow


Introduction

This notebook describes how to process data from your User Storage. Please bear in mind that currently only data from Airbus is processable from storage. We aim to extend this functionality to other types of data in storage.

The reason for this limitation is that this functionality relies on the presence of a data.json on the platform, which is a GeoJSON file with the asset metadata. For data retrieved via a block, the block generates it. For data retrieved from the long term archive, this feature generates the data.json file.

There are then three possile use cases:

  1. Data that you retrieved via a block (Reflectance or Display product) through the OneAtlas API (Airbus).
  2. Data that you retrieved via an order (Reflectance product) through the OneAtlas API (Airbus).
  3. Data that you obtained through satellite tasking from Airbus and that was delivered to your storage.

Here we describe how to do it in the console but bear in mind that this is a block intended for advanced users, not for beginners.

Building the workflow

First we need to create a workflow that makes use of this data. For the sake of simplicity, we are going to use the pansharpening block to pansharpen an image that was delivered to our storage from a tasking operation. This image is from a Pléiades Reflectance product, therefore it is delivered with 5 bands: Red, Green, Blue, Near InfraRed (NIR) and Panchromatic (shades of gray). The highest resolution band is the panchromatic and through pansharpening we use the additional information in the panchromatic band to enrich the information in the RGB bands, so that we have as a result a higher resolution RGB image.

Workflow for pansharpening an image from storage
Workflow for pansharpening an image from storage

Configuring the job

The next step is configuring the job, by setting up the respective parameters for each block.

In this case, we will only set the parameters for the Processing from Storage block. This block accepts a single parameter: an array of asset IDs.

These are the asset IDs of the images that you wish to pansharpen from your user storage. In this case we use a single asset, hence the array has a single element.

We get the asset ID by selecting the three dots at the previous to the right most column of the asset listing.

Getting the assert ID from the User Storage asset list
Getting the assert ID from the User Storage asset list

In this case, the asset ID is: 11f8498e-8568-41fb-892c-aa71fc0b8376. This is the single value of the asset ID array that the Processing from Storage block takes as a single parameter.

The job configuration parameters are:

{
  "pansharpen:1": {
    "method": "SFIM",
    "clip_to_aoi": false,
    "include_pan": false
  },
  "up42-processing-from-storage:1": {
    "asset_ids": [
      "11f8498e-8568-41fb-892c-aa71fc0b8376"
    ]
  }
}

The parameters of the pansharpening block are the default ones and for the Processing from Storage block, we set the array asset_ids to have a single element which is the ID of the asset we want to pansharpen.

For multiple assets to pansharpen, there will then be corresponding array elements. This is a Javascript array and a comma separates each entry. For example, if we also had an asset with the ID: c57e852d-4ca9-486b-8f55-9b1d578eda7b the job configuration would be:

{
  "pansharpen:1": {
    "method": "SFIM",
    "clip_to_aoi": false,
    "include_pan": false
  },
  "up42-processing-from-storage:1": {
    "asset_ids": [
      "11f8498e-8568-41fb-892c-aa71fc0b8376",
       "c57e852d-4ca9-486b-8f55-9b1d578eda7b"
    ]
  }
}

Job parameters for pansharpening an image from storage
Job parameters for pansharpening an image from storage

We are now ready to launch and run the job.

Downloading and visualizing the results

After running the job successfully, we can download the results and visualize them in QGIS.

QGIS visualization of the pansherpened image
QGIS visualization of the pansherpened image

Bear in mind that the band order of the delivered image is such that the Red and Blue bands are switched so you need to adjust the band order in QGIS. The correct (RGB) band order is 3, 2 and 1. This applies to the Pléaides Reflectance products.

The RGB color interpolation is defined on the DIMAP file, which is a Airbus and CNES defined XML based metadata exchange format.

This file contains a multitude of metadata that allows for further processing of the image, e.g., computing the Bottom-Of-Atmosphere (BOA) product from the delivered Top-Of-Atmosphere (TOA) product. However, delving further into this topic is beyond the scope of this tutorial.