/

Process datasets from storage

How to further process geospatial datasets delivered to storage.


Introduction

In this article, you will learn how to further process geospatial datasets that you previously purchased either through the data platform (collections, catalog or tasking orders) or the processing platform (data blocks) and are delivered to storage.

Currently, UP42 provides 2 methods for processing datasets from storage: the data block Processing from Storage and the OneAtlas data blocks.

The difference between the 2 methods is explained in the table below.

ActionsProcessing from StorageOneAtlas Data Blocks
Process previously ordered datasets (collections, catalog)
Process previously ordered tasking datasets
Process datasets previously retrieved through data blocks
Requirement

This feature is intended for advanced users and it requires basic familiarity with JavaScript Object Notation.

Asset identifiers

An asset is a unique item generated from a successful job run using a OneAtlas data block, a completed order or a fresh image captured through sensor tasking and it contains the downloadable geospatial dataset.

Every asset gets a unique asset identifier (ID). The newly-generated asset ID is displayed in your storage, where each asset can be downloaded.

These asset IDs can be used to further process your previously delivered datasets on the UP42 platform.

Processing from Storage

Requirements

The objective of this method is to further process geospatial datasets that have been delivered to storage through job runs of workflows or ordering (collections, catalog or tasking orders).

Steps

Please follow the steps below:

  1. Go to Storage.
  2. In the dashboard, locate the asset name, then scroll to the right.
  3. Click on the first grey circle and click Image Information.

Copy the asset ID from storage
Copy the asset ID from storage

  1. Scroll to the right and copy the asset ID under Asset Identifier.

View the image information and copy the asset identifier
View the image information and copy the asset identifier

  1. Create a new workflow and add the block Processing from Storage. For more information about building workflows, please refer to Create a workflow.

Create a workflow
Create a workflow

  1. In the job configuration window, find the parameter asset_ids and paste the asset ID in an array. For more information about jobs, please refer to Configure job.
  2. You don’t need to adjust the JSON parameters of the processing block that follows the data block.
  3. Click Run Job.
  4. Done.

Example

A workflow that contains the data block Processing from Storage, the processing block Pansharpening and multiple asset IDs:

Loading...
Useful tip

Please note that you can add more asset IDs inside an array, where each item is separated by a comma. Example: ["3475e76d-4e04-420e-a683-bc4831df824f", "d266389c-798c-42b5-a164-721caed9b0c5"].

Please note that the capabilities between this block and the following processing blocks cannot be verified. We recommend exploring the capabilities of the corresponding dataset. For instance, the SPOT 6/7 Display or SPOT 6/7 Reflectance product will have different compatible processing blocks.

OneAtlas data blocks

Requirements

This section assumes that you already created a workflow that contains a OneAtlas data block and ran a job of the workflow. For more information, please refer to Create a workflow.

If you run a job of this workflow for the first time, a new dataset is retrieved and an asset ID is generated.

Currently, the OneAtlas data blocks that generate asset IDs are shown in the table below.

To process the datasets that were previously retrieved using OneAtlas data blocks, you need to use the same OneAtlas data block that was used the first time. For example, if you previously ran a job of a workflow that contains a Pléiades Reflectance (Download) data block, the newly-generated asset ID can only be used with the same Pléiades Reflectance (Download) data block.

Steps

Please follow the steps below:

  1. Go to Storage.
  2. In the dashboard, locate the asset name, then scroll to the right.
  3. Click on the rightmost grey circle to copy the asset ID. Please note that only the assets with metadata can be processed with OneAtlas data blocks. Assets with metadata are indicated by this rightmost grey circle.

Copy the asset ID from storage
Copy the asset ID from storage

  1. Go to your previously-created workflow and click on Update & Configure Job.

Update a workflow
Update a workflow

  1. In the job configuration window, find the parameter asset_ids and paste the asset ID in an array. For more information about jobs, please refer to Configure job.
  2. You don’t need to adjust the JSON parameters of the processing block that follows the data block.
  3. Click Run Job.
  4. Done.

Example

A workflow that contains the data block Pleiades Display (Download) and the processing block DIMAP→ GeoTIFF Conversion:

Loading...
Good to know

This functionality relies on the presence of the file data.json, which contains the asset metadata. This file is generated during data retrieval through data blocks.

Useful tip

Please note that you can also add more asset IDs in the form of an array, where each item is separated by a comma. Example: ["3475e76d-4e04-420e-a683-bc4831df824f", "d266389c-798c-42b5-a164-721caed9b0c5"].