Process datasets from storage

Introduction

In this article, you will learn how to further process geospatial datasets that you previously purchased either through the data platform (collections, catalog or tasking orders) or the processing platform (data blocks) and are delivered to storage.

Currently, UP42 provides 2 methods for processing datasets from storage: the data block Processing from Storage and the OneAtlas data blocks.

The difference between the 2 methods is explained in the table below.

Asset identifiers

An asset is a unique item generated from a successful job run using a OneAtlas data block, a completed order or a fresh image captured through sensor tasking and it contains the downloadable geospatial dataset.

Every asset gets a unique asset identifier (ID). The newly-generated asset ID is displayed in your storage, where each asset can be downloaded.

These asset IDs can be used to further process your previously delivered datasets on the UP42 platform.

Processing from Storage

Requirements

The objective of this method is to further process geospatial datasets that have been delivered to storage through job runs of workflows or ordering (collections, catalog or tasking orders).

Steps

Please follow the steps below:

1. Go to Storage.
2. In the dashboard, locate the asset name, then scroll to the right.
3. Click on the first grey circle and click Image Information.

4. Scroll to the right and copy the asset ID under Asset Identifier.

5. Create a new workflow and add the block Processing from Storage. For more information about building workflows, please refer to Create a workflow.

6. In the job configuration window, find the parameter asset_ids and paste the asset ID in an array. For more information about jobs, please refer to Configure job.
7. You don’t need to adjust the JSON parameters of the processing block that follows the data block.
8. Click Run Job.
9. Done.

Example

A workflow that contains the data block Processing from Storage, the processing block Pansharpening and multiple asset IDs:

OneAtlas data blocks

Requirements

This section assumes that you already created a workflow that contains a OneAtlas data block and ran a job of the workflow. For more information, please refer to Create a workflow.

If you run a job of this workflow for the first time, a new dataset is retrieved and an asset ID is generated.

Currently, the OneAtlas data blocks that generate asset IDs are shown in the table below.

To process the datasets that were previously retrieved using OneAtlas data blocks, you need to use the same OneAtlas data block that was used the first time. For example, if you previously ran a job of a workflow that contains a Pléiades Reflectance (Download) data block, the newly-generated asset ID can only be used with the same Pléiades Reflectance (Download) data block.

Steps

Please follow the steps below:

1. Go to Storage.
2. In the dashboard, locate the asset name, then scroll to the right.
3. Click on the rightmost grey circle to copy the asset ID. Please note that only the assets with metadata can be processed with OneAtlas data blocks. Assets with metadata are indicated by this rightmost grey circle.

4. Go to your previously-created workflow and click on Update & Configure Job.

5. In the job configuration window, find the parameter asset_ids and paste the asset ID in an array. For more information about jobs, please refer to Configure job.
6. You don’t need to adjust the JSON parameters of the processing block that follows the data block.
7. Click Run Job.
8. Done.

Example

A workflow that contains the data block Pleiades Display (Download) and the processing block DIMAP→ GeoTIFF Conversion: