/
Open Console
Home/Processing Platform/Developing Blocks

Data Transfer Format

Transferring data between blocks.

Data transfer between blocks in a running workflow is handled by the workflow orchestrator. This means that you do not need to handle the storing and retrieving of data for the next block in the workflow yourself.

For every block that runs, any input data will be placed on the filesystem in the /tmp/input directory.

Any block you write should save its output in the /tmp/output directory.

The structure inside these directories should always be as follows:

  • A single data.json file, containing the GeoJSON metadata.

  • Any number of additional resources/files, which are linked to from the GeoJSON metadata.

The output of the last block in the workflow will be treated as the result of any job run based on that workflow.

data.json

The metadata in the data.json file should be a GeoJSON FeatureCollection:

{
    "type": "FeatureCollection",
    "features": [
        // ...
    ]
}

The "features" list can contain zero or more GeoJSON Feature objects, which look like the following:

{
    "type": "Feature",
    "properties": {
        // ...
    },
    "geometry": {
        // ...
    }
}

where "geometry" is a GeoJSON geometry, and "properties" is an arbitrary key/value mapping. A Feature can have an id. This is usally optional, but a requirement for data blocks that provide quicklook images. If the id is set, then it has to be used as the stem of output file names.

up42.data_path property

Any feature in the data.json should contain a property up42.data_path with a relative file path pointing to the respective file or output directory in the /tmp/output folder. This is the case for output imagery as well as vector data.

Example:

{
    "type": "FeatureCollection",
    "features": [
        {
            "type": "Feature",
            "geometry": {
                // ...
            },
            "properties": {
                "up42.data_path": "aedf023/block_output_vectors.geojson",
                // ...
            }
        },
        {
            "type": "Feature",
            "geometry": {
                // ...
            },
            "properties": {
                "up42.data_path": "be051fa1/block_output_vectors.geojson",
                // ...
            }
        }
    ]
}

The corresponding file layout of the block’s output files would be:

/tmp/output/data.json
/tmp/output/aedf0123/block_output_vectors.geojson
/tmp/output/be051fa1/block_output_vectors.geojson

Quicklooks

To give costumers an idea how a dataset looks like before the actual data is fetched, it is possible to supply quicklook images in RGB or grayscale. The images need to be saved in a separate folder named /tmp/quicklooks and should use the Feature id as file name as the actual dataset (+ the corresponding file extension), e.g., if the output GeoJSON looks like the following:

{
    "type": "FeatureCollection",
    "features": [
        {
            "type": "Feature",
            "id": "e18542c4-d3b6-4e74-9eb6-8899ad4276be",
            "geometry": { ... },
            "properties": {
                "up42.data_path": "e18542c4-d3b6-4e74-9eb6-8899ad4276be.tif"
            }
        }
    ]
}

The quicklook image is stored as /tmp/quicklooks/e18542c4-d3b6-4e74-9eb6-8899ad4276be.jpg or /tmp/quicklooks/e18542c4-d3b6-4e74-9eb6-8899ad4276be.png. The image should be of format PNG or JPEG and of size 512 × 512 pixels and will be displayed both in DRY_RUN mode as well as during actual fetching of image data.

Common data types: scenes and mosaics

Most data blocks will be in one of two formats:

  • A full-scene block

  • A mosaic of WMTS tiles, clipped as closely as possible to the specified AOI

For more information: Blocks.

Tools and references