Data transfer between blocks in a running workflow is handled by the workflow orchestrator. This means that you do not need to handle the storing and retrieving of data for the next block in the workflow yourself.
For every block that runs, any input data will be placed on the filesystem in the /tmp/input
directory.
Any block you write should save its output in the /tmp/output
directory.
The structure inside these directories should always be as follows:
A single
data.json
file, containing the GeoJSON metadata.Any number of additional resources/files, which are linked to from the GeoJSON metadata.
The output of the last block in the workflow will be treated as the result of any job run based on that workflow.
data.json
The metadata in the data.json
file should be a GeoJSON FeatureCollection:
{
"type": "FeatureCollection",
"features": [
// ...
]
}
The "features"
list can contain zero or more GeoJSON Feature
objects, which look like the following:
{
"type": "Feature",
"properties": {
// ...
},
"geometry": {
// ...
}
}
where "geometry"
is a GeoJSON geometry, and "properties"
is an arbitrary key/value mapping. A Feature
can have an id
. This is usally optional, but a requirement for data blocks that provide quicklook images. If the id
is set, then it has to be used as the stem of output file names.
up42.data_path property
Any feature in the data.json
should contain a property up42.data_path
with a relative file path pointing to the respective file or output directory in the /tmp/output
folder. This is the case for output imagery as well as vector data.
Example:
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
// ...
},
"properties": {
"up42.data_path": "aedf023/block_output_vectors.geojson"
// ...
}
},
{
"type": "Feature",
"geometry": {
// ...
},
"properties": {
"up42.data_path": "be051fa1/block_output_vectors.geojson"
// ...
}
}
]
}
The corresponding file layout of the block’s output files would be:
/tmp/output/data.json
/tmp/output/aedf0123/block_output_vectors.geojson
/tmp/output/be051fa1/block_output_vectors.geojson
Quicklooks
To give costumers an idea how a dataset looks like before the actual data is fetched, it is possible to supply quicklook images in RGB or grayscale. The images need to be saved in a separate folder named /tmp/quicklooks
and should use the Feature
id
as file name as the actual dataset (+ the corresponding file extension), e.g., if the output GeoJSON looks like the following:
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": "e18542c4-d3b6-4e74-9eb6-8899ad4276be",
"geometry": { ... },
"properties": {
"up42.data_path": "e18542c4-d3b6-4e74-9eb6-8899ad4276be.tif"
}
}
]
}
The quicklook image is stored as /tmp/quicklooks/e18542c4-d3b6-4e74-9eb6-8899ad4276be.jpg
or /tmp/quicklooks/e18542c4-d3b6-4e74-9eb6-8899ad4276be.png
. The image should be of format PNG or JPEG and of size 512 × 512 pixels and will be displayed both in DRY_RUN
mode as well as during actual fetching of image data.
Common data types: scenes and mosaics
Most data blocks will be in one of two formats:
A full-scene block
A mosaic of WMTS tiles, clipped as closely as possible to the specified AOI
For more information: Blocks.
Tools and references
- GeoJSON deep dive.
- GeoJSON schemas:
- Geometry.
- FeatureCollection.
- Feature.
- geojson.io for testing and visualizing GeoJSON objects.