Overview
The block manifest defines block metadata and is used to update the block information every time a new version is published.
Step 2. Choose a machine type
Step 3. Specify additional parameters
Step 4. Set input and output capabilities
Step 5. Validate your manifest file
Step 6. Add your manifest to the Dockerfile
Step 1. Create a manifest file
Create a JSON file named UP42Manifest.json
in the custom block folder and populate it with manifest parameters. For more information, see manifest schema.
{
"_up42_specification_version": 2,
"name": "processing-block-1",
"type": "processing",
"display_name": "Processing block template",
"description": "This is a template processing block.",
"machine": {
"type": "small"
},
"parameters": {},
"input_capabilities": {},
"output_capabilities": {}
}
Parameter | Overview |
---|---|
_up42_specification_version | integer / required The version of the specification. The value should be 2 . |
name | string / required The name of the block. Must be unique for your account. |
type | string / required The block type. The value should be "processing" . |
tags | array of strings A list of tags used for searching and filtering blocks in the UP42 platform. |
display_name | string / required The human-readable name of your block that will be displayed on the UP42 platform. This name isn't checked for uniqueness. |
description | string / required The human-readable description of your block that will be displayed on the UP42 platform. |
machine | object / required The computing resource on which the block will be running. Learn more about machine types. |
parameters | object / required Additional parameters of the block. Can be left empty. Learn more about additional parameters. |
input_capabilities | object / required The input capabilities of the block. The blocks connect if the output of the previous block matches the input schema of the next block. Can be left empty. Learn more about capabilities. |
output_capabilities | object / required The output capabilities of the block. The blocks connect if the output of the previous block matches the input schema of the next block. Can be left empty. Learn more about capabilities. |
Step 2. Choose a machine type
Machine type | CPU | GPU | RAM |
---|---|---|---|
small | 0.5 | None | 2 GB |
medium | 1 | None | 5 GB |
large | 2 | None | 10 GB |
xlarge | 4 | None | 20 GB |
xxlarge | 8 | None | 40 GB |
xxxlarge | 16 | None | 80 GB |
gpu_nvidia_tesla_k80 | 4 | 1 | 20 GB |
gpu_p100_xxl | 8 | 1 | 40 GB |
An example of setting NVIDIA Tesla K80 GPU as a machine type:
{
"machine": {
"type": "gpu_nvidia_tesla_k80"
}
}
Different use cases require different machine types:
small
- Use with a data block that downloads images.
- Use with a processing block with a very basic algorithm.
medium
- Use with a processing block with a basic algorithm. For example, it's used with NDVI.
large
- Use with multi-band imagery analysis that can't be parallelized with raster tiling.
xlarge
- Use with advanced algorithms. For example, it's used with K-means clustering.
gpu_nvidia_tesla_k80
- Use with GPU-intensive ML algorithms. For example, it's used with Super-resolution Sentinel-2.
The chosen machine type affects the infrastructure costs for running a job with this block. For more information on infrastructure costs, see Pricing.
Step 3. Specify additional parameters
You can specify a schema with additional parameters that modify block behavior.
There are no restrictions on the names of the parameters, except for the reserved parameters that should be of a specific type:
Parameter name | Type |
---|---|
intersects | geometry |
contains | geometry |
bbox | array of floats |
ids | array of strings |
limit | integer |
An example of parameters:
{
"parameters": {
"bbox": { "type": "array", "default": null },
"intersects": { "type": "geometry", "default": null },
"contains": { "type": "geometry", "default": null },
"polarisations": {
"type": "array",
"required": false,
"description": "Requested polarisations for the output",
"default": ["VV"],
"items": {
"type": "string",
"enum": ["VV", "VH"]
}
},
"calibration_band": { "type": "array", "default": ["sigma"] },
"speckle_filter": { "type": "boolean", "default": true },
"mask": { "type": "array", "default": null, "items": { "type": "string", "enum": ["land", "sea"] } },
"tcorrection": { "type": "boolean", "default": true },
"clip_to_aoi": { "type": "boolean", "default": false },
"linear_to_db": { "type": "boolean", "default": true }
}
}
An example of how to use the example parameters in a job configuration:
{
"snap-polarimetric:1": {
"bbox": [13.35, 52.48, 13.39, 52.5],
"mask": null,
"contains": null,
"intersects": null,
"clip_to_aoi": true,
"tcorrection": true,
"linear_to_db": true,
"polarisations": ["VV"],
"speckle_filter": true,
"calibration_band": ["sigma"]
}
}
Step 4. Set input and output capabilities
Combinations of blocks are limited, which means that a block can only be connected to certain other blocks. Block capabilities dictate the specific input and output types and parameters of the block. For example, a block that outputs a GeoTIFF is compatible with the next block that accepts GeoTIFF as input.
- Input capabilities define what data types are accepted as inputs.
- Output capabilities define what kind of data types are generated by a block. Can be left empty for the last block of a workflow.
Operators
Operator | Overview |
---|---|
> | The propagation operator. It forwards output capabilities of the previous block to the input capabilities of the next block. Use only in output capabilities. |
or | The boolean OR operator. Given an array of values for a parameter, it has to match at least one of the values. |
${parameter} | Inserts the values of parameters from the job configuration into the capabilities. |
See capabilities of a block that uses the propagation and boolean operators
{
"input_capabilities": {
"raster": {
"up42_standard": {
"format": "DIMAP",
"sensor": {
"or": ["Pleiades", "SPOT"] // Uses the OR operator
},
"bands": ["red", "green", "blue", "nir", "pan"]
}
}
},
"output_capabilities": {
"raster": {
"up42_standard": {
"format": "GTiff",
"bands": {
"or": [
["red", "green", "blue", "nir"],
["red", "green", "blue", "nir", "pan"]
]
},
"sensor": ">", // Forwards data to the next block
"resolution": ">",
"processing_level": ">",
"dtype": ">"
}
}
}
}
See capabilities of a block that uses the propagation operator and inserts job configuration values into the capabilities
{
"input_capabilities": {
"raster": {
"up42_standard": {
"format": "GTiff"
}
}
},
"output_capabilities": {
"raster": {
"up42_standard": {
"format": "GTiff",
"bands": ">",
"resolution": ">",
"sensor": ">",
"dtype": ">",
"processing_level": ">",
"tile_width": "${tile_width}", // Inserts values from job parameters
"tile_height": "${tile_height}"
},
"custom": {
"match_extents": "${match_extents}"
}
}
}
}
Data types
Data types precede other parameters in a capability object and correspond to the types of data that can be created or consumed inside a workflow.
Data type | Overview |
---|---|
raster | A file type that represents geospatial data in the form of pixels or grid cells. Geospatial images that are captured by satellites and aircraft are raster. See raster type specification. |
vector | A file type that represents geospatial data in the form of points, lines, and polygons with added information attributes. An example of vector data is a dataset of line geometries that represent streets, with street names as attributes for each line. See vector type specification. |
misc | A miscellaneous file type: CSV, XML, or JSON. See misc type specification. |
A data type object needs to contain at least one of the following objects: up42_standard
or custom
.
up42_standard
A capability that conforms to the UP42 standard.
For example:
{
"input_capabilities": {
"raster": {
"up42_standard": {
"format": "GTiff",
"dtype": "uint8"
}
}
}
}
See standard nested parameters of raster, vector, and misc objects.
custom
A custom capability that isn't in the UP42 standard.
For example:
{
"output_capabilities": {
"vector": {
"custom": {
"object_type": "ships"
}
}
}
}
See capabilities of a block that returns a standard and a custom object as the output
{
"input_capabilities": {
"raster": {
"up42_standard": {
"format": "GTiff",
"dtype": "uint8",
"resolution": 1.5,
"tile_width": 768,
"tile_height": 768
}
}
},
"output_capabilities": {
"vector": {
"up42_standard": {
"format": "GeoJSON",
"geometry_type": "Polygon"
},
"custom": {
"object_type": "ships"
}
}
}
}
See capabilities of a block that returns a PNG as the output
{
"input_capabilities": {
"raster": {
"up42_standard": {
"format": "GTiff",
"dtype": "uint8"
}
}
},
"output_capabilities": {
"raster": {
"custom": {
"format": "PNG",
"depth": 8,
"has_alpha": true,
"dtype": "uint8"
}
}
}
}
Standard capabilities
Parameter | Overview |
---|---|
Capabilities of a raster object | |
format | string / required The file format. For allowed values, see raster types. |
dtype | string / required The data type according to the C99 language specification. For allowed values, see raster types. |
sensor | string The name of the sensor and product type. For allowed values, see raster types. |
resolution | unsigned integer or float The spatial resolution, in meters. For blocks providing multiple spectral bands, the value corresponds to the highest resolution possible. |
bands | array of strings
|
processing_level | string / required The processing level of the image product. For allowed values, see raster types. |
tile_width | integer The tile width in pixels.
|
tile_height | integer The tile height in pixels.
|
Capabilities of a vector object | |
format | string / required The file format. For allowed values, see vector types. |
geometry_type | string The GeoJSON vector type. For allowed values, see vector types. |
Capabilities of a misc object | |
format | string / required The file format. For allowed values, see misc types. |
Step 5. Validate your manifest file
To validate the block manifest, run the following code in your terminal from the block folder:
curl -X POST -H 'Content-Type: application/json' -d @UP42Manifest.json https://api.up42.com/validate-schema/block
If you get this response, the block manifest is valid:
{
"data": {
"valid": true,
"errors": []
},
"error": null
}
Otherwise, the endpoint will provide a detailed error message.
If you use an IDE, you can configure it to provide automatic hinting and validation by fetching the manifest schema.
Step 6. Add your manifest to the Dockerfile
Add the following lines below the FROM
directive that sets the base image:
ARG manifest
LABEL "up42_manifest"=$manifest