Block manifest

Create a JSON file that describes custom block metadata.


Overview

The block manifest defines block metadata and is used to update the block information every time a new version is published.

Step 1. Create a manifest file

Step 2. Choose a machine type

Step 3. Specify additional parameters

Step 4. Set input and output capabilities

Step 5. Validate your manifest file

Step 6. Add your manifest to the Dockerfile

Step 1. Create a manifest file

Create a JSON file named UP42Manifest.json in the custom block folder and populate it with manifest parameters. For more information, see manifest schema.

{
  "_up42_specification_version": 2,
  "name": "processing-block-1",
  "type": "processing",
  "display_name": "Processing block template",
  "description": "This is a template processing block.",
  "machine": {
    "type": "small"
  },
  "parameters": {},
  "input_capabilities": {},
  "output_capabilities": {}
}
ParameterOverview
_up42_specification_versioninteger / required
The version of the specification. The value should be 2.
namestring / required
The name of the block. Must be unique for your account.
typestring / required
The block type. The value should be "processing".
tagsarray of strings
A list of tags used for searching and filtering blocks in the UP42 platform.
display_namestring / required
The human-readable name of your block that will be displayed on the UP42 platform. This name isn't checked for uniqueness.
descriptionstring / required
The human-readable description of your block that will be displayed on the UP42 platform.
machineobject / required
The computing resource on which the block will be running. Learn more about machine types.
parametersobject / required
Additional parameters of the block. Can be left empty. Learn more about additional parameters.
input_capabilitiesobject / required
The input capabilities of the block. The blocks connect if the output of the previous block matches the input schema of the next block. Can be left empty. Learn more about capabilities.
output_capabilitiesobject / required
The output capabilities of the block. The blocks connect if the output of the previous block matches the input schema of the next block. Can be left empty. Learn more about capabilities.

Step 2. Choose a machine type

Machine typeCPUGPURAM
small0.5None2 GB
medium1None5 GB
large2None10 GB
xlarge4None20 GB
xxlarge8None40 GB
xxxlarge16None80 GB
gpu_nvidia_tesla_k804120 GB
gpu_p100_xxl8140 GB

An example of setting NVIDIA Tesla K80 GPU as a machine type:

{
  "machine": {
    "type": "gpu_nvidia_tesla_k80"
  }
}

Different use cases require different machine types:

  • small
    • Use with a data block that downloads images.
    • Use with a processing block with a very basic algorithm.
  • medium
    • Use with a processing block with a basic algorithm. For example, it's used with NDVI.
  • large
    • Use with multi-band imagery analysis that can't be parallelized with raster tiling.
  • xlarge
  • gpu_nvidia_tesla_k80

The chosen machine type affects the infrastructure costs for running a job with this block. For more information on infrastructure costs, see Pricing.

Step 3. Specify additional parameters

You can specify a schema with additional parameters that modify block behavior.

There are no restrictions on the names of the parameters, except for the reserved parameters that should be of a specific type:

Parameter nameType
intersectsgeometry
containsgeometry
bboxarray of floats
idsarray of strings
limitinteger

An example of parameters:

{
  "parameters": {
    "bbox": { "type": "array", "default": null },
    "intersects": { "type": "geometry", "default": null },
    "contains": { "type": "geometry", "default": null },
    "polarisations": {
      "type": "array",
      "required": false,
      "description": "Requested polarisations for the output",
      "default": ["VV"],
      "items": {
        "type": "string",
        "enum": ["VV", "VH"]
      }
    },
    "calibration_band": { "type": "array", "default": ["sigma"] },
    "speckle_filter": { "type": "boolean", "default": true },
    "mask": { "type": "array", "default": null, "items": { "type": "string", "enum": ["land", "sea"] } },
    "tcorrection": { "type": "boolean", "default": true },
    "clip_to_aoi": { "type": "boolean", "default": false },
    "linear_to_db": { "type": "boolean", "default": true }
  }
}

An example of how to use the example parameters in a job configuration:

{
  "snap-polarimetric:1": {
    "bbox": [13.35, 52.48, 13.39, 52.5],
    "mask": null,
    "contains": null,
    "intersects": null,
    "clip_to_aoi": true,
    "tcorrection": true,
    "linear_to_db": true,
    "polarisations": ["VV"],
    "speckle_filter": true,
    "calibration_band": ["sigma"]
  }
}

Step 4. Set input and output capabilities

Combinations of blocks are limited, which means that a block can only be connected to certain other blocks. Block capabilities dictate the specific input and output types and parameters of the block. For example, a block that outputs a GeoTIFF is compatible with the next block that accepts GeoTIFF as input.

  • Input capabilities define what data types are accepted as inputs.
  • Output capabilities define what kind of data types are generated by a block. Can be left empty for the last block of a workflow.

Operators

OperatorOverview
>The propagation operator. It forwards output capabilities of the previous block to the input capabilities of the next block. Use only in output capabilities.
orThe boolean OR operator. Given an array of values for a parameter, it has to match at least one of the values.
${parameter}Inserts the values of parameters from the job configuration into the capabilities.
See capabilities of a block that uses the propagation and boolean operators
{
  "input_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "DIMAP",
        "sensor": {
          "or": ["Pleiades", "SPOT"] // Uses the OR operator
        },
        "bands": ["red", "green", "blue", "nir", "pan"]
      }
    }
  },
  "output_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "GTiff",
        "bands": {
          "or": [
            ["red", "green", "blue", "nir"],
            ["red", "green", "blue", "nir", "pan"]
          ]
        },
        "sensor": ">", // Forwards data to the next block
        "resolution": ">",
        "processing_level": ">",
        "dtype": ">"
      }
    }
  }
}

See capabilities of a block that uses the propagation operator and inserts job configuration values into the capabilities
{
  "input_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "GTiff"
      }
    }
  },
  "output_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "GTiff",
        "bands": ">",
        "resolution": ">",
        "sensor": ">",
        "dtype": ">",
        "processing_level": ">",
        "tile_width": "${tile_width}", // Inserts values from job parameters
        "tile_height": "${tile_height}"
      },
      "custom": {
        "match_extents": "${match_extents}"
      }
    }
  }
}

Data types

Data types precede other parameters in a capability object and correspond to the types of data that can be created or consumed inside a workflow.

Data typeOverview
rasterA file type that represents geospatial data in the form of pixels or grid cells. Geospatial images that are captured by satellites and aircraft are raster.

See raster type specification.
vectorA file type that represents geospatial data in the form of points, lines, and polygons with added information attributes. An example of vector data is a dataset of line geometries that represent streets, with street names as attributes for each line.

See vector type specification.
miscA miscellaneous file type: CSV, XML, or JSON.

See misc type specification.

A data type object needs to contain at least one of the following objects: up42_standard or custom.

up42_standard

A capability that conforms to the UP42 standard.

For example:

{
  "input_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "GTiff",
        "dtype": "uint8"
      }
    }
  }
}

See standard nested parameters of raster, vector, and misc objects.

custom

A custom capability that isn't in the UP42 standard.

For example:

{
  "output_capabilities": {
    "vector": {
      "custom": {
        "object_type": "ships"
      }
    }
  }
}
See capabilities of a block that returns a standard and a custom object as the output
{
  "input_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "GTiff",
        "dtype": "uint8",
        "resolution": 1.5,
        "tile_width": 768,
        "tile_height": 768
      }
    }
  },
  "output_capabilities": {
    "vector": {
      "up42_standard": {
        "format": "GeoJSON",
        "geometry_type": "Polygon"
      },
      "custom": {
        "object_type": "ships"
      }
    }
  }
}

See capabilities of a block that returns a PNG as the output
{
  "input_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "GTiff",
        "dtype": "uint8"
      }
    }
  },
  "output_capabilities": {
    "raster": {
      "custom": {
        "format": "PNG",
        "depth": 8,
        "has_alpha": true,
        "dtype": "uint8"
      }
    }
  }
}

Standard capabilities

ParameterOverview
Capabilities of a raster object
formatstring / required
The file format. For allowed values, see raster types.
dtypestring / required
The data type according to the C99 language specification. For allowed values, see raster types.
sensorstring
The name of the sensor and product type. For allowed values, see raster types.
resolutionunsigned integer or float
The spatial resolution, in meters. For blocks providing multiple spectral bands, the value corresponds to the highest resolution possible.
bandsarray of strings
  • For optical images: Bands provided by the block.
  • For radar images: Types of polarization provided by the block.
For allowed values, see raster types.
processing_levelstring / required
The processing level of the image product. For allowed values, see raster types.
tile_widthinteger
The tile width in pixels.
  • For blocks that require tiling: Should be specified in input_capabilities
  • For blocks that provide tiling: Should be specified in output_capabilities
tile_heightinteger
The tile height in pixels.
  • For blocks that require tiling: Should be specified in input_capabilities
  • For blocks that provide tiling: Should be specified in output_capabilities
Capabilities of a vector object
formatstring / required
The file format. For allowed values, see vector types.
geometry_typestring
The GeoJSON vector type. For allowed values, see vector types.
Capabilities of a misc object
formatstring / required
The file format. For allowed values, see misc types.

Step 5. Validate your manifest file

To validate the block manifest, run the following code in your terminal from the block folder:

curl -X POST -H 'Content-Type: application/json' -d @UP42Manifest.json https://api.up42.com/validate-schema/block

If you get this response, the block manifest is valid:

{
  "data": {
    "valid": true,
    "errors": []
  },
  "error": null
}

Otherwise, the endpoint will provide a detailed error message.

If you use an IDE, you can configure it to provide automatic hinting and validation by fetching the manifest schema.

Step 6. Add your manifest to the Dockerfile

Add the following lines below the FROM directive that sets the base image:

ARG manifest

LABEL "up42_manifest"=$manifest