Block Manifest

Introduction

The block manifest defines the block metadata and it is used to update the block information every time a new block version is published.

A manifest file looks like the example below:

{
  "_up42_specification_version": 2,
  "name": "pansharpen",
  "type": "processing",
  "tags": ["imagery", "processing", "preprocessing"],
  "display_name": "Pan-sharpening SPOT/Pléiades",
  "description": "Pansharpens images from Pléiades or SPOT.",
  "parameters": {
    "method": {
      "type": "string",
      "default": "SFIM"
    },
    "include_pan": {
      "type": "boolean",
      "default": false
    }
  },
  "machine": {
    "type": "large"
  },
  "input_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "DIMAP",
        "sensor": {
          "or": ["Pleiades", "SPOT"]
        },
        "bands": ["red", "green", "blue", "nir", "pan"]
      }
    }
  },
  "output_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "GTiff",
        "bands": {
          "or": [
            ["red", "green", "blue", "nir"],
            ["red", "green", "blue", "nir", "pan"]
          ]
        },
        "sensor": ">",
        "resolution": ">",
        "processing_level": ">",
        "dtype": ">"
      }
    }
  }
}

Manifest File

The manifest parameters are described in the table below.

Manifest Parameters	Description
`_up42_specification_version`	Currently, it should always be set to 2.
`name`	The name of your block, which must be unique for your account.
`type`	Block type (`"data"` or `"processing"`). This information is crucial when validating workflows. The major difference between data and processing blocks is whether they require input data (data blocks only generate output data and do not require input data).
`tags`	A list of tags used for searching and filtering blocks in the UP42 Platform.
`display_name`	The human-readable name of your block, as displayed on the UP42 Platform. This name is not checked for uniqueness.
`description`	The human-readable explanation about your block.
`parameters`	For data blocks, the list of all JSON parameters the block supports. For processing blocks, the run-time parameters that your block can optionally specify. For more information: JSON Parameters
`machine`	Machine type (specified by `type`), which defines the computing resource on which the block will be running. You can choose from a list of predefined machine types (see table below).
`input_capabilities`	The block capabilities that your block requires to run correctly. The blocks connect with each other if their input and output capabilities match. For example, a block that outputs a GeoTIFF is compatible with the next block that accepts GeoTIFF as input. This field may be left empty, although it is not recommended.
`output_capabilities`	The block capabilities that your block requires to run correctly. The blocks connect with each other if their input and output capabilities match. For example, a block that outputs a GeoTIFF is compatible with the next block that accepts GeoTIFF as input. This field may be left empty, although it is not recommended.

Machine Types	Size
`small`	(0.5 CPU + 2GB RAM)
`medium`	(1 CPU + 5GB RAM)
`large`	(2 CPU + 10GB RAM)
`xlarge`	(4 CPU + 20GB RAM)
`xxlarge`	(8 CPU + 40GB RAM)
`xxxlarge`	(16 CPU + 80GB RAM)
`gpu_nvidia_tesla_k80`	(4 CPU + 1 GPU + 20GB RAM)
`gpu_p100_xxl`	(8 CPU + 1 GPU + 40GB RAM)

For more information about the typical loads for each machine type, please visit the article Credits.

The manifest file also supports optional features, which have the role of additional functionalities that a block can support. Currently, optional features are only available for data blocks. The example below shows the manifest file used for our built-in Landsat-8 data block.

{
  "_up42_specification_version": 2,
  "name": "pleiades-scene",
  "type": "data",
  "tags": ["Airbus", "OneAtlas", "Pléiades", "optical", "global", "high revisit", "very-high resolution"],
  "display_name": "Pléiades Download",
  "description": "Pléiades imagery in its original DIMAP format. RGB, NIR (2 m) and panchromatic (0.5 m) bands with 12 bits.",
  "parameters": {
    "order_ids": {
      "type": "array",
      "default": null
    },
    "ids": {
      "type": "array",
      "default": null
    },
    "bbox": {
      "type": "array",
      "default": null
    },
    "intersects": {
      "type": "geometry"
    },
    "contains": {
      "type": "geometry"
    },
    "time": {
      "type": "dateRange",
      "default": "2018-01-01T00:00:00+00:00/2020-12-31T23:59:59+00:00"
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 500,
      "default": 1
    },
    "time_series": {
      "type": "array",
      "default": null
    },
    "max_cloud_cover": {
      "type": "integer",
      "minimum": 0,
      "maximum": 100,
      "default": 100
    }
  },
  "machine": {
    "type": "small"
  },
  "optional_features": {
    "quicklook_supported": true,
    "dry_run_supported": true
  },
  "input_capabilities": {},
  "output_capabilities": {
    "raster": {
      "up42_standard": {
        "format": "DIMAP",
        "bands": ["red", "green", "blue", "nir", "pan"],
        "resolution": 0.5,
        "sensor": "Pleiades",
        "dtype": "uint16",
        "processing_level": "l2"
      }
    }
  }
}

The optional_features parameter is displayed. In this example, quicklook_supported indicates that the data block supports the creation of image quicklooks. For more information: Data Transfer Format.

If dry_run_supported is set to true, the data block is able to perform test queries and fetch information about the availability of datasets according to the specified JSON parameters and without actually fetching the data. For more information: Environment Variables.

Adding the Manifest

The data in the manifest file needs to be attached as a LABEL to the docker image. This enables UP42 to read the manifest file quickly, without having to repeatedly fetch entire images.

The easiest way to do this is as follows:

Keep your manifest as a separate file, so that it is easier to maintain. By convention, we name the file UP42Manifest.json.
In your Dockerfile, add the following lines below the FROM directive that sets the base image:

    ARG manifest
    LABEL "up42_manifest"=$manifest

When building the image, specify the contents of the manifest:

    $ docker build . -t <image-tags> --build-arg manifest="$(cat UP42Manifest.json)"

You can then push the block with docker push. For more information: Developing a Custom Processing Block.

Validating the Manifest

Manifest files can be validated against the endpoint provided by the UP42 platform. In the examples below, we are using cURL.

$ curl -X POST -H 'Content-Type: application/json' -d @UP42Manifest.json https://api.up42.com/validate-schema/block

Example of a valid response:

Status code: 200
Body:
{
    "error": null,
    "data": {
        "valid": true,
        "errors": []
    }
}

Example of an invalid response:

Status code: 400
Body:
{
    "error": null,
    "data": {
    "valid": false,
    "errors": [
        "#: required key [name] not found"
    ]
    }
}

It is strongly recommended that you always validate the block manifest before pushing it to the UP42 registry.

References

To learn more, you can visit the full block manifest JSON schema.