/

Your First API Request

Learn how to use the API to create a workflow and run a job from A to Z


Getting started

The Application Programming Interface (API) is dedicated to users of the scripting interface and allows users to build custom applications. The API can be easily integrated into new products and either automate or scale the usage of the UP42 platform.

This tutorial is structured in four major chapters: Getting Started, Run your first job via the API, Run a job via an API script and Going Further.

The Getting Started chapter will help you set up your working environment in the Bash shell with the required tools to use the API without any problems. You will also learn how to get your project credentials and access token for API authentication, which may involve a quick visit to our user interface (UI) or the Console.

In the chapter entitled Run your first job via the API, you will learn how to set up your first workflow and to run a job consisting of the Pléiades Reflectance Download, Pan-sharpening, and NDVI blocks solely through the API. Of course, if you already want to try a different workflow, please feel free to adjust any step of the guide to your project needs.

The chapter Run a job via an API script delves into the the same procedure outlined in the previous chapter, just all packaged into a handy script that allows you to run a job through a simple command.

The final chapter of this guide, entitled Going Futher, will provide the next steps.

For a full underestanding of our API functionalities, we highly suggest you consult the API reference on the side.

Requirements

  1. An UP42 account
  2. cURL
  3. jq

Bash is the shell used in this guide.

You are free to use any other shell as long as you use a command line HTTP client like cURL and a JSON processor like jq. Additionally, you can use jwt-cli to decode the token that is based on JSON Web Token (JWT) and uses an HMAC based on SHA-512 for signature encryption. This is more of a useful utility than a must-have for following this walkthrough.

All outputs (response bodies) deemed too large to be shown here are given as Github gists linked below as they are returned by the API.

The shell is a rich interactive environment. Although there is a trend in data science to use interactive graphical tools, e.g., Jupyter, the shell remains a baseline that everyone can use in a portable way and does not require a graphical interface. An API is meant to be invoked in a machine to machine way, hence getting familiarized with the API through a shell is a much closer approximation to a deployment reality than through a graphical interface.

Project Variables

To perform any action through the API, you will need to first create a project in the UP42 Console. Each project has its own unique project ID and project key, which you can find in the project’s Developers page. The project ID and project key is crucial to the API because these unique strings of letters and numbers are what helps the API identify which project to be working on.

Copy the alphanumeric characters displayed in each field and paste them into the variables PROJ (Project ID) and PKEY (Project API Key). The project credentials are confidential and should not be shared with anyone.

See here how to obtain your project ID and project key.

# Set the project ID.
PROJ=5a21eaff-cdaa-48ab-bedf-5454116d16ff

# Set the project key.
PKEY=aoiTOv31.hab0M74qT9cB7K57wO6ue1glddcL3t5zsxx

Please be aware that the project ID and the project key allows anyone to manipulate your project (account) so be careful and do not share it around. Someone might find it, and besides messing with your project, will also make you incur costs and thus reduce the currently available credits in our platform.

Authentication: getting the access token

Using the project key and project ID, you must generate an access token. After successful user authentication, access tokens are provided by the server to the user to be used as credentials to call the API. These tokens essentially notify the API that the bearer of the token has been authorized to access its API functions and its protected resources (Note in the following cURL commands how the bearer token is used in the authorization request headers).

# Get the token.
PTOKEN=$(curl -sX POST "https://$PROJ:$PKEY@api.up42.com/oauth/token" -H 'Content-Type: application/x-www-form-urlencoded' -d 'grant_type=client_credentials' | jq -r '.data.accessToken')

Now you can echo the token in the shell:

echo $PTOKEN

eyJ0eXAiOiJKV1QiLCJraWQiOiIxIiwidG9rZW5fdHlwZSI6IkFDQ0VTUyIsImFsZyI6IlJTNTEyIn0.eyJpc3MiOiJiYWNrZW5kLWNvcmUiLCJqdGkiOiI5ZGYyMzY3MC02NDRkLTRkMGEtYTFlNi1hODIwN2QxZGQwNDgiLCJpYXQiOjE1NjE3MTc0ODcsInN1YiI6IjVhMjFlYWZmLWNkYWEtNDhhYi1iZWRmLTU0NTQxMTZkMTZmZiIsImF1ZCI6IjVhMjFlYWZmLWNkYWEtNDhhYi1iZWRmLTU0NTQxMTZkMTZmZiIsImV4cCI6MTU2MTcxNzc4NywiYXV0aG9yaXRpZXMiOlsiUlVOX0pPQiIsIlZJRVdfUFJPSkVDVCJdfQ.DLEUuifHzksf_Q_ReMF0aQXY-MOoy_nDu-noCGu7F8_Z2dBEJXbKILcvTB1t7ABVZmnd2eGlLiBuAF5zuz-L7nGuxqqzPawYy4GMB_ICc7HTuicYnx3fOGakby6qUGRuWlOmPGbcsgS_tRbt4pcjOPMvK0LbBXKobZb1HZYMdns4wiKVHE6IEyWn57k0eVm_y5fKImLIvGbqz060AakIamQ6O9uAHADOZwej9rnbkQO9e5LqP3hbb59sluyOhke0hYuJqA5VhssX743xxa3MZpxBRRhwR5YG_oxWEdOShhFq7T9S5i8fCZvhuoR3eQSkakTEfIMxLYQfDcycdptHJqXN5twtlYJ0hKTKuW0ezgELeTHtuSobg3xbZW7M8opX7lqtnnsVPVApo19ndqdaJtfTFiU1WgcveS0o47sXkPVtB7ohug420g5ux3XRCxgAY6vFHlvNWZZP6F6bSh-Ah7Gqm5jsW76DrloZyedOVz2qVoFU6XCicyXEsBSuo0giRlVHnVtRmqmHbTvyxFjndTbsoahxSH2rKX4H1AWjIyw_jEcZGBx4XZG2dWPYSNOR1SCx59i4XL9BzTVywjxNt50MpV92eIRI7doNSK-UXo6DClrXPl8-VskJrS_fTjyK-qD8P1tCHYs8eytnfKG0BZwrlhYAVYMHumvOtxxG0NE

This token is only valid for 5 minutes. If more than 5 minutes have elapsed, issue a new token by using the above command or simply enter the following:

!PTOKEN

Since Bash does not record in the shell history all the commands that start with a space, we recommend you set the PKEY variable above such that the line starts with a space like it's shown here.

Run your first job via the API

In this section of the API walkthrough, you will use the API to build a new workflow and run your first job from an existing project.

Create a workflow

Let’s populate the project with our very first workflow. Create the workflow by first creating a variable URL_WORKFLOWS that contains the necessary endpoint.

URL_WORKFLOWS="https://api.up42.com/projects/$PROJ/workflows"

Then issue a POST request and save the response in a file called workflow-created-response.json.

curl -s -L -X POST $URL_WORKFLOWS \
     -H "Authorization: Bearer $PTOKEN" \
     -H 'Content-Type: application/json' \
     -d @create_new_workflow.json | jq '.' > workflow-created-response.json

For convenience, save the workflow ID in its own variable called WORKFLOW. This way, we do not need to continually issue the same command (as shown below) for whenever we need to recall the workflow ID. The following command simply scours through the previously generated response file and extracts the field that contains the workflow id.

WORKFLOW=$(cat workflow-created-response.json | jq -j '.data.id')

Get block IDs

At UP42, we offer a wide selection of both data and processing blocks. Without a user interface to determine which block to add to your workflow, it may be a challenge. However, each block has its own unique ID, which you can use to pick and choose to build your workflow. We can get the display names of all blocks by issuing the following cURL command:

curl GET https://api.up42.com/blocks \
     -H "Authorization: Bearer $PTOKEN" | jq '.data[].displayName'

Next we will expand on the previous command, but now we are retrieving the block IDs by specifying which blocks we want to use through the blocks’ display names. Because the main goal of our workflow for this walkthrough is to perform an NDVI analysis, we are choosing the following blocks:

# Get block ID for Pléiades Reflectance (Download)
curl GET https://api.up42.com/blocks \
     -H "Authorization: Bearer $PTOKEN" \
     | jq -r '.data[] | select(.displayName=="Pléiades Reflectance (Download)") | .id'

# Get block ID for Pan-sharpening
curl GET https://api.up42.com/blocks \
     -H "Authorization: Bearer $PTOKEN" \
     | jq -r '.data[] | select(.displayName=="Pan-sharpening") | .id'

# Get block ID for NDVI
curl GET https://api.up42.com/blocks \
     -H "Authorization: Bearer $PTOKEN" \
     | jq -r '.data[] | select(.displayName=="NDVI") | .id'

Tasks

Now that we have the appropriate block IDs that we need for our NDVI analysis, let’s officially add them to our workflow. We add them as “tasks” and create a chain of tasks to build the workflow.

Add the first task to the workflow by:

curl POST https://api.up42.com/projects/$PROJ/workflows/$WORKFLOW/tasks \
     -H "Authorization: Bearer $PTOKEN" \
     -H 'Content-Type: application/json'  \
     -d '[{"name":"first block","parentName":null,"blockId": "defb134b-ca00-4e16-afa0-639c6dc0c5fe"}]' \
     | jq '.'

Copy the content of the first task:

[{"name":"first block","parentName":null,"blockId": "defb134b-ca00-4e16-afa0-639c6dc0c5fe"}] 

And paste this content in a file with a name of your choosing. Here we called this file task1.json.

pbpaste > task1.json

Do the same for both tasks 2 and 3. The final file, task3.json, is now a completed chain of tasks and will be used as the request body for the following POST request:

jq --argjson t2 '{"name":"second block","parentName":"first block","blockId": "903f0435-d638-475e-bbe9-53b5664a22a8"}' '. += [$t2]' task1.json > task2.json

jq --argjson t3 '{"name":"third block","parentName":"second block","blockId": "d0da4ac9-94c6-4905-80f5-c95e702ca878"}' '. += [$t3]' task2.json > task3.json

curl POST https://api.up42.com/projects/$PROJ/workflows/$WORKFLOW/tasks \
     -H "Authorization: Bearer $PTOKEN" \
     -H 'Content-Type: application/json'  \
     -d @task3.json | jq '.'

Depending on the size of your request body, it can be quite cumbersome and tedious to manually enter the contents of the request body in your cURL request. Instead, store the contents of the request body in a separate file and use @and the file name to do the same task... but in a more efficient way!

Job Parameters

At this step, you may want to check in the Console to see if your workflow is updated with not only the right blocks but also the right order of blocks. You can also easily retrieve the job parameters directly from the Console in the job configuration page.

An example request body looks like this:

Loading...

For this walkthrough, copy the contents of this file and save them into a file called job_params.json.

pbpaste > job_params.json

Run a test query

Before you run a real job, it’s always good practice to make sure that the image you need for your given criteria is available in the first place. You can perform this check by running a test query. Test queries are defined in the job parameters file as a dry run as shown below.

"config": {
    "mode": "DRY_RUN"
  },

First, create a url variable for the jobs of your workflow.

URL_POST_JOB="https://api.up42.com/projects/$PROJ/workflows/$WORKFLOW/jobs"

Next, run a test query using the newly created url, access token and your job_params.json as the request body:

curl POST "$URL_POST_JOB?name=Test+query" \
     -H "Authorization: Bearer $PTOKEN" \
     -H "Content-Type: application/json"\
     -d @job_params.json | jq '.'

Run a job

Now that we have checked to make sure that an image exists for our given job parameters through a test query, let’s go ahead and run a real job.

Make sure to remove this part from the job parameters file job_params.json to signify that you’re not running a test query:

"config": {
    "mode": "DRY_RUN"
  },

Another option is to replace DRY_RUN with DEFAULT, which flips the switch and makes the job run as a real job instead.

Now let's run the job by creating another cURL request using the url you just defined and the job_params.json file as the request body.

curl -L -s -X POST "$URL_POST_JOB?name=Real+job" \
     -H 'Content-Type: application/json' \
     -H "Authorization: Bearer $PTOKEN" \
     -d @job_params.json

Check job status

After running a job, it makes sense to have some indication of the job's status and to not just be left in the dark. You can check this by issuing the following request and extracting the field .data.status from the response body (copy the value of $JOB from the .data.id field returned by the previous POST request):

curl -L -s https://api.up42.com/projects/$PROJ/jobs/$JOB \
     -H "Authorization: Bearer $PTOKEN" \
     -H 'Content-Type: application/json' | jq -r '.data.status'

Status that may be returned include: RUNNING, FAILED, or SUCCESSFUL.

Download results

Once the job has successfully run, we can go ahead and download the results. First, create a url variable that contains the endpoint for extracting downloads from a job:

DOWNLOAD_URL="https://api.up42.com/projects/$PROJ/jobs/$JOB/downloads"

Extract the url from the response body and download the results:

TARBALL_URL=$(curl -s -L "$DOWNLOAD_URL/results" -H "Authorization: Bearer $PTOKEN" | jq -j '.data.url')
curl -o output-$JOB.tar.gz $TARBALL_URL

Finally, extract your results from the tarball:

   tar -xvzf output-$JOB.tar.gz

Run a job via an API script

Now that you are fully capable of creating a workflow and running a job through the API both in theory and in practice, you can assemble the commands in a script in order to automate and scale the procedure for your case study.

In this chapter, we will make use of a shell script written in Bash that uses only the project variables as inputs. The workflow for this script is based off of a demo workflow.

Download the script

We will use a provided script that handles the API commands. Clone the script using git, go into the script directory and make the file executable:

git clone https://gist.github.com/cb241bbd104aab30274d80ced9e4b313.git up42-first-api-script
cd up42-first-api-script
chmod 755 first_job_run.sh

Run the script

Retrieve your project variables (project key and project ID) as illustrated in the previous chapter.

Then run the script in the command line by providing the project key and project ID as arguments:

 ./first_job_run.sh -k <project API key> -p <project ID>

For example:

 ./first_job_run.sh -k pRj1h8Nv.Mmguja9BsLJXPWlvWt3h9vwAIftlcSHQSj1 -p 1ae70a40-07e9-47a7-9c25-bea19af1c451

UP42 API usage in the bash command line
UP42 API usage in the bash command line

As a response, you will get information about the status of the job, e.g. "Job <job ID> s RUNNING.". Here, <job ID> is a the random identifier (following the UUID standard) for the job you just created. If you invoke the script multiple times, each new job will get a unique job ID and a unique job name.

If you try to launch a second job while a job is already running, you will receive an error message corresponding to a 429 HTTP status code:

{
  "error": {
    "code": "TOO_MANY_REQUESTS",
    "message": "Your project has too many non-finished jobs",
    "details": null
  },
  "data": null
}

Going Further

Now that you can create a workflow and run a job from scratch through the API, you may consider consulting the API Walkthrough to learn how to do a catalog search for images, get an order price estimation, and modify already existing workflows, jobs, and tasks.