Cloud bucket upload

Upload data from storage to your cloud bucket.

Overview

Use cloud bucket upload to copy CNAM-transformed data items from UP42 storage to your own cloud bucket. We run an upload job on your behalf using cross-cloud identity federation between Google Cloud Platform (GCP) and your bucket.

Once configuration and verification are complete, uploads run automatically. Every new data item that appears in your storage is copied to your bucket without any action on your side. Data items that were already in storage before setup aren’t uploaded retroactively.

Each upload contains the JSON file of the data item and its associated assets, using a fixed path structure:

DescriptionObject key
Data item metadataup42/<version>/<data-item-id>/item.json
Asset fileup42/<version>/<data-item-id>/<asset-id>

The <version> variable identifies the storage layout version, for example, v0.

Requirements

To use cloud bucket upload, ensure the following:

  • You have an UP42 Enterprise plan account.
  • You have an Amazon S3 bucket in your own Amazon Web Services (AWS) account.

Set up cloud bucket upload

Reach out to your UP42 contact to request the cloud bucket upload feature and start the setup process below. Steps are tagged by who typically completes them:

Stakeholder

A project lead or other decision-maker responsible for planning, approvals, and configuration decisions.

SRE

A Site Reliability Engineer or Cloud Infrastructure Engineer in your organization.

1

Request access

Stakeholder
  1. Open the console.
  2. In the bottom-left corner, click your user icon → Account management.
  3. To copy your account ID, click Copy inline-icon.
  4. Send the following information to your UP42 contact:
    • S3 bucket region
    • S3 bucket name you want to use

Once we receive this information from you, we’ll create a GCP service account and share with you the OAuth 2 client ID required for the AWS trust policy.

2

Create an S3 bucket

SRE
  1. In your AWS account, create the bucket in the agreed-upon region.
  2. Enable default encryption and block all public access.
  3. Confirm the bucket is reachable via public S3 endpoints.
3

Create an IAM role

SRE

In your AWS account, create a role. Use the GCP account ID we provided in the trust policy in the following template:

JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Federated": "accounts.google.com" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:sub": "<OAUTH_2_CLIENT_ID>"
}
}
}
]
}

No separate IAM OIDC provider setup is needed for Google, as AWS already includes accounts.google.com as a built-in federated principal.

4

Add S3 permissions

SRE

Attach a policy that grants the upload job the following permissions on arn:aws:s3:::your-bucket/*:

  • s3:PutObject
  • s3:AbortMultipartUpload
  • s3:DeleteObject
  • s3:ListBucket (used for verification)
  • s3:GetObject (used for verification)

If you use AWS KMS, the IAM role must also include kms:Decrypt and kms:GenerateDataKey permissions for the relevant key.

5

Verify integration

SRE

Provide your IAM role ARN (for example, arn:aws:iam::123456789012:role/upload-executor). We need it so our GCP service account can assume your role as part of the identity federation during upload.

We’ll then run an end-to-end upload test for a data item in your UP42 account. Once verification succeeds, we’ll confirm that the functionality is ready for you to use.

6

Find uploaded data

Stakeholder

When a new data item is delivered to your storage, the item and its assets will be automatically be uploaded to your bucket under up42/<version>/<data-item-id>/.

For example, a data item with ID abc-123 and two assets will appear in your bucket the following way:

Text
your-bucket/
up42/
v0/
abc-123/
item.json
1234-5678
5678-1234

Troubleshooting

Can I use a custom S3 key prefix? Can I use a custom S3 key prefix?

No. We write objects under the fixed up42/v0/ prefix. Custom key prefixes aren’t supported.

My bucket uses VPC endpoints or IP allowlists. Will upload work? My bucket uses VPC endpoints or IP allowlists. Will upload work?

Our upload job writes to S3 over the public S3 API. Buckets restricted to VPC endpoints, IP allowlists, or private-only access won’t work unless private connectivity is arranged separately. Contact support to discuss options.

Who should configure AWS on our side? Who should configure AWS on our side?

Cloud bucket upload requires IAM and S3 configuration in your AWS account. Coordinate with SRE or cloud infrastructure team within your organization.