Amazon S3

Overview

Connect Superblocks to S3 to build apps that can list, read, delete, and upload files in S3:

• Read data from S3 and utilize it in scheduled reports

• Upload data from API steps or local files to S3

Setting up Amazon S3

1. Add integration

Select Amazon S3 from the integrations page.

2. Configure settings

Fill out the form with the following settings:

Cloud

Setting	Required	Description
Name	TRUE	Name that will be displayed to users when selecting this integration in Superblocks
Region	TRUE	AWS region where the S3 bucket is hosted, e.g. us-east-1
Access Key ID	TRUE	Access key ID for your AWS account
Secret Key	TRUE	Secret access key for your AWS account
IAM Role ARN	FALSE	ARN of the role for Superblocks to assume for accessing S3 resources

On-Premise Agent

Setting	Required	Description
Name	TRUE	Name that will be displayed to users when selecting this integration in Superblocks
Auth Type	TRUE	Choose between: Access Key - Use access key ID and secret access key authorization Token File - Use Service Account Token Volume Projection to authorize your On-Premise Agent to connect to your S3 Buckets EC2 Instance Metadata - Use EC2 Instance Metadata to authorize your On-Premise Agent to connect to your S3 Buckets. If you are using Kube2Iam, you would select this option.
Region	If Auth Type is Access Key	AWS region where the S3 bucket is hosted, e.g. us-east-1
Access Key ID	If Auth Type is Access Key	Access key ID for your AWS account
Secret Key	If Auth Type is Access Key	Secret access key for your AWS account
IAM Role ARN	FALSE	ARN of the role for Superblocks to assume for accessing S3 resources

3. Test and save

Click Test Connection to check that Superblocks can connect to the data source.

info

If using Superblocks Cloud, add these Superblocks IPs to your allowlist (not necessary for On-Premise-Agent).

After connecting successfully, click Create to save the integration.

4. Set profiles

Optionally, configure different profiles for separate development environments.

tip

Amazon S3 connected Now you can use Amazon S3 in any Application, Workflow, or Scheduled Job.

Creating Amazon S3 steps

Connect to your S3 integration from Superblocks by creating steps in Application APIs, Workflows, and Scheduled Jobs. An S3 step can perform the following actions:

List Files

Read File

Delete Files

Upload File

Upload Multiple Files

List Buckets

info

Superblocks also supports connecting to AWS services with Boto3 in Python steps if you require additional functionality.

Use cases

Applications

Drag files into an application using the FilePicker component, and upload them to S3. See more details in the FilePicker guide here.

Workflows

Export a Google Sheet as a CSV to an S3 bucket.

Scheduled Jobs

Query order analytics, update an inventory prediction model, and upload it to S3 for the data science team to use.

Retrieving More Than 1000 Files

caution

Please keep in mind when retrieving large amounts of data and returning it to the browser that there are memory constraints. See Returning data from the Application backend to the frontend for more information.

Amazon S3 limits the number of Objects retrieved up to 1000. Here is an example of handling this limitation using a Backend JavaScript step. Please keep in mind you may need to modify or port this example to Python to get it working for your use case:

const AWS = require('aws-sdk');

const s3 = new AWS.S3({
  region: 'eu-central-1',
  accessKeyId: 'AWS_ACCESS_KEY_ID',
  secretAccessKey: 'AWS_SECRET_ACCESS_KEY',
});

async function listAllObjectsFromS3Bucket(bucket, prefix = '') {
  let isTruncated = true;
  let marker;
  const elements = [];
  
  while (isTruncated) {
    const params = {
      Bucket: bucket,
      Prefix: prefix,
      Marker: marker,
    };

    try {
      const response = await s3.listObjects(params).promise();

      response.Contents.forEach((item) => {
        elements.push(item.Key);
      });

      isTruncated = response.IsTruncated;

      if (isTruncated) {
        marker = response.Contents.slice(-1)[0].Key;
      }
    } catch (error) {
      throw error;
    }
  }
  
  return elements;
}

// example call
listAllObjectsFromS3Bucket('<your bucket name>', '<optional prefix>')
  .then((elements) => console.log(elements))
  .catch((error) => console.error('An error occurred: ', error));

Troubleshooting

Check out our guide on common errors across database integrations. If you are encountering an error that you don't see in the guide, or the provided steps are insufficient to resolve the error, please contact us at help@superblocks.com.