Skip to main content

​
Setting up Amazon S3

​
1. Add integration

Select Amazon S3 from the integrations page.

​
2. Configure settings

Fill out the form with the following settings:
SettingRequiredDescription
Name✓Name that will be displayed to users when selecting this integration in Superblocks
Region✓AWS region where the S3 bucket is hosted, e.g. us-east-1
Access Key ID✓Access key ID for your AWS account
Secret Key✓Secret access key for your AWS account
IAM Role ARN✗ARN of the role for Superblocks to assume for accessing S3 resources

​
3. Test and save

Click Test Connection to check that Superblocks can connect to the data source.
If using Superblocks Cloud, add these Superblocks IPs to your allowlist (not necessary for Hybrid or Cloud-Prem deployments).
After connecting successfully, click Create to save the integration.

​
4. Set profiles

Optionally, configure different profiles for separate development environments.
Amazon S3 connected!
You can now use Clark to build with Amazon S3.

​
Creating Amazon S3 steps

Connect to your S3 integration from Superblocks by creating steps in Application APIs, Workflows, and Scheduled Jobs. An S3 step can perform the following actions:
  • Delete Files
  • Generate presigned URL
  • List bucket objects
  • List buckets
  • Read file
  • Upload file
  • Upload multiple files
  • List files (deprecated)
Superblocks also supports connecting to AWS services with Boto3 in Python steps if you require additional functionality.

​
Retrieving More Than 1000 Files

Amazon S3 limits the number of objects retrieved up to 1000. To handle this limitation and fetch more than 1000 objects, use the following JavaScript in a backend API (modify or port to Python as needed): 
const AWS = require('aws-sdk');

const s3 = new AWS.S3({
  region: 'eu-central-1',
  accessKeyId: 'AWS_ACCESS_KEY_ID',
  secretAccessKey: 'AWS_SECRET_ACCESS_KEY',
});

async function listAllObjectsFromS3Bucket(bucket, prefix = '') {
  let isTruncated = true;
  let marker;
  const elements = [];
  
  while (isTruncated) {
    const params = {
      Bucket: bucket,
      Prefix: prefix,
      Marker: marker,
    };

    try {
      const response = await s3.listObjects(params).promise();

      response.Contents.forEach((item) => {
        elements.push(item.Key);
      });

      isTruncated = response.IsTruncated;

      if (isTruncated) {
        marker = response.Contents.slice(-1)[0].Key;
      }
    } catch (error) {
      throw error;
    }
  }
  
  return elements;
}

// example call
listAllObjectsFromS3Bucket('<your bucket name>', '<optional prefix>')
  .then((elements) => console.log(elements))
  .catch((error) => console.error('An error occurred: ', error));