Learning Center

AWS S3 ​

This guide explains how to configure AWS credentials for use with the Data Factory HTTP task using AWS Signature V4 authentication.

Overview ​

AWS Signature Version 4 (SigV4) is the authentication method used to access AWS services via HTTP requests. The Data Factory HTTP task supports SigV4 for services like:

  • Amazon S3 - Object storage
  • API Gateway - Custom APIs
  • Lambda - Serverless functions (via API Gateway)
  • OpenSearch - Search and analytics
  • DynamoDB - NoSQL database (via HTTP API)

Prerequisites ​

  • An AWS account
  • Access to the AWS IAM console
  • Product-Live account with access to the Data Factory platform

Step 1: Create an IAM User ​

  1. Sign in to the AWS Management Console
  2. Navigate to IAM (Identity and Access Management)
  3. In the left sidebar, click Users
  4. Click Create user
  5. Enter a User name (e.g., product-live-data-factory)
  6. Click Next

Step 2: Set Permissions ​

  1. Choose Attach policies directly
  2. Select the appropriate policies based on your use case:

For S3 Access ​

PolicyDescription
AmazonS3ReadOnlyAccessRead-only access to all S3 buckets
AmazonS3FullAccessFull access to all S3 buckets

Or create a custom policy for specific bucket access:

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

For API Gateway Access ​

PolicyDescription
AmazonAPIGatewayInvokeFullAccessInvoke any API Gateway endpoint
  1. Click Next
  2. Review and click Create user

Step 3: Create Access Keys ​

  1. Click on the newly created user
  2. Go to the Security credentials tab
  3. Scroll down to Access keys
  4. Click Create access key
  5. Select Application running outside AWS
  6. Click Next
  7. (Optional) Add a description tag
  8. Click Create access key

Important

Save your credentials immediately! The Secret Access Key will only be shown once. Download the .csv file or copy both values:

  • Access key ID
  • Secret access key

Step 4: Configure Data Factory Variables ​

Create the following variables in your Data Factory project to store the AWS credentials securely:

Variable NameDescriptionExample
aws_access_key_idAWS Access Key IDAKIAIOSFODNN7EXAMPLE
aws_secret_access_keyAWS Secret Access KeywJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
aws_regionAWS Regioneu-west-1
aws_s3_bucket_nameS3 Bucket Name (for S3 operations)my-bucket

Security Best Practice

Mark the aws_secret_access_key variable as a secret in Data Factory to prevent it from being displayed in logs.

Step 5: Use in HTTP Task ​

Example configuration for listing objects in an S3 bucket:

json
{
    "name": "protocol-http-perform",
    "taskReferenceName": "list-s3-objects",
    "type": "SUB_WORKFLOW",
    "inputParameters": {
        "scheme": "HTTPS",
        "method": "GET",
        "domain": "${workflow.variables.aws_s3_bucket_name}.s3.${workflow.variables.aws_region}.amazonaws.com",
        "path": "/",
        "queryParameters": {
            "list-type": "2",
            "max-keys": "100"
        },
        "authentication": {
            "useAuthentication": true,
            "type": "AWS_SIGNATURE_V4",
            "accessKeyId": "${workflow.variables.aws_access_key_id}",
            "secretAccessKey": "${workflow.variables.aws_secret_access_key}",
            "region": "${workflow.variables.aws_region}",
            "service": "s3"
        },
        "responses": ["STRING", "JSON"],
        "connectionTimeOutMilliseconds": 10000
    }
}

Common S3 Operations ​

Upload a File ​

json
{
    "scheme": "HTTPS",
    "method": "PUT",
    "domain": "${workflow.variables.aws_s3_bucket_name}.s3.${workflow.variables.aws_region}.amazonaws.com",
    "path": "/my-folder/my-file.txt",
    "headers": {
        "Content-Type": "text/plain"
    },
    "body": {
        "type": "PLAIN",
        "contentType": "text/plain",
        "text": "Hello from Product-Live Data Factory!"
    },
    "authentication": {
        "useAuthentication": true,
        "type": "AWS_SIGNATURE_V4",
        "accessKeyId": "${workflow.variables.aws_access_key_id}",
        "secretAccessKey": "${workflow.variables.aws_secret_access_key}",
        "region": "${workflow.variables.aws_region}",
        "service": "s3"
    }
}

Download a File ​

json
{
    "scheme": "HTTPS",
    "method": "GET",
    "domain": "${workflow.variables.aws_s3_bucket_name}.s3.${workflow.variables.aws_region}.amazonaws.com",
    "path": "/my-folder/my-file.txt",
    "authentication": {
        "useAuthentication": true,
        "type": "AWS_SIGNATURE_V4",
        "accessKeyId": "${workflow.variables.aws_access_key_id}",
        "secretAccessKey": "${workflow.variables.aws_secret_access_key}",
        "region": "${workflow.variables.aws_region}",
        "service": "s3"
    },
    "responses": ["FILE"]
}

Delete a File ​

json
{
    "scheme": "HTTPS",
    "method": "DELETE",
    "domain": "${workflow.variables.aws_s3_bucket_name}.s3.${workflow.variables.aws_region}.amazonaws.com",
    "path": "/my-folder/my-file.txt",
    "authentication": {
        "useAuthentication": true,
        "type": "AWS_SIGNATURE_V4",
        "accessKeyId": "${workflow.variables.aws_access_key_id}",
        "secretAccessKey": "${workflow.variables.aws_secret_access_key}",
        "region": "${workflow.variables.aws_region}",
        "service": "s3"
    }
}

AWS Regions Reference ​

Region CodeRegion Name
us-east-1US East (N. Virginia)
us-west-2US West (Oregon)
eu-west-1Europe (Ireland)
eu-west-3Europe (Paris)
eu-central-1Europe (Frankfurt)
ap-northeast-1Asia Pacific (Tokyo)
ap-southeast-1Asia Pacific (Singapore)

For a complete list, see AWS Regions and Endpoints.

Troubleshooting ​

Error: "The request signature we calculated does not match the signature you provided" ​

  • Verify that the Access Key ID and Secret Access Key are correct
  • Ensure the region matches the bucket's actual region
  • Check that the service name is correct (s3 for S3, execute-api for API Gateway)

Error: "Access Denied" ​

  • Verify the IAM user has the required permissions
  • Check the bucket policy allows access from the IAM user
  • Ensure the bucket name is correct

Error: "NoSuchBucket" ​

  • Verify the bucket name is spelled correctly
  • Ensure you're using the correct region for the bucket

Security Best Practices ​

  1. Use IAM roles when possible - For EC2/Lambda, use IAM roles instead of access keys
  2. Rotate credentials regularly - Create new access keys and delete old ones periodically
  3. Use least privilege - Only grant the minimum permissions required
  4. Never commit credentials - Use Data Factory variables/secrets instead
  5. Enable MFA - For the root account and privileged users
  6. Monitor with CloudTrail - Enable logging to track API calls