GCP Cloud Storage

This guide explains how to configure Google Cloud Platform (GCP) service account credentials for use with the Data Factory HTTP task.

Overview

GCP Service Account authentication allows the Data Factory HTTP task to access Google Cloud services such as:

• Cloud Storage - Object storage (GCS)
• BigQuery - Data warehouse
• Cloud Functions - Serverless functions
• Pub/Sub - Messaging service
• Any Google API - Using OAuth2 tokens

Prerequisites

• A Google Cloud Platform account
• A GCP project created
• Access to the GCP Console (IAM & Admin)
• Product-Live account with access to the Data Factory platform

Step 1: Create a Service Account

1. Sign in to the Google Cloud Console
2. Select your project from the dropdown at the top
3. Navigate to IAM & Admin → Service Accounts
4. Click Create Service Account
5. Fill in the details:
   • Service account name: product-live-data-factory
   • Service account ID: (auto-generated)
   • Description: Service account for Product-Live Data Factory integration
6. Click Create and Continue

Step 2: Grant Permissions

Assign roles based on your use case:

For Cloud Storage Access

Role	Description
Storage Object Viewer	Read-only access to objects
Storage Object Creator	Create objects only
Storage Object Admin	Full access to objects
Storage Admin	Full access to buckets and objects

For BigQuery Access

Role	Description
BigQuery Data Viewer	Read access to datasets
BigQuery Data Editor	Read and write access to datasets
BigQuery Job User	Run queries

For Custom/Restricted Access

Create a custom role with specific permissions:

1. Go to IAM & Admin → Roles
2. Click Create Role
3. Add only the required permissions

Example permissions for Cloud Storage:

• storage.objects.create
• storage.objects.get
• storage.objects.list
• storage.objects.delete

After selecting roles, click Continue and then Done.

Step 3: Create a JSON Key

1. Click on the newly created service account
2. Go to the Keys tab
3. Click Add Key → Create new key
4. Select JSON format
5. Click Create

Important

The JSON key file will be downloaded automatically. Store it securely! This file contains sensitive credentials and should never be committed to version control.

The downloaded JSON file will look like this:

{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "abc123def456...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "product-live-data-factory@your-project.iam.gserviceaccount.com",
  "client_id": "123456789012345678901",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."
}

Step 4: Extract Credentials

From the JSON key file, extract the following values:

Field	JSON Key	Description
Project ID	project_id	Your GCP project identifier
Private Key ID	private_key_id	Unique identifier for the key
Private Key	private_key	RSA private key (PEM format)
Client Email	client_email	Service account email address
Client ID	client_id	Numeric client identifier

Step 5: Configure Data Factory Variables

Create the following variables in your Data Factory project:

Variable Name	Description	Mark as Secret
gcp_project_id	Project ID	No
gcp_private_key_id	Private Key ID	No
gcp_private_key	Private Key (full PEM string)	Yes
gcp_client_email	Service Account Email	No
gcp_client_id	Client ID	No
gcp_bucket_name	GCS Bucket Name (for storage operations)	No

Private Key Format

The private key must include the full PEM format with headers:

-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASC...
-----END PRIVATE KEY-----

Make sure to preserve the \n characters when copying.

Step 6: Use in HTTP Task

Example configuration for uploading a file to Cloud Storage:

{
    "name": "protocol-http-perform",
    "taskReferenceName": "upload-to-gcs",
    "type": "SUB_WORKFLOW",
    "inputParameters": {
        "scheme": "HTTPS",
        "method": "POST",
        "domain": "storage.googleapis.com",
        "path": "/upload/storage/v1/b/${workflow.variables.gcp_bucket_name}/o?uploadType=media&name=my-folder/my-file.txt",
        "headers": {
            "Content-Type": "text/plain"
        },
        "body": {
            "type": "PLAIN",
            "contentType": "text/plain",
            "text": "Hello from Product-Live Data Factory!"
        },
        "authentication": {
            "useAuthentication": true,
            "type": "GCP_SERVICE_ACCOUNT_KEY",
            "gcpServiceAccountKeyProjectId": "${workflow.variables.gcp_project_id}",
            "gcpServiceAccountKeyPrivateKeyId": "${workflow.variables.gcp_private_key_id}",
            "gcpServiceAccountKeyPrivateKey": "${workflow.variables.gcp_private_key}",
            "gcpServiceAccountKeyClientEmail": "${workflow.variables.gcp_client_email}",
            "gcpServiceAccountKeyClientId": "${workflow.variables.gcp_client_id}",
            "gcpServiceAccountKeyScope": ["https://www.googleapis.com/auth/cloud-platform"]
        },
        "responses": ["JSON"],
        "connectionTimeOutMilliseconds": 30000
    }
}

Common Cloud Storage Operations

List Objects in a Bucket

{
    "scheme": "HTTPS",
    "method": "GET",
    "domain": "storage.googleapis.com",
    "path": "/storage/v1/b/${workflow.variables.gcp_bucket_name}/o",
    "queryParameters": {
        "maxResults": "100"
    },
    "authentication": {
        "useAuthentication": true,
        "type": "GCP_SERVICE_ACCOUNT_KEY",
        "gcpServiceAccountKeyProjectId": "${workflow.variables.gcp_project_id}",
        "gcpServiceAccountKeyPrivateKeyId": "${workflow.variables.gcp_private_key_id}",
        "gcpServiceAccountKeyPrivateKey": "${workflow.variables.gcp_private_key}",
        "gcpServiceAccountKeyClientEmail": "${workflow.variables.gcp_client_email}",
        "gcpServiceAccountKeyClientId": "${workflow.variables.gcp_client_id}",
        "gcpServiceAccountKeyScope": ["https://www.googleapis.com/auth/cloud-platform"]
    },
    "responses": ["JSON"]
}

Download an Object

{
    "scheme": "HTTPS",
    "method": "GET",
    "domain": "storage.googleapis.com",
    "path": "/storage/v1/b/${workflow.variables.gcp_bucket_name}/o/my-folder%2Fmy-file.txt",
    "queryParameters": {
        "alt": "media"
    },
    "authentication": {
        "useAuthentication": true,
        "type": "GCP_SERVICE_ACCOUNT_KEY",
        "gcpServiceAccountKeyProjectId": "${workflow.variables.gcp_project_id}",
        "gcpServiceAccountKeyPrivateKeyId": "${workflow.variables.gcp_private_key_id}",
        "gcpServiceAccountKeyPrivateKey": "${workflow.variables.gcp_private_key}",
        "gcpServiceAccountKeyClientEmail": "${workflow.variables.gcp_client_email}",
        "gcpServiceAccountKeyClientId": "${workflow.variables.gcp_client_id}",
        "gcpServiceAccountKeyScope": ["https://www.googleapis.com/auth/cloud-platform"]
    },
    "responses": ["FILE"]
}

URL Encoding

Object names with / must be URL-encoded as %2F in the path.

Upload a Binary File

{
    "scheme": "HTTPS",
    "method": "POST",
    "domain": "storage.googleapis.com",
    "path": "/upload/storage/v1/b/${workflow.variables.gcp_bucket_name}/o?uploadType=media&name=my-folder/image.png",
    "headers": {
        "Content-Type": "image/png"
    },
    "body": {
        "type": "BINARY_FILE",
        "file": {
            "url": "${previous_task.output.file.url}"
        }
    },
    "authentication": {
        "useAuthentication": true,
        "type": "GCP_SERVICE_ACCOUNT_KEY",
        "gcpServiceAccountKeyProjectId": "${workflow.variables.gcp_project_id}",
        "gcpServiceAccountKeyPrivateKeyId": "${workflow.variables.gcp_private_key_id}",
        "gcpServiceAccountKeyPrivateKey": "${workflow.variables.gcp_private_key}",
        "gcpServiceAccountKeyClientEmail": "${workflow.variables.gcp_client_email}",
        "gcpServiceAccountKeyClientId": "${workflow.variables.gcp_client_id}",
        "gcpServiceAccountKeyScope": ["https://www.googleapis.com/auth/cloud-platform"]
    },
    "responses": ["JSON"]
}

Delete an Object

{
    "scheme": "HTTPS",
    "method": "DELETE",
    "domain": "storage.googleapis.com",
    "path": "/storage/v1/b/${workflow.variables.gcp_bucket_name}/o/my-folder%2Fmy-file.txt",
    "authentication": {
        "useAuthentication": true,
        "type": "GCP_SERVICE_ACCOUNT_KEY",
        "gcpServiceAccountKeyProjectId": "${workflow.variables.gcp_project_id}",
        "gcpServiceAccountKeyPrivateKeyId": "${workflow.variables.gcp_private_key_id}",
        "gcpServiceAccountKeyPrivateKey": "${workflow.variables.gcp_private_key}",
        "gcpServiceAccountKeyClientEmail": "${workflow.variables.gcp_client_email}",
        "gcpServiceAccountKeyClientId": "${workflow.variables.gcp_client_id}",
        "gcpServiceAccountKeyScope": ["https://www.googleapis.com/auth/cloud-platform"]
    }
}

OAuth2 Scopes Reference

Scope	Description
https://www.googleapis.com/auth/cloud-platform	Full access to all GCP services
https://www.googleapis.com/auth/devstorage.read_only	Read-only access to Cloud Storage
https://www.googleapis.com/auth/devstorage.read_write	Read/write access to Cloud Storage
https://www.googleapis.com/auth/devstorage.full_control	Full control of Cloud Storage
https://www.googleapis.com/auth/bigquery	Full access to BigQuery

Best Practice

Use the most restrictive scope possible. For Cloud Storage operations, prefer devstorage.read_write over cloud-platform.

Troubleshooting

Error: "Invalid grant: Invalid JWT Signature"

• The private key is incorrect or corrupted
• Ensure the full PEM format is preserved (including -----BEGIN PRIVATE KEY----- headers)
• Check that \n characters are properly interpreted

Error: "Permission denied"

• The service account doesn't have the required IAM roles
• Check bucket-level permissions (go to Cloud Storage → Bucket → Permissions)
• Verify the bucket name is correct

Error: "The caller does not have permission"

• The OAuth2 scope doesn't include the required permissions
• Add the appropriate scope to gcpServiceAccountKeyScope

Error: "Not Found"

• The bucket or object doesn't exist
• Check the bucket name and object path
• Ensure the region is correct (some operations require region-specific endpoints)

Security Best Practices

1. Use least privilege - Grant only the minimum required roles
2. Rotate keys regularly - Create new keys and delete old ones periodically
3. Use separate service accounts - One per application/environment
4. Never commit keys - Use Data Factory secrets for the private key
5. Enable audit logging - Monitor service account activity in Cloud Audit Logs
6. Set key expiration - Configure automatic key rotation if possible
7. Use VPC Service Controls - For sensitive data, restrict access to specific networks

GCP Regions Reference

Region	Location
us-central1	Iowa, USA
us-east1	South Carolina, USA
europe-west1	Belgium
europe-west3	Frankfurt, Germany
europe-west9	Paris, France
asia-east1	Taiwan
asia-northeast1	Tokyo, Japan

For a complete list, see GCP Regions and Zones.

Related Documentation

• HTTP Task Reference
• Cloud Storage Integrations Overview
• AWS S3
• Azure Blob Storage
• OVH Object Storage
• Scaleway Object Storage
• Data Factory Variables
• Create an Object in a GCP Bucket (Use Case)
• GCP IAM Documentation
• Cloud Storage JSON API