--- title: "Export Pigment data to Amazon S3" slug: "export-amazon-s3" description: "Transfer Pigment data to Amazon S3 using our REST API and AWS Lambda. Follow our guide for seamless exports and scheduling with EventBridge." tags: ["Amazon S3", "AWS Lambda", "Importing Exporting Data", "Pigment API"] updated: 2025-05-30T13:33:44Z published: 2025-08-22T11:56:47Z --- > ## Documentation Index > Fetch the complete documentation index at: https://kb.pigment.com/llms.txt > Use this file to discover all available pages before exploring further. # Export Pigment data to Amazon S3 You can transfer data from your Pigment Views, Lists, Metrics, and Tables to Amazon S3 with our REST API. First, you create a Amazon Lambda function that runs a Python script. This script retrieves the Pigment API data and stores it in an Amazon S3 bucket. ## Recommended reading In case you need it, we’ve compiled some useful background information for you: - [Using AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) - [Using Amazon IAM Manager](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) - [Using Lambda environment variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html) - [Using Amazon EventBridge Scheduler](https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html) - [How to export data from Pigment with APIs](/v1/docs/export-data-with-apis) - [How to trigger an import with APIs](/v1/docs/trigger-import-apis) ## Before you begin Before you start your export, we recommend that you complete the following tasks in Amazon S3 and Pigment. ### Amazon S3 - You need an Amazon S3 account to perform this export. - Consult with your Amazon S3 admin. You may need to obtain permissions and information to successfully complete your export. ### Pigment > [!WARNING] > **⚠️****Important** > > Only Pigment members with the account type Workspace Admin or Security Admin can access the [Access API key management page](/v1/docs/manage-api-keys) and manage API keys. - Obtain your export API key. This is explained in [Manage API Keys](/v1/docs/manage-api-keys). - Obtain your View ID. This is explained in [How to export data from Pigment with APIs](/v1/docs/export-data-with-apis). - Obtain your List, Metric, or Table IDs. This is explained in [Export raw data from a Pigment Block](/v1/docs/raw-data-block). ## Create an AWS Lambda function Here’s how you create an AWS Lambda function that loads Pigment API data into a S3 bucket. > [!NOTE] > ℹ️ **Note** > > In the following example, we use `view_ID`, however you can use `listID`, `metricID`, or `tableID` as appropriate. 1. In AWS Secrets Manager, create a secret using your Export API key as a value. 2. In Lambda, create a new AWS Lambda function, and complete these steps: 1. In the Function name field, enter your new function name. 2. In the Runtime menu, select `Python 3.7` ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/3dbf0f7d-1ab3-4fd3-ac64-0a4f45fa4ee9.png) 3. Click **Configuration**. 4. Enter the following environment variable values: - `view_ID`: The View ID of the block you want to export from Pigment. For information on obtaining this ID information, see [here](/v1/docs/export-data-with-apis). -`AWS_secret_name`: The secret name that corresponds to the Export API Key in AWS Secrets Manager. For example: `pigment_export_api_key` - `AWS_secret_region`: The region your AWS Secrets Manager entry resides in. For example: `us-east-1` - `S3_bucket_name`:the name of your S3 bucket that holds your files. - `secret_key`: the secret key corresponding to your Pigment Export API Key entry in AWS Secrets Manager. ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/2c2da8d2-c55d-4e42-9c86-7270d670805a.png) 5. In the Configuration pane, click **Permissions**, and complete the following steps: 1. Under role name, click the required role for the export. ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/e43f2a76-241d-4404-9fe9-39df94897f76.png) 2. Configure the AWS IAM manager so your Lambda project's IAM user can: - read/write to S3 - read from AWS Secrets Manager 3. Save your changes. > [!NOTE] > ℹ️ **Note** > > Depending on your organization's access management policy, you can use either a role or a policy. ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/164584d9-50b6-46e9-9bcd-4e9daed27d6e.png) 6. Click the **Code** tab, and add the following Python code: ```python from datetime import datetime import os import json import boto3 import requests from botocore.exceptions import ClientError # Init AWS Secrets Manager def get_secret():    # Replace with your AWS secrets manager details via the environment variables    SECRET_NAME = os.environ['AWS_SECRET_NAME']    REGION_NAME = os.environ['AWS_SECRET_REGION']    SECRET_KEY = os.environ['SECRET_KEY']    # Create a Secrets Manager client    session = boto3.session.Session()    client = session.client(        service_name='secretsmanager',        region_name=REGION_NAME    )    try:        get_secret_value_response = client.get_secret_value(            SecretId=SECRET_NAME        )    except ClientError as e:        # For a list of exceptions thrown, see        # https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html        raise e    # Decrypts secret using the associated KMS key.    secret_response = get_secret_value_response['SecretString']    secret_dict = json.loads(secret_response)    return secret_dict[SECRET_KEY] # Replace with your S3 Bucket name and desired S3 name (object name) # Adding a folder for each export attempt, feel free to remove if overkill S3_BUCKET_NAME = os.environ['S3_BUCKET_NAME'] DT = datetime.now() DT_ISO = DT.strftime("%Y-%m-%dT%H:%M:%SZ") S3_FILE = f'pigment_exports/{DT_ISO}/export.csv' def lambda_handler(event, context):    # Yor Pigment block's App ID and View ID    VIEW_ID = os.environ['VIEW_ID']    # Compose export API URI    API_URL = f'https://pigment.app/api/export/view/{VIEW_ID}'      KEY = get_secret()    HEADERS = {'Authorization': 'bearer '+KEY}    # Initialize S3 client    s3 = boto3.client('s3')    try:        # Make the HTTP GET request to the API        response = requests.get(API_URL, headers=HEADERS)        if response.status_code == 200:            # Save the content to S3            s3.put_object(Bucket=S3_BUCKET_NAME, Key=S3_FILE, Body=response.content)            return {                'statusCode': 200,                'body': json.dumps('File saved to S3 successfully!')            }        else:            return {                'statusCode': response.status_code,                'body': json.dumps('Failed to retrieve data from the API.')            }    except Exception as e:        return {            'statusCode': 500,            'body': json.dumps(f'Error: {str(e)}')        } ``` 7. Click **Deploy** and then **Test**. 8. Go to your S3 bucket. 9. Locate the folder that corresponds to the ISO 8601 date and time when you ran the test. If your test was successful, you’ll find your exported Pigment data in the CSV file. ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/3bdd22e3-a38e-4904-9004-d36460d1efe6.png) ## Schedule regular exports with EventBridge Scheduler 1. In AWS, go to EventBridge Scheduler and click **Create rule**. 2. In the Schedule name field, enter your rule name. ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/c0e62d72-e835-4560-b26c-c2560464d330.png) 3. Define your schedule, and click **Next**. ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/0c5218f4-dc3b-497a-8b65-a0c56e8a325a.png) 4. In the Select Target pane, select **AWS Lambda.** ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/57b2bcf7-64ce-4a3c-be4d-a3a10b9d74a6.png) 5. Select your Lambda function. ![](https://cdn.document360.io/e47cfe35-dc28-40c7-a083-6cf003073d8e/Images/Documentation/3782c7ba-2b95-42ff-a214-f7d5a0ee6f5b.png) 6. Complete the remaining setup tasks. Your scheduling job is set up, and runs at 9:00AM GMT daily.