By Konstantinos Patronas —

AWS: How to trigger a Lambda function from an S3 event

This tutorial will guide you through the process of integrating an AWS Lambda function with an S3 bucket to process JSON files…

AWS: How to trigger a Lambda function from an S3 event
Photo by From Marwool on Unsplash

This tutorial will guide you through the process of integrating an AWS Lambda function with an S3 bucket to process JSON files automatically.

We’ll cover the following steps:

  1. Create an S3 bucket

2. Create an IAM policy and role for Lambda

3. Enable optional logging with CloudTrail

4. Deploy a Lambda function

5. Configure the S3 bucket to trigger the Lambda function

6. Test the setup with sample data

Step 1: Create an S3 Bucket

First, create an S3 bucket to store the files that will trigger the Lambda function.

aws s3api create-bucket --bucket <bucket> --region us-east-1

Optional: Add Bucket Tagging

You can add tagging to the bucket for easier management:

aws s3api put-bucket-tagging --bucket <bucket> \ 
--tagging 'TagSet=[{Key=Owner,Value=<bucket>-rds-load-test}]'

Step 2: Create IAM Policy and Role

Lambda needs permissions to interact with your S3 bucket. We’ll create an IAM policy and role to grant these permissions.

Create IAM Policy

Save the following policy as s3-lambda-policy.json:

{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Effect": "Allow", 
            "Action": [ 
                "s3:ListBucket", 
                "s3:GetObject", 
                "s3:PutObject" 
            ], 
            "Resource": [ 
                "arn:aws:s3:::<bucket>", 
                "arn:aws:s3:::<bucket>/*" 
            ] 
        } 
    ] 
}

Run the command to create the policy:

aws iam create-policy --policy-name <bucket>LambdaS3AccessPolicy \ 
--policy-document file://s3-lambda-policy.json

Create IAM Role

Save the following trust policy as trust-policy.json:

{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Effect": "Allow", 
            "Principal": { 
                "Service": "lambda.amazonaws.com" 
            }, 
            "Action": "sts:AssumeRole" 
        } 
    ] 
}

Run the command to create the role:

aws iam create-role --role-name <bucket>LambdaExecutionRole \ 
--assume-role-policy-document file://trust-policy.json

Attach Policies to the Role

Attach the custom policy:

aws iam attach-role-policy --role-name <bucket>LambdaExecutionRole \ 
--policy-arn arn:aws:iam::<account-id>:policy/<bucket>LambdaS3AccessPolicy

Attach the AWS-managed Lambda execution role policy:

aws iam attach-role-policy \ 
--role-name <bucket>LambdaExecutionRole \ 
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Step 3: Enable CloudTrail Logging (Optional)

To log all S3 operations, enable CloudTrail.

Create a CloudTrail Policy

Save the following policy as s3bucketpolicy.json:

{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Sid": "AWSCloudTrailWrite", 
            "Effect": "Allow", 
            "Principal": { 
                "Service": "cloudtrail.amazonaws.com" 
            }, 
            "Action": "s3:PutObject", 
            "Resource": "arn:aws:s3:::<bucket>/AWSLogs/<account-id>/*", 
            "Condition": { 
                "StringEquals": { 
                    "s3:x-amz-acl": "bucket-owner-full-control" 
                } 
            } 
        }, 
        { 
            "Sid": "AWSCloudTrailList", 
            "Effect": "Allow", 
            "Principal": { 
                "Service": "cloudtrail.amazonaws.com" 
            }, 
            "Action": "s3:GetBucketAcl", 
            "Resource": "arn:aws:s3:::<bucket>" 
        } 
    ] 
}

Commands to Enable CloudTrail

aws s3api put-bucket-policy --bucket <bucket> --policy file://s3bucketpolicy.json 
aws cloudtrail create-trail --name <bucket>LambdaS3RDSLoad \ 
--s3-bucket-name <bucket> 
aws cloudtrail start-logging --name <bucket>LambdaS3RDSLoad

Step 4: Deploy a Lambda Function

Package the Lambda function code into a ZIP file:

zip function.zip lambda_function.py

Lambda Function Code (lambda_function.py)

import json 
import boto3 
 
def lambda_handler(event, context): 
    print("Received event:", json.dumps(event, indent=2))

Deploy the Lambda function:

aws lambda create-function \ 
    --function-name ProcessJsonFiles \ 
    --runtime python3.9 \ 
    --role arn:aws:iam::<account-id>:role/<bucket>LambdaExecutionRole \ 
    --handler lambda_function.lambda_handler \ 
    --zip-file fileb://function.zip \ 
    --region us-east-1

Step 5: Configure S3 to Trigger Lambda

Allow S3 to invoke the Lambda function:

aws lambda add-permission \ 
    --function-name ProcessJsonFiles \ 
    --statement-id AllowS3Invoke \ 
    --action lambda:InvokeFunction \ 
    --principal s3.amazonaws.com \ 
    --source-arn arn:aws:s3:::<bucket>

Configure S3 event notifications:

aws s3api put-bucket-notification-configuration \ 
    --bucket <bucket> \ 
    --notification-configuration '{ 
        "LambdaFunctionConfigurations": [ 
            { 
                "LambdaFunctionArn": "arn:aws:lambda:us-east-1:<account-id>:function:ProcessJsonFiles", 
                "Events": ["s3:ObjectCreated:*"], 
                "Filter": { 
                    "Key": { 
                        "FilterRules": [ 
                            {"Name": "suffix", "Value": ".json"} 
                        ] 
                    } 
                } 
            } 
        ] 
    }'

Step 6: Test the Setup

Option 1: Directly Invoke Lambda

Create a test payload in Base64 format:

echo '{"key": "value", "example": "test"}' | base64 > payload.b64

Decode the payload to confirm the content (optional):

cat payload.b64 | base64 -d

Invoke the Lambda function directly:

aws lambda invoke \ 
    --function-name ProcessJsonFiles \ 
    --payload "$(cat payload.b64)" \ 
    response.json

Option 2: Trigger Lambda by Uploading a File to S3

Upload a .json file to the bucket:

aws s3 cp test.json s3://<bucket>/

The Lambda function will automatically be triggered by the S3 event.

Key Differences

  1. Direct Lambda Invocation:
  • Used for testing or debugging the Lambda function in isolation.
  • Requires you to manually craft a payload and invoke the function using the AWS CLI.
  • Does not involve any actual S3 interaction.

2. S3 File Upload:

  • Simulates the real-world behavior where the Lambda function is triggered by an S3 event.
  • Allows you to test end-to-end integration between S3 and Lambda.
  • Automatically provides the event data generated by S3, eliminating the need to craft it manually.

Why Use Base64 Encoding?

  • Binary Safety: Prevents issues when transferring binary data over text-based protocols.
  • Cross-Platform Compatibility: Standardizes data encoding to avoid platform-specific issues.
  • AWS CLI Requirement: Ensures the payload is properly formatted for Lambda CLI invocation.

Conclusion

By following this tutorial, you have successfully set up an AWS Lambda function that automatically processes JSON files uploaded to an S3 bucket. You now understand how to configure S3 event notifications, IAM roles, and policies to ensure seamless integration. This setup is a foundational pattern for building serverless applications, and you can expand it further by incorporating additional AWS services, such as DynamoDB or SNS, to handle more complex workflows. Remember to monitor and optimize your setup for performance and cost-efficiency as you scale your solution.

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go: