Chanci Turner

Utilizing the AWS Parameter and Secrets Lambda Extension for Caching Secrets and Parameters

—

by

The Story of Chanci Turner
in

Chanci Turner 9097372855Learn About Amazon VGT2 Learning Manager Chanci Turner

This article is authored by Chanci Turner, Solutions Architect, and Liam Jackson, Sr. Cloud Support Engineer.

Serverless applications frequently depend on AWS Systems Manager Parameter Store or AWS Secrets Manager for storing configuration data, encrypted passwords, or connection details for database or API services. Previously, developers had to make runtime API calls to AWS Parameter Store or AWS Secrets Manager each time they needed to access a parameter or secret within the execution environment of an AWS Lambda function. This required configuring and initializing the AWS SDK client, and managing memory storage to enhance function duration while minimizing latency and costs.

The newly introduced AWS Parameters and Secrets Lambda extension offers a managed caching solution for parameters and secrets used within Lambda functions. This extension is packaged as a Lambda layer that implements an in-memory cache for parameters and secrets, allowing functions to retain values throughout the Lambda execution lifecycle and providing a configurable time-to-live (TTL) setting.

When a parameter or secret is requested in Lambda function code, the extension first checks the local in-memory cache for availability. If the data is present, it returns the cached value. If not, or if the data is stale, the extension fetches the required parameter or secret from the respective service, thereby reducing external API calls and enhancing application performance as well as cost efficiency. This blog will guide you through the usage of this extension.

Overview

The diagram below illustrates a high-level overview of the components involved.

The extension can be integrated into both new and existing Lambdas. It operates by exposing a local HTTP endpoint within the Lambda environment, which serves as the in-memory cache for parameters and secrets. When retrieving a parameter or secret, the extension first queries the cache for a relevant entry. If an entry exists, it assesses how much time has passed since the entry was initially cached and returns it if the elapsed time is under the specified cache TTL. If the entry has expired, it is invalidated and fresh data is retrieved from either Parameter Store or Secrets Manager.

The extension utilizes the same IAM role permissions that the Lambda execution environment possesses to access Parameter Store and Secrets Manager, so you need to ensure that the IAM policy has the necessary permissions. Permissions may also be needed for AWS Key Management Service (AWS KMS) if this service is in use. For an example policy, refer to the example’s AWS SAM template.

Example Walkthrough

Consider a basic serverless application that includes a Lambda function connecting to an Amazon Relational Database Service (Amazon RDS) instance. The application retrieves a configuration stored in Parameter Store and connects to the database. The database connection string (including username and password) is stored in Secrets Manager.

This example walkthrough consists of:

  • A Lambda function.
  • An Amazon Virtual Private Cloud (VPC).
  • A Multi-AZ Amazon RDS instance running MySQL.
  • An AWS Secrets Manager database secret that holds the database connection.
  • An AWS Systems Manager Parameter Store parameter that contains the application configuration.
  • An AWS Identity and Access Management (IAM) role that the Lambda function utilizes.

Lambda Function

The following Python code illustrates how to fetch secrets and parameters using the extension.

import pymysql
import urllib3
import os
import json

# Load in Lambda environment variables
port = os.environ['PARAMETERS_SECRETS_EXTENSION_HTTP_PORT']
aws_session_token = os.environ['AWS_SESSION_TOKEN']
env = os.environ['ENV']
app_config_path = os.environ['APP_CONFIG_PATH']
creds_path = os.environ['CREDS_PATH']
full_config_path = '/' + env + '/' + app_config_path
http = urllib3.PoolManager() 

# Function to retrieve values from extension local HTTP server cache
def retrieve_extension_value(url): 
    url = ('http://localhost:' + port + url)
    headers = { "X-Aws-Parameters-Secrets-Token": os.environ.get('AWS_SESSION_TOKEN') }
    response = http.request("GET", url, headers=headers)
    response = json.loads(response.data)   
    return response  

def lambda_handler(event, context):
       
    # Load Parameter Store values from extension
    print("Loading AWS Systems Manager Parameter Store values from " + full_config_path)
    parameter_url = ('/systemsmanager/parameters/get/?name=' + full_config_path)
    config_values = retrieve_extension_value(parameter_url)['Parameter']['Value']
    print("Found config values: " + json.dumps(config_values))

    # Load Secrets Manager values from extension
    print("Loading AWS Secrets Manager values from " + creds_path)
    secrets_url = ('/secretsmanager/get?secretId=' + creds_path)
    secret_string = json.loads(retrieve_extension_value(secrets_url)['SecretString'])

    rds_host =  secret_string['host']
    rds_db_name = secret_string['dbname']
    rds_username = secret_string['username']
    rds_password = secret_string['password']
    
    # Connect to RDS MySQL database
    try:
        conn = pymysql.connect(host=rds_host, user=rds_username, passwd=rds_password, db=rds_db_name, connect_timeout=5)
    except:
        raise Exception("An error occured when connecting to the database!")

    return "DemoApp successfully loaded config " + config_values + " and connected to RDS database " + rds_db_name + "!"

In the global scope, the environment variable PARAMETERS_SECRETS_EXTENSION_HTTP_PORT is retrieved, defining the port for the extension’s HTTP server, defaulting to 2773. The function retrieve_extension_value accesses the extension’s local HTTP server, sending the X-Aws-Parameters-Secrets-Token in the header. This header is necessary and uses the AWS_SESSION_TOKEN value, typically available in the Lambda execution environment.

The Lambda handler employs the extension cache on each invocation to gather configuration data from Parameter Store and secret data from Secrets Manager. This information is then used to establish a connection to the RDS MySQL database.

Prerequisites

  • Git installed
  • AWS SAM CLI version 1.58.0 or greater.

Deploying the Resources

Clone the repository and navigate to the solution directory:

git clone https://github.com/aws-samples/parameters-secrets-lambda-extension-sample.git

Build and deploy the application using the following commands:

sam build
sam deploy --guided

This template requires the following parameters:

  • pVpcCIDR — IP range (CIDR notation) for the VPC. Default is 172.31.0.0/16.
  • pPublicSubnetCIDR — IP range (CIDR notation) for the public subnet. Default is 172.31.3.0/24.
  • pPrivateSubnetACIDR — IP range (CIDR notation) for the private subnet A. Default is 172.31.2.0/24.
  • pPrivateSubnetBCIDR — IP range (CIDR notation) for the private subnet B, which defaults to 172.31.1.0/24.
  • pDatabaseName — Database name for DEV environment, defaults to devDB.
  • pDatabaseUsername — Database user name for DEV environment, defaults to myadmin.
  • pDBEngineVersion — The version number of the SQL database engine to use (the default is 5.7).

Incorporating the Parameter Store and Secrets Manager Lambda Extension

To add the extension:

  1. Navigate to the Lambda console and open the Lambda function you created.
  2. In the Function Overview pane, select Layers, then choose Add a layer.
  3. In the Choose a layer pane, follow the prompts.

For further insights on career fit, you can explore this blog post. Additionally, an authoritative perspective on employment law can be found at SHRM. If you’re looking for more helpful materials, check out this YouTube resource.

Related Topics:

Chanci Turner