Learn About Amazon VGT2 Learning Manager Chanci Turner
Many organizations seek reliable automated code delivery for their applications. To achieve this, they utilize multi-account continuous integration/continuous deployment (CI/CD) pipelines, enabling them to deploy code and conduct automated tests across various environments before moving to production. However, when the testing strategy is specific to each release, updates to the pipeline are necessary before every deployment. Traditional pipeline stages are often predefined and static, making it challenging to adapt them once established.
In this article, we introduce a configuration-driven dynamic CI/CD solution tailored for individual repositories. The pipeline state is managed and governed by configurations stored in Amazon DynamoDB, allowing for automatic customization for each release based on testing requirements.
By following this guide, you will configure a dynamic multi-account CI/CD solution. Your pipeline will deploy and test a sample pet store API application. For further insights on this application, check out this blog post. New code deployments will follow custom pipeline stages derived from the pipeline configuration you establish. This solution leverages services such as AWS Cloud Development Kit (AWS CDK), AWS CloudFormation, Amazon DynamoDB, AWS Lambda, and AWS Step Functions.
Solution Overview
The architecture of the solution is depicted in the following diagram:
Figure 1: Architecture Diagram
Users can insert, update, or delete entries in the DynamoDB table. The Step Function Trigger Lambda is activated upon any modifications. This Lambda assesses the incoming event and performs the following:
- On insert and update, it triggers the Step Function.
- On deletion, it locates the corresponding CloudFormation stack and removes it.
The steps within the Step Function are as follows:
- Collect Information (Pass State) – Filters relevant details from the event, including repositoryName and referenceName.
- Get Mapping Information (Backed by CodeCommit event filter Lambda) – Retrieves mapping information from the Pipeline config stored in DynamoDB.
- Deployment Configuration Exist? (Choice State) – If StatusCode equals 200, the DynamoDB entry is confirmed, and the Initiate CloudFormation Stack step is invoked; otherwise, the Step Function exits successfully.
- Initiate CloudFormation Stack (Backed by stack creation Lambda) – Constructs the CloudFormation parameters and creates or updates the dynamic pipeline based on the configuration from DynamoDB via CloudFormation.
Code Deliverables
The code deliverables encompass the following:
- AWS CDK App – Contains the code for all Lambdas, Step Functions, and CloudFormation templates.
- sample-application-repo – This directory holds the sample application repository utilized for deployment.
- automated-tests-repo – Contains the sample automated tests repository for validating the sample repo.
Deploying the CI/CD Solution
- Clone this repository to your local machine.
- Follow the README to deploy the solution in your primary CI/CD account. Upon successful deployment, the following resources will be created in the CI/CD account:
- A DynamoDB table
- Step Function
- Lambda Functions
- Go to the Amazon Simple Storage Service (Amazon S3) console in your main CI/CD account and look for a bucket named: cloudformation-template-bucket-. You should see two CloudFormation templates (templates/codepipeline.yaml and templates/childaccount.yaml) uploaded there.
- Execute childaccount.yaml in every target CI/CD account (Alpha, Beta, Gamma, and Prod) via the CloudFormation Console. Enter the main CI/CD account number as the “CentralAwsAccountId” parameter, and run it.
- Upon successful Stack creation, two roles will be established in the Child Accounts:
- ChildAccountFormationRole
- ChildAccountDeployerRole
Pipeline Configuration
Make an entry in the devops-pipeline-table-info for the Repository name and branch combination. A sample entry can be found in sample-entry.json. The pipeline is highly configurable, and all settings can be adjusted through the DynamoDB entry.
The primary keys include:
- RepoName: The name of the repository configured for AWS CodePipeline.
- RepoTag: The branch name utilized in CodePipeline.
- BuildImage: The build image selected for the application’s AWS CodeBuild project.
- BuildSpecFile: The buildspec file used in the CodeBuild project.
- DeploymentConfigurations: This key contains the deployment configurations for the pipeline. We’ve designated our environments as Alpha, Beta, Gamma, and Prod, but you can customize the names. Ensure that the entries in JSON correspond to those in the codepipeline.yaml CloudFormation template, as there is a 1:1 mapping.
Sub-level keys under DeploymentConfigurations include:
- EnvironmentName: This key represents the configuration for each specific environment.
- AwsAccountId: The AWS account ID for the target environment.
- Deploy: A key indicating whether the artifact should be deployed to this environment.
- ManualApproval: A key denoting if manual approval is required before deployment—insert your email or set to false.
- Tests: This top-level key contains sub-level keys with test-related information for specific environments. Each test will add a step to the CodePipeline based on its configuration, including specifying the test repository, branch name, buildspec file, and build image for the testing CodeBuild project.
Execution
Make an entry in the devops-pipeline-table-info DynamoDB table within the main CI/CD account. A sample entry can be found in sample-entry.json. Ensure you replace the configuration values with those appropriate for your environment. An explanation of these values can be found in the Pipeline Configuration section above.
Once the entry is made in the DynamoDB table, a CloudFormation stack will begin to create. This stack will deploy the CodePipeline in the main CI/CD account by referencing the entry in the DynamoDB table.
You can customize the solution for different deployment configurations by editing the pipeline settings stored in the devops-pipeline-table-info DynamoDB table. The following depicts the pipeline configured for the sample-application repository’s main branch.
Figure 2: Dynamic Multi-Account CI/CD Pipeline
Cleaning Up Your Dynamic Multi-Account CI/CD Solution and Related Resources
To prevent ongoing charges for the resources established during this process, you should delete the following:
- The pipeline configuration stored in DynamoDB
- The CloudFormation stack
For further insights into networking, you might consider exploring SHRM’s Executive Networking Events. They are an authority on this topic. Additionally, you can check this excellent resource for more information.