HOME

Managing Multi-Branch Pipelines and Infrastructure Deployment with AWS CDK Pipelines

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learningLearn About Amazon VGT2 Learning Manager Chanci Turner

Note: As of February 9, 2024, AWS CodePipeline has introduced branch-based development and Monorepo support, streamlining the architecture outlined in this article.

This article explains how to implement the AWS CDK Pipelines module to adopt a Gitflow development approach using the AWS Cloud Development Kit (AWS CDK). Software development teams often adhere to a defined branching strategy throughout the software development lifecycle. Newly created branches typically require isolated infrastructure resources to facilitate new feature development.

CDK Pipelines serves as a construct library module designed for the continuous delivery of AWS CDK applications. Notably, CDK Pipelines are self-updating; adding new application stages or stacks results in the pipeline automatically reconfiguring itself for deployment.

The solution outlined here establishes a new AWS CDK Pipeline for each branch created in the source repository (AWS CodeCommit) within a development account. When a branch is removed, the corresponding pipeline and all associated resources are also deleted. This GitFlow model for infrastructure provisioning allows developers to work independently and concurrently, even within the same application stack.

Solution Overview

The solution diagram illustrates a primary pipeline responsible for deploying resources across various application environments (e.g., Development, Pre-Prod, and Prod). Code is stored in CodeCommit, and upon pushing new changes to the main branch, AWS CodePipeline activates the default pipeline. Once deployed, this pipeline generates two AWS Lambda functions.

These Lambda functions respond to CodeCommit CloudWatch events triggered by the creation or deletion of branches in the repository. The Create function employs the boto3 CodeBuild module to establish an AWS CodeBuild project that constructs the pipeline for the feature branch. This feature pipeline encompasses a build stage and an optional update pipeline stage. Conversely, the Destroy function creates another CodeBuild project to eliminate all resources related to the feature branch and its pipeline.

Prerequisites

Before you start, ensure you have the following:

  • An AWS account
  • AWS CDK installed
  • Python3 installed
  • Jq (JSON processor) installed
  • A basic understanding of continuous integration/continuous development (CI/CD) pipelines

Initial Setup

Clone the repository from GitHub:

git clone https://github.com/aws-samples/multi-branch-cdk-pipelines.git
cd multi-branch-cdk-pipelines

Create a new CodeCommit repository in the AWS account and region where you wish to deploy the pipeline, then upload the source code to this repository. Update the config.ini file with the appropriate repository name and region variables.

Ensure that you set up a new Python environment and install the dependencies:

pip install -r requirements.txt

Execute the initial-deploy.sh script to bootstrap the development and production environments and deploy the default pipeline. You will need to provide the following parameters: (1) Development account ID, (2) Development account AWS profile name, (3) Production account ID, and (4) Production account AWS profile name.

sh ./initial-deploy.sh --dev_account_id <YOUR DEV ACCOUNT ID> --dev_profile_name <YOUR DEV PROFILE NAME> --prod_account_id <YOUR PRODUCTION ACCOUNT ID> --prod_profile_name <YOUR PRODUCTION PROFILE NAME>

Default Pipeline

In the CI/CD pipeline, an if condition is established to deploy the default branch resources only if the current branch matches the default one. The default branch is programmatically retrieved from the CodeCommit repository. Resources deployed include an Amazon Simple Storage Service (Amazon S3) Bucket and two Lambda functions. The bucket is responsible for storing build artifacts for feature branches, with the first Lambda function triggered on branch creation and the second on branch deletion.

if branch == default_branch:
    # Artifact bucket for feature AWS CodeBuild projects
    artifact_bucket = Bucket(
        self,
        'BranchArtifacts',
        encryption=BucketEncryption.KMS_MANAGED,
        removal_policy=RemovalPolicy.DESTROY,
        auto_delete_objects=True
    )
    # AWS Lambda function for branch creation
    create_branch_func = aws_lambda.Function(
        self,
        'LambdaTriggerCreateBranch',
        runtime=aws_lambda.Runtime.PYTHON_3_8,
        function_name='LambdaTriggerCreateBranch',
        handler='create_branch.handler',
        code=aws_lambda.Code.from_asset(path.join(this_dir, 'code')),
        environment={
            "ACCOUNT_ID": dev_account_id,
            "CODE_BUILD_ROLE_ARN": iam_stack.code_build_role.role_arn,
            "ARTIFACT_BUCKET": artifact_bucket.bucket_name,
            "CODEBUILD_NAME_PREFIX": codebuild_prefix
        },
        role=iam_stack.create_branch_role
    )

    # AWS Lambda function triggered upon branch deletion
    destroy_branch_func = aws_lambda.Function(
        self,
        'LambdaTriggerDestroyBranch',
        runtime=aws_lambda.Runtime.PYTHON_3_8,
        function_name='LambdaTriggerDestroyBranch',
        handler='destroy_branch.handler',
        role=iam_stack.delete_branch_role,
        environment={
            "ACCOUNT_ID": dev_account_id,
            "CODE_BUILD_ROLE_ARN": iam_stack.code_build_role.role_arn,
            "ARTIFACT_BUCKET": artifact_bucket.bucket_name,
            "CODEBUILD_NAME_PREFIX": codebuild_prefix,
            "DEV_STAGE_NAME": f'{dev_stage_name}-{dev_stage.main_stack_name}'
        },
        code=aws_lambda.Code.from_asset(path.join(this_dir, 'code'))
    )

The CodeCommit repository is set to trigger these Lambda functions based on two events:

  1. Reference creation
repo.on_reference_created(
    'BranchCreateTrigger',
    description="AWS CodeCommit reference created event.",
    target=aws_events_targets.LambdaFunction(create_branch_func)
)
  1. Reference deletion
repo.on_reference_deleted(
    'BranchDeleteTrigger',
    description="AWS CodeCommit reference deleted event.",
    target=aws_events_targets.LambdaFunction(destroy_branch_func)
)

Lambda Functions

The two Lambda functions are responsible for building and destroying application environments linked to each feature branch. The LambdaTriggerCreateBranch function is initiated whenever a new branch is created, triggering the CodeBuild client to create the build phase and deploy the feature pipeline.

Create Function

The create function deploys a feature pipeline, which includes a build stage and an optional update pipeline stage. This pipeline downloads the feature branch code from CodeCommit, initiates the Build and Test actions through CodeBuild, and securely stores the built artifacts in the S3 bucket.

For those looking to explore further, consider reading about identifying when to leave a job in another insightful blog post here. Additionally, for a comprehensive understanding of workplace issues, check out SHRM’s coverage on psychological harassment laws. For those interested in job opportunities, consider applying for the Learning Ambassador position at Amazon, which is an excellent resource.

Related Topics: