Chanci Turner

Transitioning from a Custom Deployment Agent to AWS CodeDeploy and AWS CodePipeline

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learningLearn About Amazon VGT2 Learning Manager Chanci Turner

I’m Alex Harper, a developer at Woot who focuses on deployment processes and enhancing developer experiences. Woot is renowned as the original daily deals platform, established in 2004 and acquired by Amazon in 2010.

Recently, we transitioned our web front-end deployments from Troop, a proprietary deployment agent we created, to AWS CodeDeploy and AWS CodePipeline. This shift required launching a new fleet of customer-facing EC2 web servers equipped with the CodeDeploy agent, along with managing our CodeDeploy and CodePipeline resources through AWS CloudFormation. Notably, after completing the migration, we experienced a nearly 50% decrease in HTTP 500 errors during deployments.

In this blog post, I will cover:

  • The reasons behind our choice of AWS deployment tools.
  • An architectural overview of Woot’s systems.
  • A summary of the migration project.
  • The outcomes of our migration efforts.

The Outdated Approach

We aimed to replace our in-house deployment system with a solution that could be automated and one that we wouldn’t have to maintain. Managing our build system was already challenging enough; we didn’t want to juggle additional infrastructure for our deployment pipeline.

In essence, we were looking for a cloud service. Since all our infrastructure resides in AWS, CodeDeploy was a logical choice to take over our deployment agent. CodePipeline serves as the automation orchestrator, guiding CodeDeploy on what to deploy and when.

Architectural Overview

Here’s a glimpse of Woot’s web front-end architecture:

Woot architecture overview

Project Summary

Our migration project encompassed moving five web front ends, collectively handling around 12 million requests daily, to CodeDeploy and CodePipeline while ensuring the site remained operational for our users.

The steps we undertook included:

  1. Developing new deployment scripts.
  2. Launching a new fleet of EC2 web servers with CodeDeploy capabilities.
  3. Creating a deployment pipeline for our CloudFormation-defined CodeDeploy and CodePipeline configuration.
  4. Gradually introducing our new fleet to live traffic. Hello, customers!

Deployment Scripts

Our previous deployment system did not stop or start our web servers. Instead, it attempted to interchange build artifacts while the server was live—a risky approach.

We crafted deployment scripts in PowerShell that CodeDeploy executes to manage our IIS web servers. These scripts work alongside the Elastic Load Balancing (ELB) functionality in CodeDeploy, as we certainly didn’t want to disrupt the web server during active customer traffic.

New Fleet

Since our fleet runs on Amazon EC2, we developed an Amazon Machine Image (AMI) for our web fleet with the CodeDeploy agent pre-installed. From a fleet perspective, this was largely all that was needed. With the agent in place, CodeDeploy could utilize our deployment scripts to launch our web projects.

AWS CloudFormation Deployment Pipeline

To establish a deployment pipeline and multiple CodeDeploy configurations (including a CodeDeploy application and at least one deployment group) for each web project, we opted to use AWS CloudFormation to version this setup. Our build system, TeamCity, can access our version control system and write to Amazon S3. We created a straightforward build in TeamCity to upload an AWS CloudFormation template to S3, triggering a pipeline that deploys to AWS CloudFormation. This process generates the CodePipeline and CodeDeploy resources, allowing us to conduct code reviews on our infrastructure modifications. More eyes lead to greater safety! We can also track infrastructure changes over time, just like we do with code changes.

Live Traffic Introduction

Our web fleets operate behind Classic Load Balancers. By utilizing CodeDeploy, we can leverage its new ELB features. For instance, CodeDeploy can prevent internet traffic from being directed to an instance during the deployment. Once the deployment to that instance concludes, it becomes available for traffic.

We launched new hosts with the CodeDeploy agent and deployed to them without ELB support initially. We then gradually, and manually, integrated them into our fleet while monitoring the metrics. After fully incorporating the new machines, we slowly phased out the old ones from the load balancer, achieving a completely CodeDeploy-supported fleet without any downtime.

One interesting detail: When we had two-thirds of the new fleet in our load balancer, we initiated a CodeDeploy deployment to the fleet, this time with ELB support activated. This allowed CodeDeploy to seamlessly introduce the remaining machines into the load balancer, coexisting with the old fleet, and reducing the number of actions required.

AWS CloudFormation Example

Here’s a simplified version of the AWS CloudFormation template used to manage the AWS configuration for one of our web projects. It’s deployed within a pipeline, similar to how we deploy our web projects.

Parameters:
  CodePipelineBucket:
    Type: String
  CodePipelineRole:
    Type: String
  CodeDeployRole:
    Type: String
  CodeDeployBucket:
    Type: String
Resources:

  ### Woot.Example deployment configuration ###
  ExampleDeploymentConfig:
    Type: 'AWS::CodeDeploy::DeploymentConfig'
    Properties:
      MinimumHealthyHosts:
        Type: FLEET_PERCENT
        Value: '66' # Ensuring that 2/3 of the fleet remains healthy at all times

  #Woot.Example CodeDeploy application
  WootExampleApplication:
    Type: "AWS::CodeDeploy::Application"
    Properties:
      ApplicationName: "Woot.Example"

  #Woot.Example CodeDeploy deployment groups
  WootExampleDeploymentGroup:
    DependsOn: "WootExampleApplication"
    Type: "AWS::CodeDeploy::DeploymentGroup"
    Properties:
      ApplicationName: "Woot.Example"
      DeploymentConfigName: !Ref "ExampleDeploymentConfig" 
      DeploymentGroupName: "Woot.Example.Main"
      AutoRollbackConfiguration:
        Enabled: true
        Events:
          - DEPLOYMENT_FAILURE 
          - DEPLOYMENT_STOP_ON_REQUEST 
      LoadBalancerInfo:
        ElbInfoList:
          - Name: "WootExampleInternal" 
      DeploymentStyle:
        DeploymentOption: "WITH_TRAFFIC_CONTROL" 
      Ec2TagFilters:
        - 
          Key: "Name"
          Value: "exampleweb*" 
          Type: "KEY_AND_VALUE"
      ServiceRoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/${CodeDeployRole}" 

  #Woot.Example CodePipeline
  WootExampleDeploymentPipeline:
    DependsOn: "WootExampleDeploymentGroup"
    Type: "AWS::CodePipeline::Pipeline"
    Properties:
      RoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/${CodePipelineRole}" 
      Name: "Woot.Example" 
      ArtifactStore:
        Type: S3
        Location: !Ref "CodePipelineBucket"

For further reading on onboarding strategies, you can check out this engaging article on augmented reality at Career Contessa. Additionally, if you’re interested in laws surrounding employment, especially about hiring minors, SHRM is an authoritative source. For an excellent resource on onboarding experiences, visit Reddit.

Related Topics:

Chanci Turner