Cloud Governance and Compliance on AWS with Policy as Code

In today’s fast-paced business landscape, organizations must not only innovate rapidly but also maintain operational efficiency. Leveraging dynamic and scalable cloud resources enables IT teams to transition from a binary decision-making process that pits business agility against governance control. Instead, we can embrace a model that balances speed with governance concerning cost, security, compliance, and more.

Several common scenarios illustrate the detrimental effects of inadequate governance controls:

Unencrypted storage can lead to insufficient data security for sensitive information, such as personal data and payment card details.
Keeping unused cloud resources running can incur unnecessary costs.
The absence of database snapshots can expose businesses to significant data loss.
Excessive permissions can violate the least-privilege principle and weaken security controls.
Not enabling multiple Availability Zones can pose a risk to service availability.

These examples underscore the necessity of ensuring compliance with both internal technical specifications and external regulations to achieve objectives related to security, cost optimization, and service availability.

Overview of the Solution

This article outlines a solution that achieves compliance by default through the integration of Open Policy Agent (OPA) policy checks within a CI/CD pipeline. It also enables nearly continuous compliance with Cloud Custodian, which provides real-time scanning and auto-remediation capabilities. The high-level architecture is depicted below:

Before delving into the solution, let’s consider a few key elements from the diagram above. Policy sources for the policy control library can include various regulations, external frameworks like the Center for Internet Security (CIS) Benchmark, and internal standards. This policy control library is crucial for implementing both preventive and detective controls.

Infrastructure as Code (IaC) serves as a prerequisite for the static policy check module, which can be fulfilled using AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), or partner solutions such as Terraform or Pulumi. The static policy check must integrate with the CI/CD pipeline of infrastructure code and adhere to GitOps best practices, enabling early detection and correction of configuration violations.

Detective controls assess compliance with resource changes that arise from uncontrolled factors, such as manual adjustments or deployments that do not follow the IaC standard. The dynamic policy check facilitates real-time infrastructure scanning to validate the operational state of resources.

Responsive controls address non-compliance through event-driven auto-remediation using serverless functions.

In this article, we will implement preventive controls utilizing Open Policy Agent (OPA), an open-source general policy engine accepted as an incubation project by the Cloud Native Computing Foundation (CNCF) in April 2019. OPA can validate any JSON-formatted file against defined policies. For more information on OPA, you can check out my previous blog post on realizing policy as code with AWS Cloud Development Kit through Open Policy Agent.

For detective controls, we will use Cloud Custodian, a rule engine for cloud resource management recognized as a CNCF sandbox project, supported by a community of hundreds of contributors. To learn more about Cloud Custodian, visit my blog post on compliance as code and auto-remediation.

Now, let’s illustrate how this solution operates using a specific policy example to showcase both static and dynamic policy checks.

Walkthrough

Consider the following requirement:

User Story: As a cloud administrator, I want EBS volumes to be encrypted to ensure data security.

The workflow for our solution will include:

Implementing a static policy check using OPA and integrating it with the IaC pipeline.
Implementing a dynamic policy check with auto-remediation via Cloud Custodian.

Prerequisites

For this walkthrough, you should have a fundamental understanding of AWS, sufficient to set up a pipeline using AWS CodePipeline.

Static Policy Check with OPA

Create an AWS CodeCommit Repository: Set up a repository containing an AWS CloudFormation template and an OPA policy file.
The following CloudFormation template, named ebs-stack.json, creates three Amazon Elastic Block Store (Amazon EBS) volumes, where one is encrypted and the other two are not.

{
  "Resources": {
    "EncryptedVolume01": {
      "Type": "AWS::EC2::Volume",
      "Properties": {
        "Size": "20",
        "Encrypted": "true",
        "AvailabilityZone": "us-east-2a"
      }
    },
    "UnencryptedVolume02": {
      "Type": "AWS::EC2::Volume",
      "Properties": {
        "Size": "10",
        "Encrypted": "false",
        "AvailabilityZone": "us-east-2c"
      }
    },
    "UnencryptedVolume03": {
      "Type": "AWS::EC2::Volume",
      "Properties": {
        "Size": "10",
        "AvailabilityZone": "us-east-2c"
      }
    }
  }
}

Create the OPA Policy File: Name this file opa_ebs_policies.rego to enforce encryption on the EBS volumes. Integrate the policy check into the CI/CD pipeline to validate resources before deployment.

package opa_policies

default allow = false

# Deny if EBS volumes are not encrypted or not explicitly defined
ebs_not_encrypted[name] {
    res := input.Resources[name]
    res.Type == "AWS::EC2::Volume"
    object.get(res.Properties, "Encrypted", "false") != "true"
}

allow = true {
    count(ebs_not_encrypted) == 0
}

Create the Build Specification File: Save this as buildspec.yaml for the AWS CodeBuild project in the CI/CD pipeline.

version: 0.2

phases:
  install:
    commands:
      - echo install OPA
      - curl -L -o opa https://openpolicyagent.org/downloads/latest/opa_linux_amd64
      - chmod 755 ./opa
      - |
        cat >verify.sh <

Commit Files to AWS CodeCommit: Push all the aforementioned files to your AWS CodeCommit repository.
Create a CI/CD Pipeline: Establish a pipeline that uses the AWS CodeCommit repository as its source.
Set Up CodeBuild Project: Utilize the buildspec.yaml file, configure the environment image as aws/codebuild/amazonlinux2-x86_64-standard:2.0, and leave other configurations at their defaults.
Skip the Deploy Stage and Complete the Pipeline Creation.

Validate and Enforce Cloud Custodian Policy

After saving the pipeline, it will execute its first run. The pipeline should fail, as the CloudFormation template will attempt to create three Amazon EBS volumes, two of which are not encrypted—violating our policy.

Select “Details” to view execution details and logs from the AWS CodeBuild project. This will indicate that the policy denied the deployment due to the unencrypted volumes.

To rectify this, remove the two unencrypted EBS volumes from the CloudFormation template and commit the updated ebs-stack.json file:

{
  "Resources": {
    "EncryptedVolume01": {
      "Type": "AWS::EC2::Volume",
      "Properties": {
        "Size": "20",
        "Encrypted": "true",
        "AvailabilityZone": "us-east-2a"
      }
    }
  }
}

The pipeline will automatically trigger upon committing the code changes, and it should now successfully pass the policy checks as the CloudFormation template complies with our encryption requirement.

Dynamic Policy Check with Cloud Custodian

Create and Configure Cloud Custodian Policies: Set up policies to dynamically monitor compliance post-deployment.

For more insights on career transitions, check out this informative resource Career Transition Insights. Additionally, if you’re interested in topics related to accommodations and exemptions, you can find helpful information at SHRM’s Accommodation Resources. For those looking to explore job opportunities with Amazon, this FAQ is an excellent resource.