HOME

Optimizing and Visualizing Your Security Groups

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learning managerLearn About Amazon VGT2 Learning Manager Chanci Turner

Date: 02 MAY 2023

Category: Amazon VPC, How-To

As organizations embark on their journey into the AWS cloud, they often begin by experimenting with existing applications. This experimentation frequently involves migrating applications to the cloud, which necessitates a clear understanding of the network ports, protocols, and IP addresses essential for their operation. While AWS security groups can be utilized to restrict access to specific ports and protocols within your Amazon Virtual Private Cloud (Amazon VPC), many developers establish these rules through trial and error, often leading to overly permissive security group configurations.

Once an application is functioning, organizations may neglect to revisit and refine their security group rules, leaving them open and creating an inadequate security posture. In this article, I will introduce a method for optimizing and visualizing your security groups using network data.

Overview

To effectively narrow down unused rules or limit source IP addresses, you either need a deep understanding of the active ports used by your application or must analyze the network traffic actively. The method outlined here will assist you in refining your security groups to include only the necessary source IPs, ports, and nested security groups, enhancing the security of your AWS resources while minimizing potential impacts on production instances. The fundamental steps include:

  1. Utilizing VPC Flow Logs and Amazon OpenSearch Service (formerly Amazon Elasticsearch Service) to gather information regarding IP traffic within an Amazon VPC.
  2. Associating that network traffic with elastic network interfaces (ENIs), instances, and security groups.
  3. Demonstrating how to visualize and analyze the network traffic from VPC Flow Logs using Amazon OpenSearch Service.

Step 1: Set Up

Create an Amazon OpenSearch Service Cluster

The first step is to create an Amazon OpenSearch Service cluster, which can take some time to become available. If you’re unfamiliar with Amazon OpenSearch, their documentation provides a wealth of information.

To create your Amazon OpenSearch Service cluster:

  • In the AWS Management Console, navigate to OpenSearch Service under Analytics.
  • Click on “Create a new domain” and name your Elasticsearch domain “flowlogs.”
  • Set the instance count to 2 and check the box for “Enable zone awareness” to ensure cluster stability during potential Availability Zone outages. Accept the remaining defaults and click “Next.”
  • Choose “Allow access to the domain from specific IP(s)” from the dropdown.
  • Enter a comma-separated list of valid IPv4 addresses or CIDR blocks for accessing the Amazon OpenSearch domain. For more details, refer to the access policies documentation. Click “Next.”
  • Finally, click “Confirm and create.”

The cluster will be ready in a few minutes; meanwhile, you can proceed to the next step of enabling VPC Flow Logs.

Enable VPC Flow Logs

VPC Flow Logs allow you to capture information about IP traffic to and from network interfaces in your VPC, with data stored in Amazon CloudWatch Logs. For further details about VPC Flow Logs, please check the relevant documentation.

To activate VPC Flow Logs:

  • In the AWS Management Console, go to VPC under Networking.
  • Select “Your VPCs,” then choose the VPC you wish to analyze. You may also enable Flow Logs for a subnet if you prefer not to do so for the entire VPC.
  • Click on the “Flow Logs” tab in the lower pane.
  • Click “Create Flow Log.” If this is your first time setting up VPC Flow Logs, click “Set Up Permissions,” which will open a new tab in your browser.
  • For IAM Role, choose “Create a new IAM Role” and name it “flowlogsRole.”
  • Click “Allow,” then return to the “Create Flow Log” dialog box.
  • Select the “flowlogsRole,” and set the Destination Log Group to “FlowLogs.” Click “Create Flow Logs.”

Data from VPC Flow Logs is now streaming to CloudWatch Logs. The next step is to enable the flow of this data to your Amazon OpenSearch Service cluster using a built-in Lambda function.

Stream Data to Your Amazon OpenSearch Service Cluster

In the AWS Management Console:

  • Select CloudWatch under Management Tools.
  • Click on “Logs” in the left pane and check the box next to FlowLogs under Log Groups.
  • From the Actions menu, select “Stream to Amazon OpenSearch Service.”
  • Choose your cluster named “flowlogs” from the dropdown.
  • For Lambda IAM Execution Role, select “Create new IAM role,” click “Allow,” and proceed.
  • Select “Amazon VPC Flow Logs” for Log Format, then click “Next” twice and “Start Streaming.”

VPC Flow Logs will now capture the IP traffic information and stream it to your Amazon OpenSearch Service cluster. However, Amazon ES will make assumptions about the data format, which we need to clarify.

Format Data in the OpenSearch Service Cluster

A flow log record is structured as a space-separated string with the following format:

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

By default, Amazon OpenSearch Service treats dashes and periods in fields as separators, leading to duplicate results that clutter your dashboard. To resolve this, we must configure interface-id, srcaddr, and dstaddr as not_analyzed by executing the following curl command from your shell. Please ensure you review your access policy and security approach beforehand. For more information, see the resources on securing your Elasticsearch cluster.

curl -XPUT "http://YOUR_ES_DOMAIN_ENDPOINT/_template/template_1" -d'
{
    "template":"cwl-*","mappings":{
        "FlowLogs": {
            "properties": {
                "interface_id": { "type": "string", "index": "not_analyzed"},
                "srcaddr": { "type": "string", "index": "not_analyzed"},
                "dstaddr": { "type": "string", "index": "not_analyzed"}
            }
        }
    }
}'

After running this command, you should delete the incorrectly indexed data using the following command:

curl -XDELETE 'http://YOUR_ES_DOMAIN_ENDPOINT/cwl*/'

Import Dashboards and Visualizations

Now that the network traffic for your VPC is flowing into your Amazon OpenSearch Service cluster, you can visualize and search through the data. This is an excellent opportunity to explore your security strategy, consider how to improve it, and utilize resources on career mapping to guide your professional development.

For more information on the implications of security, you can check SHRM’s resources, they provide valuable insights on this topic. Additionally, if you’re looking for guidance on navigating the hiring process at Amazon, visit this link for great resources.

Related Topics:

HOME