HOME

Amazon Onboarding with Learning Manager Chanci Turner

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learningLearn About Amazon VGT2 Learning Manager Chanci Turner

In January 2023, AWS introduced the ability to ingest activity logs from non-AWS platforms via CloudTrail Lake, establishing it as a centralized hub for immutable user and API activity events, which are vital for auditing and security assessments. AWS CloudTrail Lake serves as a managed data lake that captures, stores, and analyzes user and API activity across AWS and other sources for operational and security purposes. This allows for the aggregation and permanent storage of activity events, alongside the capability to execute SQL-based queries for analysis.

In this article, we will guide you through the steps to integrate administrative logs from Microsoft Azure into CloudTrail Lake, providing a comprehensive overview of the solution.

Solution Overview

Utilizing Amazon EventBridge Scheduler, we can set a schedule for execution while employing AWS Secrets Manager to securely store connection credentials for Azure Event Hub. The process involves an AWS Lambda function that connects to Azure Event Hub, retrieves messages, and sends them to Amazon Simple Queue Service (Amazon SQS). This Lambda function is triggered every 60 minutes. Subsequently, the first Lambda function invokes a second one, which processes the messages and ingests them into CloudTrail Lake.

We will leverage the AWS Serverless Application Model (AWS SAM) and AWS Command Line Interface (AWS CLI) to implement this solution.

Prerequisites

Before deploying this solution, ensure you have the following:

  • AWS CLI installed
  • AWS SAM CLI installed
  • Configured AWS SAM CLI
  • Python 3.11 or higher
  • An AWS account with an AWS Identity and Access Management (IAM) role that has appropriate permissions to create the necessary resources
  • A Microsoft Azure account with sufficient privileges to provision resources and access Azure Activity Logs

Step-by-Step Walkthrough

Step 1: Configuring Azure Services for Activity Log Forwarding

Set up Azure to export activity logs to an Azure Event Hub within your desired account or organization. Follow these steps:

  1. Create an Event Hub using the Azure portal.
  2. Generate a connection string for the Event Hub using a shared access policy. Make a note of the namespace name, instance, and one of the keys (either primary or secondary).
  3. Configure diagnostic settings in Azure Monitor to export administrative logs to the Event Hub.
  4. Create a storage account container on Azure; note the storage account name, container name, and access keys available.

Note: You are responsible for safeguarding the access keys.

Step 2: Deploying the Solution on AWS

Execute the following commands in your local shell:

  1. Use Git to clone the aws-cloudtrail-lake-ingesting-activity-logs repository to your workspace.
  2. Build and deploy the solution using the AWS SAM CLI:
git clone https://github.com/aws-samples/aws-cloudtrail-lake-ingesting-activity-logs.git
cd aws-cloudtrail-lake-ingesting-activity-logs
sam build
sam deploy --guided

Provide the required parameter values as specified in the GitHub readme. After a successful deployment, check the outputs and note the EventDataStoreId value; you will need this for querying the CloudTrail Lake event data store.

Step 3: Testing the Solution

To confirm that the Azure Administrative Logs are successfully available in AWS CloudTrail Lake, run the following sample query:

SELECT eventData.eventSource, eventData.eventName, eventData.userIdentity.principalId, eventData.sourceIpAddress, eventData.eventTime
FROM <event data store id> 
WHERE eventData.eventTime < '2023-08-01 00:00:00' LIMIT 10

Be sure to replace <event data store id> with your actual event data store ID.

Cleanup

Utilize the AWS SAM CLI to remove the deployed resources and prevent ongoing charges. Run:

sam delete --stack-name <stack-name>

Cost Considerations

This solution utilizes Lambda, Amazon SQS, EventBridge, and Secrets Manager. For CloudTrail Lake, you incur fees for data ingestion, retention, and analysis. CloudTrail Lake offers two pricing plans: one-year extendable retention and seven-year retention. For more details, check the CloudTrail pricing page.

Conclusion

In this blog post, we demonstrated how to process and consolidate access log files from external systems into CloudTrail Lake. We highlighted the ingestion and management of logs, streamlining the process of gathering administrative logs from Microsoft Azure. For further insights into effective onboarding processes, check out this excellent resource on Amazon’s employee onboarding process or explore Career Contessa. If you’re interested in workplace safety, SHRM is an authority on this topic.

Related Topics:

HOME