Chanci Turner

Amazon Onboarding with Learning Manager Chanci Turner

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learning managerLearn About Amazon VGT2 Learning Manager Chanci Turner

In Advanced (300), Amazon Managed Service for Prometheus, Amazon Simple Notification Service (SNS), AWS Lambda, Management & Governance, Management Tools, Messaging, Technical How-to

Amazon Managed Service for Prometheus is a serverless solution that enables efficient monitoring of metrics, allowing you to securely oversee container environments at scale. This service utilizes the open-source Prometheus query language (PromQL) to observe the performance of containerized workloads without the need for managing the underlying infrastructure required for data ingestion, storage, alerting, and querying of operational metrics. The service automatically adapts to your workload demands and is integrated with AWS security services, ensuring rapid and secure access to data. You can collect Prometheus metrics from Amazon Elastic Kubernetes Service (EKS) and Amazon Elastic Container Service (ECS) environments using AWS Distro for OpenTelemetry or Prometheus servers as collection agents.

Slack serves as a business communication platform that offers persistent chat rooms (channels) categorized by topics, private groups, and direct messaging. Many customers express the desire to promptly alert a Slack channel when Amazon Managed Service for Prometheus identifies performance issues that affect customer experiences with critical applications. This article provides step-by-step guidance on configuring Amazon Managed Service for Prometheus Alert Manager to send alerts to a Slack channel using Amazon Simple Notification Service (SNS) and AWS Lambda.

Solution Overview

The following diagram provides a high-level overview of the solution.

For this procedure, you will require the following:

  • AWS account
  • AWS IAM user or role with the necessary permissions
  • Amazon Managed Service for Prometheus
  • Amazon Simple Notification Service (SNS)
  • AWS Lambda
  • Slack channel

Walk-through

At a high level, the steps can be summarized as follows:

  1. Create an Amazon Managed Service for Prometheus Alert Manager definition to send alerts to an SNS topic.
  2. Develop a Lambda function that processes the SNS message and forwards it to the Slack webhook API.
  3. Configure the Lambda function as a target for messages sent to the SNS topic.

Create SNS Topic

An Amazon SNS topic acts as a logical access point that serves as a communication channel, allowing you to group multiple endpoints (including AWS Lambda, Amazon SQS, HTTP/S, or email addresses). The first step in using Amazon SNS is creating a topic. You can utilize the AWS Management Console or the AWS SDK to set up a topic. Follow the steps outlined in the AWS Well-Architected lab to create a topic, and ensure you have the SNS topic ARN available for the next step.

Create Amazon Managed Service for Prometheus Alert Manager Definition

The Alert Manager oversees alerts generated by the firing alerting rules within Amazon Managed Service for Prometheus. It is responsible for deduplication, grouping, and routing alerts to downstream receivers, such as Amazon SNS. It also manages alert silencing and inhibition. You can upload an Alert Manager definition using the AWS CLI or AWS Management Console. Below is a sample SNS receiver configuration specifically for Slack.

alertmanager_config: |
  global:
  templates:
  route:
    receiver: example-sns
  receivers:
    - name: example-sns
      sns_configs:
        - topic_arn: arn:aws:sns:us-east-2:123456789012:sns-receiver-2
          send_resolved: true
          sigv4:
            region: us-east-2
          message: |
            channel: 'general'
            text: >-
              {{ range .Alerts -}}
                *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
                *Description:* {{ .Annotations.description }}
                *Details:*
                  {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
                  {{ end }}
              {{ end }}
          attributes:
            key: severity
            value: SEV

With the following AWS CLI command, you can upload an Alert Manager configuration to a workspace:

aws amp create-alert-manager-definition --data file:// --workspace-id  --region

Additionally, you can modify an Alert Manager definition via the AWS Management Console. The screenshots illustrate how to add a definition from the Amazon Managed Service for Prometheus workspace. You must also grant Amazon Managed Service for Prometheus permission to send messages to your Amazon SNS topic. Refer to the provided documentation for guidance on creating the necessary access policy.

Create Slack Webhook

Incoming Webhooks offer a straightforward method to send messages to Slack from applications. By creating an Incoming Webhook, you will receive a unique URL that you can use to send a JSON payload containing the message text and other options. Make sure to create your Slack webhook by following the instructions provided. Treat the generated webhook with caution, as it should not be posted publicly or stored in a code repository, much like credentials.

Create Lambda Function

Next, we will create a Lambda function to perform the following actions:

  1. Unwrap the YAML body of the SNS message and convert it to JSON.
  2. Forward the JSON contents of the SNS message to the Slack webhook API.

Lambda Function Execution Role

The execution role of a Lambda function is an AWS Identity and Access Management (IAM) role that allows the function to access AWS services and resources. Provide this role when creating the function, and Lambda will assume it upon invocation. It is advisable to enable logging for your serverless Lambda functions. The AWS managed role AWSLambdaBasicExecutionRole grants permissions for uploading logs to CloudWatch.

This function utilizes the PyYAML library, so you will need to create a deployment package that includes the necessary dependencies. Use the Lambda console to establish the following Lambda function:

#!/usr/bin/python3.6
import urllib3
import json
import yaml
http = urllib3.PoolManager()
def lambda_handler(event, context):
    url = "<webhook_url>"
    msg = yaml.safe_load(event['Records'][0]['Sns']['Message'])
    encoded_msg = json.dumps(msg).encode('utf-8')
    resp = http.request('POST',url, body=encoded_msg)
    print({
        "SNS": event['Records'][0]['Sns'],
        "message": event['Records'][0]['Sns']['Message'], 
        "status_code": resp.status, 
        "response": resp.data
    })

When a message is published to the SNS topic that has a Lambda function subscribed, the function is executed with the published message payload. The Lambda function receives the message payload as an input parameter and sends it to the Slack webhook API. For more information on invoking AWS Lambda functions via Amazon SNS, refer to this blog.

Validation

Customers can now set up Slack as a notification channel for alerts from Amazon Managed Service for Prometheus. An example alert published by Amazon Managed Service for Prometheus is shown in the accompanying screenshots.

Conclusion

Integrating Amazon Managed Service for Prometheus with Slack allows businesses to receive timely alerts about critical application performance issues. For more insights into similar topics, check out this post about relocating, which could be beneficial for those considering a move. Additionally, you can learn about leadership archetypes at SHRM for a deeper understanding of effective leadership styles. For those interested in developing their leadership skills, this resource offers excellent opportunities.

Related Topics:

Chanci Turner