Automating Multi-AZ High Availability for WebLogic Administration Server with DNS: Part 2

on 16 OCT 2024

in Amazon EC2, Architecture, AWS Lambda, Partner solutions

In the first segment of this series, we explored how to utilize a floating virtual IP (VIP) to establish hands-free high availability (HA) for the WebLogic Admin Server. In this second installment, we will implement a more effective approach through Domain Name System (DNS) resolution.

Utilizing DNS for WebLogic Admin Server Address Resolution

Let’s examine the reference architecture for deploying WebLogic on AWS, as illustrated in the accompanying diagram.

This solution is divided into two key components:

Configuring the environment to leverage DNS for locating the admin server.
Establishing a method for automatically updating the DNS entry when the admin server is initiated.

Environment Configuration

A WebLogic domain is set up within the private subnets of a Virtual Private Cloud (VPC). The admin server operates on its own Amazon Elastic Compute Cloud (Amazon EC2) instance situated in one of these private subnets. In this scenario, the admin server is linked to the private IP address of the EC2 host, which is also associated with a hostname/DNS record configured in Amazon Route 53.

We deploy WebLogic using a multi-Availability Zone (multi-AZ) active-active architecture. For simplicity, we will consider a single WebLogic domain and one admin server. To fulfill this requirement, we will:

Create an EC2 launch template for the admin server.
Associate this launch template with an Amazon EC2 Auto Scaling group called wlsadmin-asg, with a minimum, maximum, and desired capacity of 1. Note that this group name is important for later steps.

The Auto Scaling group monitors the health of EC2 instances and Availability Zones, launching a new instance in a different AZ if the current one becomes unavailable. To facilitate access, we set up two route tables: one for private subnets and another for public subnets.

Next, we utilize the Amazon Route 53 DNS service to abstract the IPv4 address of the WebLogic admin server:

We create a private hosted zone in Amazon Route 53; for our example, we use example.com.
An A record is created for the admin server, pointing to the IP address of the EC2 instance that hosts the admin server. We set the TTL to 60 seconds to ensure that the managed servers’ DNS records are propagated before the admin server completes its startup.
Make a note of the hosted zone ID, as it will be needed later for creating an IAM role with permissions to update the DNS A record, as well as for an environment variable for an AWS Lambda function that will handle the update.

We then modify the WebLogic domain configuration to set the WebLogic Admin server listen address to the chosen DNS name. For this example, we modify the line in the WebLogic Admin server configuration to <listen-address>wlsadmin.example.com</listen-address> in the WebLogic domain configuration file located at $DOMAIN_HOME/config/config.xml.

Automatic DNS A Record Updates upon Admin Server Launch

In traditional on-premises environments, updating a DNS record as part of a server’s lifecycle is often frowned upon due to the complexities of inter-team operations. However, in the cloud, we possess the tools and security models that facilitate such processes.

Several strategies exist for this automation, and it’s crucial to assess the prototypes we developed and why certain options were rejected before outlining our recommended approach:

Rejected Option 1 – Simple: The user data script makes an API call to update the A record (with an appropriate IAM instance policy). This option was dismissed because if a server is compromised, it could manipulate the A record for malicious purposes.
Rejected Option 2 – Better: The user data script invokes a Lambda function to modify the A record, including checks to prevent misuse, such as preventing public address assignment. However, this still requires granting permissions for the instance to invoke the Lambda function and establishing the logic to validate the IP address.
Accepted Option 3 – Best: We do not provide the EC2 instance with additional permissions to update the DNS A Record. Instead, we depend on the Auto Scaling group’s event lifecycle as shown in the diagram.

When the Auto Scaling group successfully launches a new admin server through a scale-out action, an “EC2 Instance Launch Successful” event is generated in Amazon EventBridge. An EventBridge rule subsequently triggers an AWS Lambda function, passing the event data as a JSON object.

The Lambda function:

Parses the event data to extract the EC2 Instance ID.
Retrieves the IP address of the new server using the Instance ID.
Updates the DNS A Record for the admin server in the previously created Hosted Zone with the new IP address.

The Lambda function requires permissions to:

Describe EC2 instances within the account (to obtain the IP address).
Update the A record in the Hosted Zone created earlier.

To implement this, we will first create the IAM Policy, followed by the Lambda function (which will reference the policy), and finally, we’ll create the EventBridge rule (which links to the Lambda function).

IAM Policy

We create a policy named AllowWeblogicAdminServerUpdateDNS with the following JSON content. Ensure to replace <MY_HOSTED_ZONE_ID> with the ID you recorded earlier.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"route53:ChangeResourceRecordSets"
			],
			"Resource": "arn:aws:route53:::hostedzone/<MY_HOSTED_ZONE_ID>",
			"Condition": {
				"ForAllValues:StringLike": {
					"route53:ChangeResourceRecordSetsNormalizedRecordNames": [
						"wlsadmin.example.com"
					]
				},
				"ForAnyValue:StringEquals": {
					"route53:ChangeResourceRecordSetsRecordTypes": "A"
				}
			}
		},
		{
			"Effect": "Allow",
			"Action": [
				"ec2:DescribeInstances"
			],
			"Resource": "*"
		}
	]
}

Lambda Function

Next, we create a Lambda function named wlsAdminARecordUpdater with default settings for runtime (Node.js), architecture (x86_64), and permissions. We will add an environment variable named WLSHostedZoneID with the value of the Hosted Zone ID created earlier. A role will be generated for the Lambda function with a name that begins with wlsAdminARecordUpdater-role-. Attach the policy AllowWeblogicAdminServerUpdateDNS to this role.

Finally, we implement the following code into the Lambda function before saving and deploying it.

import { EC2Client, DescribeInstancesCommand } from "@aws-sdk/client-ec2"; 
import { Route53Client, ChangeResourceRecordSetsCommand } from "@aws-sdk/client-route-53"; 
				
export const handler = async (event, context, callback) => {
  				  
  const ec2input = {
    "InstanceIds": [
      event.detail.EC2InstanceId 
    ]
  };
				
  const ec2client = new EC2Client({region: event.region});
  const route53Client = new Route53Client({region: event.region});
				  
  const ec2command = new DescribeInstancesCommand(ec2input);
  const ec2data = await ec2client.send(ec2command);
  const ec2privateip = ec2data.Reservations[0].Instances[0].PrivateIpAddress;
				    
  const r53input = {
  "ChangeBatch": {
    "Changes": [
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "wlsadmin.example.com",
          "ResourceRecords": [
            {
              "Value": ec2privateip
            }
          ],
          "TTL": 60,
          "Type": "A"
        }
      }
    ],
    "Comment": "weblogic admin server"
  }
};

This process ensures that your WebLogic admin server is seamlessly integrated with DNS for high availability, providing a robust and automated solution. For further insights on structuring your professional profile, check out this resource. Additionally, for more information on employment law compliance, visit SHRM. If you’re interested in job opportunities, you can explore this excellent resource.