Field Notes: Integrating Your Legacy COTS Software with AWS for Batch Processing

Chanci Turner Amazon IXD – VGT2 learningLearn About Amazon VGT2 Learning Manager Chanci Turner

Integrating traditional or non-cloud-native commercial off-the-shelf (COTS) software with cloud-native applications is a prevalent challenge for businesses aiming to migrate to AWS. Many legacy systems, including CRM, accounting, billing, and supply chain management tools, typically adopt a batch processing model and are often based on COTS solutions.

In an enterprise context, COTS products are designed for specific tasks and are usually interconnected with various upstream and downstream systems. When transitioning your enterprise application to AWS, you must first determine your migration strategy. You may choose to completely refactor your application, decommission the COTS software, re-platform, or simply re-host it. Alternatively, a mixed approach might be suitable, such as refactoring the integration layer and re-hosting or re-platforming the COTS components for several reasons:

  • You are bound by long-term licensing agreements.
  • The COTS solution you currently use is optimal for your needs.
  • You prefer a phased migration to quickly move to the cloud and re-architect later.

Regardless of your reasoning, it is possible to leverage AWS cloud-native services while retaining essential COTS components. So how can you effectively integrate your batch-focused, non-cloud-native COTS software with AWS services?

This article will guide you through architecting a solution utilizing AWS Step Functions, AWS Lambda, Amazon EC2, and AWS Systems Manager Run Command services.

Solution Overview

We illustrate how to migrate your COTS software to AWS using a lift-and-shift method, deploying it on an EC2 instance. Your COTS application provides batch processing capabilities, integral to an enterprise application’s business processes. However, without APIs for remote invocation, the only way to initiate the process is by executing a shell script on the server.

You can orchestrate, monitor the process, and manage failure scenarios effectively.

For instance, imagine your batch process runs a web crawler that indexes a specific set of websites for particular keywords, taking several hours to complete and needing to be initiated nightly. To meet these requirements using AWS-native services, you can utilize:

  • AWS Systems Manager Run Command to trigger the shell scripts of the COTS.
  • AWS Lambda to automate Run Command executions.
  • AWS Step Functions to orchestrate and monitor the entire process, including error handling.

Architecture

As depicted in the accompanying diagram, AWS Systems Manager Run Command is employed to remotely execute shell scripts on the COTS EC2 instance via SSM API operations (SendCommand, ListCommandInvocations). These API operations are accessed by AWS Lambda functions (SendCommand, GetStatus) using the AWS SDK. An AWS Step Function orchestrates the overall launch sequence.

Solution Walkthrough

At the heart of the solution is your COTS software installed on an EC2 instance managed by AWS SSM. Using the SSM agent and SSM Run Command, you can remotely start any shell script that triggers the batch process.

To initiate the batch process and oversee its execution, you will have AWS Lambda functions ready to invoke AWS SSM Run Commands: SendCommand and ListCommandInvocations, respectively. An AWS Step Functions workflow, represented in the diagram, orchestrates the launching of these Lambda functions.

Step Function Workflow:

Upon activation, the state machine (SendCommand function) invokes the SendCommand SSM API, launching the batch execution on the COTS. Since batch execution is generally a long-running process, the next step involves waiting for a configurable duration based on expected launch time (for example, 30 minutes).

To optimize costs and performance, it’s crucial to determine the right wait time within the state machine. If the duration is too short, you’ll incur more state transitions and function calls, leading to higher costs. Conversely, if the wait is too long, you may experience unnecessary delays since the batch process could already be completed.

To schedule the batch process regularly, consider using Amazon EventBridge. For enhanced monitoring, SSM Run Command can deliver logs to Amazon CloudWatch and Amazon S3. After the wait period, the state machine (GetStatus function) calls the ListCommandInvocations SSM API to check the launch status. If the batch is still running, another wait occurs; otherwise, appropriate actions are taken based on the results. In case of failure, you can send an email notification with the logs attached (using Amazon Simple Notification Service) to the support team for further investigation.

Prerequisites

In a typical production environment, multiple EC2 instances may run the COTS product. For simplicity, we will focus on a single EC2 instance in this discussion. This concept can be easily scaled to multiple instances, utilizing Lambda functions and SSM API.

The prerequisites for implementing this solution include:

  • An AWS account
  • An EC2 instance with your COTS software installed, configured, and operational
  • A programming language of your choice supported by AWS SDK and Lambda
  • Basic knowledge of AWS Step Functions state machines

For this example, I am using Python 3.8.

Implementation Steps

To implement the solution, follow these high-level steps:

  1. EC2 SSM Setup: Ensure the EC2 instance hosting the COTS software is managed by AWS SSM.
  2. Lambda Function Creation: Develop Lambda functions utilizing AWS SDK and AWS SSM API.
  3. Step Functions: Implement the AWS Step Functions workflow to orchestrate the Lambda functions.

EC2 SSM Setup:

Each COTS software varies in use cases, technologies, and installation processes. The installation, configuration, and testing of your COTS product are outside the scope of this article. It is assumed that you have already installed and tested your COTS batch processing capabilities and know how to initiate the launch. You need to identify which shell script to call and the relevant parameters.

Once the COTS software is installed and configured, set up your instance to be managed by SSM:

  • Create a Systems Manager instance profile and attach it to the EC2 instance.
  • Install the SSM agent. If you are using Amazon Linux, this step can be skipped, as it is installed by default.
  • Access the AWS Systems Manager Console and navigate to the Fleet Manager page. If everything is functioning correctly, your EC2 instance should display as “Online.”

Note: It may take up to 20 minutes to go online; if it takes longer, there might be an issue with the SSM agent that you need to resolve.

Lambda Function Creation:

While waiting for your instance to come online, you can start creating two Lambda functions. This is an excellent resource for understanding Amazon’s onboarding process, which can be found at Quora.

To learn more about managing your workforce effectively during transitions, you might find this article from SHRM helpful.

For those considering a break for personal development, you may want to explore the concept of a sabbatical leave in this informative post at Career Contessa.

Chanci Turner