New AWS Lambda Features for Stream Processing and Asynchronous Invocations

Chanci Turner Amazon IXD – VGT2 learning manager

AWS Lambda has rolled out new controls designed to enhance the handling of asynchronous and stream processing invocations. These newly introduced features empower users to tailor their responses to Lambda function errors, enabling the construction of more robust event-driven and stream-processing applications.

Stream Processing Function Invocations

When managing data from sources like Amazon Kinesis Data Streams and Amazon DynamoDB Streams, Lambda retrieves records in batches through shards—distinct sequences of data records. Your function processes these records from the batch in a specific order. If an error occurs, Lambda will retry processing the batch until it either succeeds or the data expires. While this retry mechanism is useful in many scenarios, it can lead to challenges:

Until the problem is resolved, no data from the shard can be processed. A single flawed record can halt processing for the entire shard, as the “in order” guarantee mandates that failed batches are retried until the data record expires. This is often referred to as a “poison pill” event, obstructing the overall system from processing data.
In certain situations, retrying may not be beneficial if future invocations are likely to fail as well.
If the function isn’t idempotent, retries could lead to unexpected outcomes, possibly resulting in duplicate entries (e.g., multiple database entries or business transactions).

New Lambda Controls for Stream Processing Invocations

With these new customizable controls, users can manage how function errors and retries affect stream processing invocations. A new event source-mapping configuration subresource allows developers to definitively specify which controls will apply to their invocations.

DestinationConfig: {
    OnFailure: {
       Destination: "SNS/SQS arn (String)"
    }
}

{
    "MaximumRetryAttempts": integer,
    "BisectBatchOnFunctionError": boolean,
    "MaximumRecordAgeInSeconds": integer
}

Features Explained:

MaximumRetryAttempts
This sets a limit on the number of retry attempts for batches before they are skipped, helping to unblock processing and prevent duplicate outputs.
Minimum: 0 | Maximum: 10,000 | Default: 10,000
MaximumRecordAgeInSeconds
This defines the maximum age of a record in seconds, with expired records being skipped to continue processing. Data records that are not successfully processed within the defined submission age will be skipped.
Minimum: 60 | Default/Maximum: 604,800
BisectBatchOnFunctionError
This feature allows for the recursive splitting of the failed batch and retries on a smaller subset of records, ultimately isolating the metadata causing the error.
Default: false
On-failure Destination
If either MaximumRetryAttempts or MaximumRecordAgeInSeconds is reached, a record will be skipped. If ‘Destination’ is configured, metadata about the skipped records can be sent to a target ARN (like Amazon SQS or Amazon SNS). Without a designated target, the record will be discarded.

Getting Started with Error Handling for Stream Processing Invocations

To demonstrate the new controls, we will set up a customized error handling configuration for stream processing invocations. You will configure the maximum retry attempts to 1, set the maximum record age to 60 seconds, and send metadata of skipped records to an Amazon Simple Queue Service (SQS) queue.

Create a Kinesis Stream for record input and an SQS queue for the metadata of exhausted or expired data records.
In the Lambda console, select Create function. Choose Author from scratch and name your function “customStreamErrorExample.” Set Runtime to Node.js.12.x and create the function.
To trigger Kinesis and send failed invocation metadata to the SQS queue, you need to grant the Lambda execution role the necessary permissions.
Under the Execution Role section, select the view link below the Existing role drop-down.
Add an inline policy in JSON format, replacing {yourSQSarn} and {yourKinesisarn} with your respective ARNs, and name the policy CustomSQSKinesis.
Return to your Lambda function and add Kinesis as a trigger, selecting your stream.
Access the Additional settings and paste the ARN of the previously created SQS queue. Set Maximum retry attempts to 1 and Maximum record age to 60. Click Add.
In the Function code section, enter the following code with an intentional syntax error to force a failure response:

exports.handler = async (event) => {
    // TODO implement
    console.log(event);
    const response = {
        statusCode: 200,
        body: event,
    };
    return response;
};

You can trigger the Lambda function by adding a record to the Kinesis Stream using the AWS CLI.

aws kinesis put-record --stream-name {YourStreamName} --partition-key 123 --data testdata

In the AWS Lambda console, navigate to Monitoring > View logs in CloudWatch to see the latest logs. The Lambda function should fail as expected, with only a single retry attempt due to the set maximum retry attempt value of 1.
Check your SQS queue in the AWS Management Console to view the metadata of your failed invocation, which has been successfully sent to the SQS destination for further analysis.

Asynchronous Function Invocations

When a function is invoked asynchronously, Lambda places the event in a queue before processing it. If exceptions occur in the function code, the invocation will be retried twice, with delays of one minute and then two minutes. Some invocations might not execute due to throttling or other issues, which complicates the overall process.

To explore more about effective job applications, check out this resume format quiz. For a deeper understanding of employment law compliance, you can visit SHRM’s overview on Oregon Equal Pay Acts. If you’re interested in a career opportunity, consider this position at Amazon, which could be an excellent resource.