Automate File Retrieval from S3 Glacier Using FSx for Lustre

Learn About Amazon VGT2 Learning Manager Chanci Turner

Modern high-performance computing (HPC) workloads on AWS are essential for processing vast datasets efficiently while maintaining performance. These workloads depend on high-performance file systems to provide low-latency and high-throughput data access. Amazon FSx for Lustre serves as a fully managed storage service that offers a scalable, high-performance file system ideal for applications such as machine learning, HPC, and video rendering.

Many AWS users connect their FSx for Lustre file systems with Amazon S3 buckets. This integration allows for concurrent data access from both the high-performance file system and the S3 API. When linked to an S3 bucket, FSx for Lustre seamlessly presents objects as files, enabling workloads to run without the need to manage data transfers from S3. As data within the S3 bucket evolves, FSx for Lustre automatically updates the file system with the latest information necessary to execute workloads.

To reduce S3 storage expenses, customers often transition older and infrequently accessed S3 objects to Amazon S3 Glacier and S3 Glacier Deep Archive. These storage classes are secure, durable, and cost-effective options for data archiving and long-term backup. However, objects that have been moved to S3 Glacier or S3 Glacier Deep Archive cannot be accessed using Amazon FSx until they are restored by a cloud storage administrator. This blog post outlines how to set up automatic file retrieval from S3 Glacier when an FSx for Lustre file system needs to load files from S3.

Solution Overview

When FSx for Lustre is integrated with S3, all file metadata is accessible in the FSx for Lustre file system. File data is lazily loaded from S3 as files are accessed through Amazon FSx. If the file data is unavailable and the corresponding S3 object is archived in S3 Glacier, an attempt to access the file from Amazon FSx will return an error. To make the file available through FSx for Lustre, it must first be retrieved from S3 Glacier and restored to the S3 Standard storage class. This process can be automated using Amazon EventBridge and AWS Lambda. Below is an architectural diagram illustrating the event flow and component interactions in this solution.

Event Flow:

1. A user attempts to access a file in the FSx for Lustre file system.
2. FSx for Lustre holds only the file metadata, not the contents. Thus, it issues an S3 GetObject API call to load the file from S3.
3. FSx for Lustre does not find the file in S3 since it has been archived to S3 Glacier, prompting S3 to respond with an error code of InvalidObjectState.
4. AWS CloudTrail is configured to alert Amazon EventBridge upon receiving an S3 GetObject response with the InvalidObjectState error code.
5. EventBridge triggers a Lambda function that initiates the retrieval of the file from S3 Glacier using an S3 RestoreObject API call.
6. The file becomes available in S3 and can be accessed by users through FSx for Lustre.

Configuration Walkthrough

1. Set Up AWS CloudTrail with Amazon CloudWatch Logs
   Ensure that S3 GetObject calls are logged in Amazon CloudWatch Logs. These logs will be sent to EventBridge, which will subsequently initiate the Lambda function. Start by creating a CloudTrail trail with CloudWatch Logs enabled. This will monitor S3 GetObject calls on the S3 bucket associated with FSx for Lustre.
2. Create IAM Role for Lambda Function Execution
   Next, an IAM Role must be created, allowing the Lambda function to execute and make S3 RestoreObject calls. Navigate to the IAM console, create an IAM policy with the necessary permissions, and then create a role that uses this policy.
3. Create a Lambda Function
   Finally, create a Lambda function that will trigger when an S3 GetObject call to the S3 bucket linked to FSx for Lustre results in an InvalidObjectState error. In the Lambda console, create the function and input the required code to initiate the ObjectRestore call to S3 Glacier.

For those interested in career growth in tech, check out this blog post for tips on leveraging AI in job searches. It’s also important to stay informed about the implications of recent changes to employment practices. SHRM provides great insights into equal pay transparency laws. Furthermore, Amazon employees are getting paid to learn new skills; explore this excellent resource for more information.