Learn About Amazon VGT2 Learning Manager Chanci Turner
The growing demand from various applications and the rapid expansion of unstructured data from tasks such as high-performance computing (HPC), video rendering, and editorial workflows have made traditional Network Attached Storage (NAS) solutions inadequate due to their scalability limitations and performance bottlenecks.
Innovative use cases arise when scale-out file storage extends beyond the confines of customer datacenters, leveraging Amazon Web Services (AWS) to achieve unparalleled scalability, security, and cost efficiency. The AWS Cloud serves as an optimal environment for these types of file-centric, frequently accessed workloads. Solutions from AWS Partner Network (APN) partners utilize AWS services as foundational elements to deliver distinctive offerings tailored to file synchronization and sharing requirements.
In this article, we will explore Qumulo, an APN Standard Technology Partner, whose continuous replication capability allows for seamless data sharing while supporting use cases such as workload bursting across diverse locations.
Qumulo File Fabric (QF2) for AWS
Headquartered in Seattle, Qumulo operates both on-premises and in the AWS Cloud, providing an enterprise file storage system designed for organizations managing data-intensive workloads that require scalable storage solutions.
Qumulo File Fabric (QF2) is particularly well-suited for scenarios demanding high storage performance alongside real-time insights into storage utilization at scale. It is also ideal for extending existing on-premises deployments to AWS, thus offering the necessary elasticity.
QF2 utilizes a distributed architecture where numerous individual computing nodes collaborate to create a cluster with scalable performance and capacity, unified under a single file system. Clusters can begin with as few as four instances and can expand to accommodate up to 1,000 instances. Clients can access the file system using NFS or SMB protocols, allowing both Windows and Linux hosts to interact with the same file share.
What sets Qumulo apart from other file synchronization and sharing solutions is its ability to deliver real-time insights into data usage along with up-to-the-minute analytics for billions of files and directories. The aggregated metadata facilitates this visibility, even as the system scales.
Capacity and performance trends for the cluster can be monitored down to file-level granularity. This analytics capability also equips administrators with instant access to vital information for diagnosing issues, as well as enabling proactive planning for future capacity and performance needs, regardless of scale. This immediate insight into cluster usage can be effectively paired with the real-time quota feature.
For more information about the solution, visit Qumulo’s official page.
Components
The AWS components necessary to construct QF2 are outlined in the accompanying diagram. It comprises Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Block Store (Amazon EBS) SSD volumes, and Amazon EBS throughput-optimized volumes (st1).
Deployment
Qumulo is accessible through the AWS Marketplace and offers a variety of preconfigured editions (20TB Node, 5TB Node, and AWS free edition). The 5TB or 20TB node deployments enable users to establish clusters of different sizes, and the AWS free edition allows users to commence with QF2 on AWS.
The subsequent illustrations depict AWS CloudFormation template topologies for the AWS free trial edition and a 20TB cluster constructed from four 5TB nodes.
QF2 can be deployed using an Amazon Machine Image (AMI) or a CloudFormation template. For deployment, visit the AWS Marketplace, select “QF2 for AWS – Free Edition,” and subscribe.
Once the CloudFormation setup is finalized, access the URL shown in outputs. A login screen will appear, where you can enter the username “admin” and the password established during setup.
The QF2 dashboard provides an overview of the cluster’s status and activity, including various settings such as cluster protection status, cloud-based monitoring, DHCP, and NTP. Additionally, it displays performance metrics such as IOPS and throughput.
To create an SMB share, navigate to the QF2 dashboard, select Sharing > SMB Shares, and input the required details. Once saved, the share will appear in the QF2 dashboard.
When deploying Qumulo in a production setting, consider factors such as the optimal deployment strategy for handling failures, scaling the cluster, and seeking performance and sizing guidance. For comprehensive information on deployment models in AWS, refer to the Qumulo Knowledgebase Documents or this resource on deployment models.
Qumulo Use Case with FuseFX
FuseFX, a provider of visual effects for major Hollywood studios, simultaneously manages over 60 television shows, each with large visual effects rendering jobs that must meet stringent deadlines.
Recognizing AWS’s potential to deliver the elastic compute resources necessary for their workflows, FuseFX sought a modern, highly scalable file storage solution capable of meeting the high-performance computing demands of their workloads within AWS.
Already a Qumulo user, FuseFX was eager to explore the capabilities of QF2 for AWS. They established a four-node QF2 cluster to support 1,000 32-Core Amazon EC2 instances, achieving peak performance of 40,000 IOPS and 3.87GB/sec throughput.
The flexibility of utilizing the AWS Cloud for both extensive rendering and storage enabled FuseFX to meet commitments, expedite production, and mitigate the risk of downtime associated with on-premises render farms.
Summary
By employing the continuous replication feature in QF2, you can seamlessly integrate your namespace spanning both on-premises and Amazon EC2 instances. With QF2 on AWS, your workloads can leverage the on-demand scalability and elasticity offered by Amazon EC2 instances running alongside Qumulo, effectively eliminating the need for over-provisioning for individual projects, costly rental fees, and time-consuming deployments of temporary or permanent infrastructure onsite.
Numerous other applications, such as genomic sequencing and editorial workflows, stand to benefit from the Qumulo solution. To learn more and get started, check out the AWS Marketplace.
In the meantime, if you’re considering a return to work, this insightful blog post may provide valuable guidance. For a comprehensive overview of what to expect on your first day at Amazon, this resource is excellent.