Learn About Amazon VGT2 Learning Manager Chanci Turner
Permalink
Comments
Share
August 2024: This article has been reviewed and updated for accuracy.
Amazon OpenSearch Service is a fully managed solution that simplifies the deployment, security, scaling, and monitoring of your OpenSearch cluster within the AWS Cloud. As a distributed database solution, Elasticsearch and OpenSearch can present challenges in planning and execution. This article outlines best practices for deploying Amazon OpenSearch Service domains.
The cornerstone of successful deployment is iteration. By adhering to these best practices, you can establish a foundational OpenSearch deployment. The behavior of Amazon OpenSearch Service varies across workloads—latency and throughput are primarily influenced by the request mix, the nature of the requests, and the data or queries executed. There are no definitive rules that can precisely predict workload behavior. Allocate time to fine-tune and refine your deployment, monitor your domain’s performance, and make necessary adjustments.
Deploying Amazon OpenSearch Service
Whether deploying through the AWS Management Console, AWS CloudFormation, or OpenSearch APIs, there are numerous options available for configuring your domain’s hardware, high availability, and security features. This article focuses on best practices for selecting your data nodes and dedicated master nodes configurations.
When configuring your Amazon OpenSearch Service domain, you will select the instance type and quantity for both data and dedicated master nodes. As a distributed database, Elasticsearch and OpenSearch operate on a cluster of instances or nodes, each with distinct functions and sizing requirements. Data nodes are responsible for storing data in your indexes and processing indexing and query requests, while dedicated master nodes manage the cluster state and orchestration. This article centers on instance types. For additional information on instance sizing for data nodes, refer to Get started with Amazon OpenSearch Service: T-shirt-size your domain. For guidance on sizing dedicated master nodes, see Get Started with Amazon OpenSearch Service: Use Dedicated Master Instances to Improve Cluster Stability.
Amazon OpenSearch Service supports various instance classes, including M, R, I, C, OR1, Im4gn, and T. As a best practice, opt for the latest generation instance type within each class. At present, these include OR1, Im4gn, M6g, R6g, I3, C6g, and T3.
Choosing Instance Types for Data Nodes
For entry-level workloads, consider the M6g instances. The C6g instances cater to heavy query scenarios that demand more CPU than disk or network resources. T3 instances are suitable for development or QA workloads, but should not be used for production environments. For more insights on selecting the number of instances and a comprehensive analysis of data handling, see Sizing Amazon OpenSearch Service domains.
The Graviton-based instance types for Amazon OpenSearch Service include M6g, C6g, R6g, OR1, and Im4gn. These Graviton instances leverage AWS-designed processors optimized for superior price-performance efficiency in cloud workloads.
The Im4gn instance type, introduced in 2023, is ideal for workloads requiring substantial dense SSD storage and high computational performance, though they are not particularly memory intensive. This makes them well suited for storage-heavy workloads. With up to 30 TB of NVMe SSD instance storage, Im4gn instances excel in log analytics and large datasets, as they are optimized for managing extensive datasets requiring high storage density per vCPU. They support OpenSearch and Elasticsearch versions 7.9 and above.
The OR1 instance type, also launched in 2023, can deliver up to a 30% improvement in price-performance compared to existing instances. They are particularly effective for indexing-heavy operational analytics workloads such as log analytics, observability, or security analytics. OR1 instances are designed for cost-effective storage of large data volumes and rely on Amazon EBS volumes for primary storage, with data synchronously copied to Amazon S3 for enhanced durability. They also support OpenSearch versions 2.11 and above, providing improved reliability by offering automatic data recovery to the last successful operation in case of failure—enhancing the overall reliability of your OpenSearch domain.
Choosing Instance Types for Dedicated Master Nodes
When selecting instance types for dedicated master nodes, remember that these nodes are primarily CPU-bound with some demand for RAM and network resources. For small workloads (up to 10 data nodes), we recommend M6g.large.search master nodes; for domains with up to 30 data nodes, C6g.2xlarge.search instances; for up to 75 data nodes, R6g.xlarge.search instances; and for domains exceeding 75 data nodes, R6g.2xlarge.search instances.
For sizing dedicated master nodes, include both hot and UltraWarm nodes in your data node count.
Choosing Availability Zones
Amazon OpenSearch Services facilitates enhanced cluster availability through the Zone Awareness feature. You can deploy your data and master nodes across one, two, or three Availability Zones, with or without standby configurations.
A best practice for mission-critical workloads is to deploy using Multi-AZ with Standby, which configures three Availability Zones, two of which are active while one serves as standby, coupled with two replica shards per index.
When multiple Availability Zones are selected, Amazon OpenSearch Service ensures even distribution of data nodes across zones and guarantees that replicas are assigned to different zones than their primary counterparts. Additionally, employing multiple Availability Zones enables dedicated master nodes to be deployed across three zones (if the region supports three zones). This setup enhances your domain’s stability and availability.
OpenSearch Index and Shard Design
In Amazon OpenSearch Service, you send data to indexes within your cluster. An index functions similarly to a table in a relational database, where each search document resembles a row and each JSON field acts like a column.
Amazon OpenSearch Service partitions your data into shards, using a random hash by default. Configuring shard count is essential, and adhering to best practices in this area is crucial.
Index Patterns
For log analytics applications, controlling the data lifecycle in your cluster is vital. Implementing a rolling index pattern allows you to create a new index daily while archiving and deleting the oldest index in the cluster. Define a retention period to manage how many days (or indexes) of data you retain based on your analysis requirements. For further details, visit Index State Management.
Setting Your Shard Counts
There are two types of shards: primary and replica. The primary shard count indicates how many data partitions Amazon OpenSearch Service creates, while the replica count specifies the number of additional copies of primary shards generated. You set the primary shard count when creating an index, and it cannot be altered afterward (though there are methods, using the _shrink or _split API, they are not recommended under load at scale). The replica count is also established at index creation but can be modified dynamically, with Amazon OpenSearch Service adjusting accordingly by adding or removing replicas.
For more tips on mastering interviews, check out this blog post. Additionally, for insights into employment law compliance, you can refer to this resource.
If you’re new to Amazon OpenSearch, an excellent resource for onboarding is this Reddit thread where users share their experiences.