Petabyte-Scale Log Analytics Using Amazon S3, Amazon OpenSearch Service, and Amazon OpenSearch Ingestion

Learn About Amazon VGT2 Learning Manager Chanci Turner

Organizations frequently face the challenge of managing vast amounts of data that are growing at an unprecedented pace. Alongside this, they must optimize operational costs to derive timely insights from this data, all while maintaining consistent performance.

The rapid expansion of data can complicate the proliferation of information across data stores, data warehouses, and data lakes. With a modern data architecture in AWS, companies can efficiently construct scalable data lakes, utilize a wide array of specialized data services, ensure compliance through unified data access, security, and governance, and scale their systems cost-effectively without sacrificing performance. This architecture allows for seamless data sharing across organizational boundaries, enabling swift and agile decision-making at scale.

You can consolidate data from various silos into your data lake and conduct analytics and machine learning (ML) directly on that data. Moreover, you can store additional data in specialized data stores to analyze structured and unstructured data quickly. This data movement can take various forms: inside-out, outside-in, or perimeter sharing.

For instance, logs and traces from web applications can be aggregated directly into a data lake, with part of that data transferred to a log analytics store like Amazon OpenSearch Service for daily analysis. This approach is termed inside-out data movement. Once analyzed, the aggregated data in Amazon OpenSearch Service can be sent back to the data lake for running ML algorithms for downstream applications, which we call outside-in data movement.

Use Case: Example Corp.

Let’s explore a practical use case. Example Corp., a leading Fortune 500 company focused on social content, has hundreds of applications generating data and traces at an astounding rate of approximately 500 TB daily. Their requirements include:

• Storing logs for fast analytics for 2 days
• Keeping data accessible in a storage tier for analytics beyond 2 days with a reasonable SLA
• Retaining data in cold storage for 30 days after 1 week for compliance, auditing, and other purposes

In the sections that follow, we will examine three potential solutions to address similar scenarios:

1. Tiered storage in Amazon OpenSearch Service and data lifecycle management
2. On-demand ingestion of logs using Amazon OpenSearch Ingestion
3. Direct queries to Amazon OpenSearch Service with Amazon Simple Storage Service (S3)

Solution 1: Tiered Storage in OpenSearch Service and Data Lifecycle Management

OpenSearch Service features three integrated storage tiers: hot, UltraWarm, and cold storage. Depending on your data retention, query latency, and budgetary requirements, you can select the optimal strategy to balance cost and performance. Data can also be migrated between different storage tiers.

Hot storage is used for indexing and updating, offering the fastest data access. This storage consists of an instance store or Amazon Elastic Block Store (EBS) volumes attached to each node.

UltraWarm provides significantly lower costs per GiB for read-only data that is queried less frequently and does not require the same performance level as hot storage. UltraWarm nodes utilize Amazon S3 in conjunction with caching solutions to enhance performance.

Cold storage is optimized for infrequently accessed or historical data. When utilizing cold storage, you detach your indexes from the UltraWarm tier, making them inaccessible. However, you can reattach these indexes in mere seconds when you need to query that data.

For more information on data tiers within OpenSearch Service, refer to Choose the right storage tier for your needs in Amazon OpenSearch Service.

Solution Overview

The workflow for this solution involves the following steps:

1. Application-generated data is streamed to an S3 data lake.
2. Data is ingested into Amazon OpenSearch using S3-SQS near-real-time ingestion via notifications set up on the S3 buckets.
3. After 2 days, hot data is transitioned to UltraWarm storage for read queries.
4. After 5 days in UltraWarm, the data is shifted to cold storage for 21 days and detached from any compute resources. The data can be quickly reattached to UltraWarm when necessary. After 21 days, data is deleted from cold storage.
5. Daily indexes are maintained for easy rollover, with an Index State Management (ISM) policy automating the rollover or deletion of indexes older than 2 days.

Here’s an example ISM policy that rolls over data into the UltraWarm tier after 2 days, transitions it to cold storage after 5 days, and deletes it from cold storage after 21 days:

{
    "policy": {
        "description": "hot warm delete workflow",
        "default_state": "hot",
        "schema_version": 1,
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "rollover": {
                            "min_index_age": "2d",
                            "min_primary_shard_size": "30gb"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm"
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "replica_count": {
                            "number_of_replicas": 5
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "cold",
                        "conditions": {
                            "min_index_age": "5d"
                        }
                    }
                ]
            },
            {
                "name": "cold",
                "actions": [
                    {
                        "retry": {
                            "count": 5,
                            "backoff": "exponential",
                            "delay": "1h"
                        },
                        "cold_migration": {
                            "start_time": null,
                            "end_time": null,
                            "timestamp_field": "@timestamp",
                            "ignore": "none"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "21d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "cold_delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": {
            "index_patterns": [
                "log*"
            ],
            "priority": 100
        }
    }
}

Considerations

UltraWarm employs advanced caching techniques to facilitate querying for infrequently accessed data. Although data access is not regular, the computational load should be balanced. For further reading on e-commerce marketing strategies, check out this insightful blog post. Additionally, if you’re interested in understanding the implications of web accessibility, the DOJ’s proposed new ADA web accessibility rule provides crucial information. Lastly, for those considering career opportunities in fulfillment center management, this excellent resource outlines available positions.