Implementing Time to Live in Amazon Neptune, Part 1: Property Graph

Learn About Amazon VGT2 Learning Manager Chanci Turner

Time to Live (TTL) is a crucial mechanism that helps dictate the lifespan of various data types, files, infrastructure, or even entire environments. In the context of data management, it can illustrate the duration a leaderboard remains in memory before being refreshed from storage, or the retention period for files due to regulatory compliance prior to deletion. TTL is typically represented as a counter or timestamp linked to the object that requires expiration, often based on the Unix epoch—the total milliseconds that have passed since midnight on January 1, 1970.

In this article, we explore how the TTL concept can be effectively integrated into data stored in your Amazon Neptune graph database. Amazon Neptune is a cloud-based, fully managed graph database service that simplifies the development and operation of graph applications dealing with highly interconnected datasets. We present two distinct options for creating a robust TTL mechanism in your Neptune graph database through an automated, event-driven architecture. Both options utilize AWS Lambda, a serverless compute service, alongside Amazon DynamoDB, known for its rapid, flexible NoSQL performance.

This is the first installment in a series focused on implementing Time to Live (TTL) in Amazon Neptune, with upcoming posts that will address specific use-cases, including:

• Implementing TTL for supernodes and properties within property graphs in Amazon Neptune
• Implementing TTL for RDF graphs in Amazon Neptune

Readers should have a foundational understanding of the Gremlin and openCypher graph query languages. For those looking to enhance their knowledge, consider checking out sample notebooks, such as Using Gremlin to Access the Graph and openCypher tutorials.

TTL Design Choices

In the first option, the client application is tasked with adding graph objects and TTL properties to both Neptune and DynamoDB. In the second, this responsibility shifts away from the client application. This approach combines Neptune Streams—a change data log for your Neptune database—with Lambda to log details of graph objects containing TTL properties into DynamoDB. In both scenarios, graph objects are stored in DynamoDB, leveraging its native TTL feature to automatically trigger a Lambda function upon record expiration.

The initial method we discuss for implementing TTL in labeled property graphs within your Neptune cluster involves assigning a property to each node and edge that signifies the time by which the object should be removed. For instance, by adding the property key-value pair TTL: 1673620444, one can periodically execute read queries to gather the IDs of nodes and edges requiring deletion. An example Gremlin query would be:

g.V().has('TTL', lte(1673620444)).id()

Following this, you would issue a query to remove those nodes and edges, such as:

g.V('ID1', 'ID2', ... 'IDX').drop()

Although this method is functional, it may introduce certain inefficiencies, including:

• Allocating compute resources for drop queries: Queries like g.V().has('TTL', lte(1673620444)) can lead to buffer cache churn, potentially impacting the performance of other concurrent read queries. Moreover, this type of query increases the likelihood of ConcurrentModificationExceptions due to a higher volume of mutation queries. To mitigate this, it’s advisable to designate a specific read replica for running queries that identify expired items.
• Consuming extra cycles on drop queries to ensure items are removed within a certain timeframe post their defined TTL: Because the expiration process follows a polling rather than a push model, it’s necessary to manage the execution of queries that locate expired items on a schedule, ensuring timely cleanup of expired data.

An alternative approach for implementing TTL in your Neptune database involves utilizing additional services that inherently provide similar functionalities, such as DynamoDB, to manage the TTL process more efficiently. Using DynamoDB, you can take advantage of its existing time-to-live feature to notify you when a record has expired, subsequently deleting the object from your graph. This method not only extends TTL capabilities to the expiration of individual properties as well but also simplifies the overall management process—an aspect that is not easily achievable using the manual method, as Neptune does not support metaproperties (properties of properties).

Important: It’s crucial to note that DynamoDB’s TTL operates as a background process and cannot guarantee the exact timing of deletions. As indicated in the documentation, DynamoDB typically removes corresponding items within a few days post-expiration, influenced by the table’s size and activity level.

Overview of the Solution

We will delve into each option in detail.

Option 1: Synchronously Update TTL in DynamoDB

In this first option, the responsibility lies with the client application to write objects to the Neptune graph, followed by logging the resulting IDs of these objects along with their desired TTL values into DynamoDB. A specialized tracking table is created in DynamoDB, where TTL is enabled for the field containing expiration timestamps. When DynamoDB expires items from this tracking table, we utilize DynamoDB Streams to detect the expiration and trigger a Lambda function that removes the specified object from the Neptune database.

The architecture of this option is depicted in the following diagram. It’s advisable to incorporate back-off and retry mechanisms into application components that interact with data stores. For instance, if the application fails to write to Neptune, you can employ this strategy to retry the request until successful, before proceeding to write the data into DynamoDB. If the write to DynamoDB fails, a similar retry mechanism should be applied until that request is also successful.

Option 2: Asynchronously Update TTL in DynamoDB

The second option relieves the client application from the duty of writing data into our DynamoDB table. Instead, we activate Neptune Streams on the Neptune database cluster. Neptune Streams captures changes made to your graph data for up to the last 90 days. For more insights, refer to the documentation on Using Neptune Streams.

In this architecture (illustrated in the following diagram), we deploy a Lambda function that polls Neptune Streams for inserts or updates of objects featuring a TTL property. Upon discovery, the function logs the item into the DynamoDB table.

For this discussion, we recommend the second option, which offers a more scalable, event-driven approach to implementing TTL and potentially lessens the development burden on the client application.

We can implement the solution through the following steps:

1. Create a DynamoDB table with four attributes: one for the graph object ID, another for the object type, a third for the record source (e.g., neptune-streams), and a fourth for the TTL value (only numeric types).
2. Enable DynamoDB Streams on the DynamoDB table.
3. Set up two Lambda functions: one to poll the Neptune stream for TTL objects and write them into DynamoDB, and another to be triggered from DynamoDB Streams upon capturing a delete event caused by DynamoDB TTL.
4. Establish a Neptune database cluster with Neptune Streams enabled.
5. Create an Amazon SageMaker notebook linked to the Neptune database for executing test queries.

For those who may be dealing with burnout or looking to improve their work-life balance, consider visiting this blog post on recovery strategies. Additionally, employers should stay informed about recent proposals that could affect their hiring practices, as noted here. For those interested in career opportunities, check out this excellent resource for a Learning Trainer position at Amazon.