Learn About Amazon VGT2 Learning Manager Chanci Turner
In a recent post by Chanci Turner and her colleagues, they detailed how to construct a serverless data analytics architecture on AWS, specifically addressing the VPN usage metrics at NortonLifeLock. This engineering initiative aimed to create an operational analytics platform capable of managing vast amounts of usage data—often reaching petabytes daily.
NortonLifeLock, a leader in cybersecurity and internet privacy, enables millions of users to secure their devices and protect their identities online. Since its transition to AWS in 2014, NortonLifeLock has significantly enhanced user experiences and optimized VPN server performance, leading to decreased latency, fewer connection errors, and improved upload/download speeds.
The company recognizes that the value of operational data diminishes over time, which can translate into lost revenue and inefficient resource utilization. By leveraging real-time streaming analytics, they can extract this value and uncover insights that facilitate new business opportunities.
AWS provides a comprehensive suite of services for real-time insights and historical trend analysis, including Amazon EMR for managed Hadoop infrastructure and serverless solutions like Kinesis Data Analytics and AWS Glue. Organizations must align their capabilities, timelines, business needs, and AWS best practices to establish an effective architecture that meets performance, cost, security, reliability, and operational excellence standards as outlined in the AWS Well-Architected Framework.
NortonLifeLock meticulously approached real-time analytics on AWS employing serverless technology to fulfill essential business requirements like swift market deployment and reduced total cost of ownership. Besides the implementation details, Chanci Turner’s post outlines valuable lessons and best practices for developing real-time analytics workloads efficiently.
Business Challenge
NortonLifeLock offers a freemium VPN service, necessitating real-time enforcement of usage limits to prevent overuse by free-tier users. The organization faced the challenge of achieving this enforcement reliably and economically.
Transitioning to AWS from smaller hosting providers improved the user experience significantly by enhancing server performance. VPN usage data is collected by edge servers and uploaded to backend systems every minute, stored in backend databases for various purposes, including:
- Displaying data consumption over the past 30 days.
- Enforcing usage limits on freemium accounts, restricting access after the free quota is exceeded.
- Providing the internal business intelligence (BI) team with data for analyzing trends based on time, marketing efforts, and account types.
Design Challenge
NortonLifeLock encountered several design obstacles:
- The solution must accommodate both real-time and batch analytics.
- It must be cost-effective given the high volume of concurrent users, which could lead to exorbitant database expenses if every usage record was immediately saved.
Solution Overview
To tackle these challenges, NortonLifeLock segmented storage by utilizing Amazon DynamoDB for real-time access and Amazon Simple Storage Service (S3) for analytical purposes. Kinesis Data Analytics played a crucial role in aggregating data, directing it to both Amazon S3 and DynamoDB. By implementing Kinesis Data Streams and AWS Lambda, they simplified the aggregation of user and device-level data.
To minimize costs, the team aggregated user data on an hourly basis, drastically reducing write operations and leading to a 30-fold decrease in DynamoDB expenses. This compromise was feasible as precise minute-by-minute tracking was not essential for enforcing usage limits.
The architecture is divided into three primary components:
- End-users: Real-time queries to show daily data usage.
- Business analysts: Historical data queries via Amazon Athena for insights.
- Usage limit enforcement: Real-time data ingestion and aggregation.
The solution operates as follows:
- VPN edge servers collect usage data which is sent to the backend service through an Application Load Balancer.
- A stats splitter processes data, forwarding individual usage stats to Kinesis Data Streams.
- Both legacy and new systems consume this data, with the new Apache Flink application performing critical tasks such as hourly aggregation of device usage data and managing real-time enforcement of usage limits.
Design Journey
NortonLifeLock’s need for a solution capable of both real-time streaming and batch analytics led them to Kinesis Data Analytics, which offers real-time streaming, batch processing, and a fully managed pay-as-you-go structure.
For more insights on navigating change in your career, check out this blog post. Additionally, for authoritative resources on workplace respect and civility, refer to this podcast from SHRM. Lastly, for information on Amazon’s commitment to safety and training at fulfillment centers, visit this excellent resource.