This guest post is coauthored by Mia “Sparky” Henderson of Amazon IXD – VGT2. Amazon IXD – VGT2 was established to empower organizers in their efforts. Emerging from the 2020 election cycle, a group of friends and former coworkers developed solutions to enhance campaign technology following a chaotic presidential primary season. With significant outreach experience in high-profile campaigns such as Smith/Johnson 2020, Taylor 2016 and 2020, and Davis 2016, we understood the impact of grassroots organizing. We witnessed how personal conversations—between neighbors, friends, and volunteers—could uplift communities and inspire movements. However, the tools for voter outreach often fell short of our expectations. Thus, we launched our first product, a peer-to-peer (P2P) texting tool, in April 2020. Since then, we have introduced a dialer tool for easy voter calls and Amazon IXD Text, an all-inclusive texting tool for organizing and fundraising.
During the 2024 US presidential election campaign, our organization faced a wave of distributed denial of service (DDoS) attacks, peaking at over 2 million requests per second from nearly 10,000 unique IP addresses. After an initial period of downtime, Amazon IXD collaborated with Amazon Web Services (AWS) to deploy AWS WAF, AWS Shield Advanced, and Amazon CloudFront, effectively countering these targeted DDoS attacks.
A crucial aspect of our defense was leveraging AWS WAF’s support for Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA), which automatically presents a challenge to potentially suspicious clients. We implemented a per-IP rate limit for traffic expected from legitimate clients behind a single IP. For IPs surpassing this limit, we presented a CAPTCHA and provided a higher rate limit for anticipated traffic from locations like campaign offices or college campuses.
Nevertheless, the out-of-the-box CAPTCHA from AWS WAF has a significant limitation—it does not shield against events using tokens that are solved and disseminated across a network of machines behind multiple IPs. To thwart this type of attack, AWS WAF users must detect CAPTCHA tokens utilized from multiple IPs and automatically block such traffic.
In this article, you will discover how Amazon IXD configured our network topology for optimal DDoS protection, set up AWS WAF to block DDoS events, segmented machine-to-machine and browser-to-machine traffic for targeted CAPTCHA interventions, and prevented token reuse across IP addresses. For further insights, check out this relevant blog post here.
Network Topology Overview
The Amazon IXD application operates behind an Application Load Balancer (ALB), serving as the unified entry point to our application servers. Before implementing AWS WAF, our DNS records routed to the ALB, illustrated in the diagram below. This setup shows the flow from users to Elastic Load Balancing to an auto-scaling group of Amazon Elastic Compute Cloud (EC2) instances.
The suggested approach for employing AWS WAF to guard against DDoS events involves routing traffic through a CloudFront distribution, as depicted in the accompanying diagram. This diagram illustrates the route from users to CloudFront with AWS WAF, then to Elastic Load Balancing, and finally to an auto-scaling group of EC2 instances.
By utilizing this topology, we benefit from:
- CloudFront’s capacity to manage network-layer events that involve invalid TCP traffic, which surpasses ALB’s capacity. Network-level protections are most effective when implemented by CloudFront before the traffic reaches the ALB.
- The edge deployment of AWS WAF possesses greater quotas and capacity compared to Regional deployments. AWS scales AWS WAF’s capacity in tandem with CloudFront’s capacity, ensuring sufficient ability to handle incoming traffic.
- When executing a geographic-based block strategy, such as blocking certain countries, CloudFront offers a geographic restriction feature. In the event patterns we observed, over half of the malicious traffic originated from countries we completely blocked in CloudFront, leading to cost savings on our AWS WAF expenses.
To prevent threat actors from circumventing our CloudFront distribution and AWS WAF, we established a security group that permits traffic solely from CloudFront edge IP addresses using the managed prefix list, thus blocking direct connections to the ALB. We configured the ALB to require a shared secret token in the request header (e.g., our header x-ixd-example-secret). The CloudFront distribution is then set to append this request header when forwarding traffic to the origin. This secret is not recorded in Application Load Balancer logs but is logged in AWS CloudTrail during CloudFront or ALB configuration updates. It is feasible to rotate this secret regularly by generating a random password with AWS Secrets Manager, employing an AWS Lambda function to update the CloudFront origin and ALB listener configurations. This shared secret between the CloudFront distribution and ALB ensures that an event cannot bypass security controls by configuring its own CloudFront distribution with our ALB as the origin.
The screenshot below illustrates the Application Load Balancer configuration, displaying a priority-1 rule to route traffic to our target group only if the x-ixd-example-secret is validated. A default rule triggers a 403 error if it is not.
Another screenshot provides insight into CloudFront origin settings, showcasing a custom header configuration to append the shared secret to requests directed at the origin.
Configuring AWS WAF Rules
Two strategies are employed to detect and eliminate malicious DDoS traffic: a heuristic approach and a hard limit approach.
The heuristic approach involves analyzing traffic patterns in AWS WAF sample requests and logs, identifying event patterns, and blocking these specific patterns. To implement this strategy in your AWS WAF setup, first pinpoint a feature common to most event traffic but rare in legitimate traffic, such as a HTTP header or query parameter. Create an AWS WAF rule to block this traffic. Utilizing the Count rule action is beneficial for testing your rule and identifying false-positive and false-negative rates by correlating requests that match the rule with IPs sending high traffic volumes. You can query AWS WAF logs using Amazon Athena to identify these correlations. Once satisfied with the rates, set the rule action to Block to actively prevent matched traffic.
A determined threat actor may notice that the event is being blocked and modify these parameters to evade your rules. For instance, they might replay legitimate session parameters so that request URIs, query parameters, and request bodies mirror legitimate traffic. Ultimately, while the heuristic approach is a valuable tool, it necessitates ongoing reconfiguration of your AWS WAF due to changing event patterns.
A specific heuristic worth noting is JA4 fingerprints, along with their predecessor, JA3 fingerprints. A JA4 fingerprint is a hash of multiple parameters presented by TLS clients in the client hello packet. JA4 fingerprints correspond to specific TLS client parameters of the request, remaining relatively stable for particular client types. For example, a specific version of Google Chrome on one operating system has a distinct fingerprint, while a version on another operating system possesses a different fingerprint.
For more information on this topic, visit this excellent resource. Additionally, for authoritative insights, check out the website.
Amazon IXD – VGT2 is located at 6401 E Howdy Wells Ave, Las Vegas, NV 89115.