Improving Resilience with Chaos Engineering on AWS Designed with Cloudairy Cloudchart

As technology systems become more intricate and spread out, ensuring their resilience and dependability becomes a significant worry. This is where chaos engineering becomes important. In this blog post, we will discuss how a particular organization used chaos engineering tests on AWS to enhance the resilience, observability, and adherence to regulations of its cloud system before deploying it to production. 

Chaos engineering is a discipline that involves intentionally introducing controlled failures or disruptions into a system to identify weaknesses, uncover hidden bugs, and assess the system's ability to recover gracefully. By simulating real-world failure scenarios in a controlled environment, organizations can proactively identify and address potential issues before they manifest in production.

For organizations operating in regulated industries, such as financial services, chaos engineering becomes even more crucial. It helps ensure that critical systems remain resilient and compliant with stringent regulatory requirements. AWS provides a suite of tools and services that enable organizations to effectively implement chaos engineering practices.

The organization in question had a three-tier application architecture deployed across multiple virtual private clouds (VPCs) in a multi-availability Zone (multi-AZ) setup. The web application resided within a public subnet, utilizing an Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling group. It was connected to an Amazon Relational Database Service (Amazon RDS) database hosted in a private subnet. Additionally, internal services were deployed in containers within a separate VPC.

Failure Scenarios Tested:

During the 3-day Experience-Based Acceleration (EBA) event organized in collaboration with AWS, the organization's cross-functional technical teams performed various chaos engineering experiments using AWS Fault Injection Service (FIS). The following failure scenarios were tested:

Amazon EC2 Instance Failure:
 
The team simulated EC2 instance failures using the aws:ec2:stop-instances and aws:ec2:terminate-instances FIS actions. This helped evaluate the resilience of running containers managed by services like Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS) when faced with instance failures.

Amazon RDS Failure:
 
To identify and troubleshoot database issues, the team simulated RDS failures, including failovers and node reboots. FIS was employed to inject reboot/failover conditions into managed RDS instances, uncovering potential bottlenecks and disaster recovery challenges.

Severe Network Latency Degradation:
 
The team injected latency into the network interfaces connecting different systems to understand the impact of data transfer delays on the application's performance and the operational team's response readiness. The FIS action aws: SSM: send-command/AWSFIS-Run-Network-Latency, utilizing the Linux traffic control (tc) utility, facilitated this testing.

Network Connectivity Disruption:

Connectivity issues were simulated using the aws:network:disrupt-connectivity action to assess the application's resilience to total or partial subnet connectivity loss and disruptions across AWS networking components.

Amazon EBS Volume Failure (IOPS Pause):
 

Disk failures were simulated by pausing I/O operations on target Amazon Elastic Block Store (EBS) volumes using the aws:ebs:pause-volume-io action. This helped evaluate the system's performance under different disk failure scenarios, particularly for volumes in the same Availability Zone attached to instances built on the AWS Nitro System. 

The chaos engineering experiments conducted during the EBA event yielded valuable insights and led to several architectural improvements. The organization was able to reduce application recovery time, enhance metric granularity, and improve alerting mechanisms. Moreover, they developed a reusable chaos engineering methodology and toolset that could be applied to future experiments.

The success of the event highlighted the effectiveness of regular in-person cross-functional collaboration in implementing a robust chaos engineering practice. By leveraging AWS services like AWS Fault Injection Service and AWS Resilience Hub, organizations can systematically improve their cloud system resilience, ensuring better preparedness for real-world disruptions and compliance with regulatory standards.

Beyond the technical aspects, chaos engineering also brings significant business benefits. By proactively identifying and addressing potential vulnerabilities, organizations can minimize the risk of costly downtime and service disruptions. This translates to improved customer satisfaction, enhanced brand reputation, and increased trust in the organization's ability to deliver reliable services.

Furthermore, chaos engineering fosters a culture of resilience and continuous improvement within the organization. It encourages teams to think proactively about failure scenarios, develop robust recovery strategies, and continuously refine their systems to withstand real-world challenges. This mindset shift from reactive to proactive resilience is crucial in today's fast-paced and highly competitive business environment.

To fully realize the benefits of chaos engineering, organizations must establish a well-defined process and framework. This includes defining clear objectives, identifying critical systems and failure scenarios, establishing monitoring and observability mechanisms, and conducting thorough post-experiment analysis and documentation. By institutionalizing chaos engineering practices and integrating them into the software development lifecycle, organizations can ensure a consistent and systematic approach to resilience testing.

Moreover, it is essential to involve stakeholders from various domains, including development, operations, security, and business, in the chaos engineering process. Collaboration and communication across teams are key to aligning chaos experiments with business objectives, ensuring buy-in from stakeholders, and fostering a shared understanding of system resilience.

In conclusion, chaos engineering on AWS empowers organizations to proactively identify and address potential vulnerabilities in their cloud systems. By embracing this discipline and utilizing the tools and services provided by AWS, organizations can enhance the resilience, observability, and regulatory compliance of their critical workloads. As the complexity of cloud environments continues to grow, chaos engineering will undoubtedly play an increasingly vital role in ensuring the reliability and stability of modern distributed systems. Organizations that adopt chaos engineering practices will be well-positioned to navigate the challenges of the digital landscape and deliver exceptional value to their customers.

Leveraging Cloudairy Cloudchart's collaborative features, organizations can design well-defined Chaos Engineering experiments, streamline team communication, and ensure all stakeholders have a clear understanding of the testing process and expected outcomes. This translates to more efficient and effective Chaos Engineering practices, ultimately leading to more resilient cloud architectures on AWS.