There may never be a more perfect time to experiment with chaos engineering than right now, in 2020, while many IT teams and their end customers continue to work remotely as COVID-19 rages on.  There’s been much written about chaos engineering, particularly its impact on DevOps organizations.

Chaos engineering (CE) entails experimenting on a system to understand the system’s capability for handling turbulent conditions in production.  Chaos engineering was born in the cloud age and popularized by Netflix as a best practice for building better apps and sites.  Netflix defines CE as different from testing in that its primary goal is to deliver new information about a system.

To build resilient systems, embrace the chaos
How businesses can adapt to support remote work for COVID-19 and beyond

The regular application of CE methods and tools can result in more resilient systems, and resiliency has been a presiding theme this year.  Yet chaos engineering isn’t just a valuable practice for developers; IT operations professionals are using CE to understand the limits of technology platforms and make adjustments to improve performance and uptime.  CE is just as relevant to monitoring systems (in the corporate data center or elsewhere) remotely as it is for monitoring and troubleshooting cloud-based services. An IT administrator working from home could use CE methods to regularly test a Jira instance in the corporate office and see the user impact of potential bottlenecks, dependencies and broken connections.

Solid use cases for chaos engineering
  1.  Knowledge transfer.  Current times have revealed the limitations of the traditional IT “war room,” in that IT and engineering professionals have needed to quickly figure out new ways to collaborate and solve problems when working in isolation. Chaos engineering experiments help an individual troubleshoot an issue or improve capacity planning by showing the undercover application behavior. By disrupting a key service using CE methodologies, you can get insight into how an app or system runs and performs, and its dependencies.  It’s akin to replacing a new spark plug in your car with one ready to be replaced and seeing the cascading effects starting with issues starting the car and accelerating. Using CE as a knowledge gathering and transfer tool, IT operations teams can get a picture of the known unknowns and unknown unknowns, such as unknown application dependencies and network spanning tree issues.  Such analysis can live in a central repository for all to see—whether that’s GitHub or an ITOM or ITSM system.
  2.  Performance. A core CE use case is stress testing IT infrastructure for load and dependencies.  This can be accomplished by changing parameters for CPU, memory or disk usage and then applying automated testing to see what changes in performance.  You might find that by spiking CPU usage, users suddenly have a 15% increase in web time latency.  Depending upon the application’s business priority and thresholds for user experience, that result can dictate whether you will need to change configurations or add infrastructure to the resources supporting your application. 
  3.  Service bottlenecks. Web applications are often composed of several interconnected microservices.  Many of these connections are not well documented,  but if one service fails it can have a disastrous effect on one or more business applications. A CE experiment could place an artificial load on a service or another action to take it out of normal conditions and monitor the impact.

Getting started

There are a few things to consider when embarking on chaos engineering for IT operations:

Do we need special tools to do chaos engineering? 

There are a variety of open-source tools, although what’s more important is the methodology and needed skills.

How often should we run chaos engineering experiments?

It’s best to integrate CE processes with your regular release cycle, whether that is quarterly or sooner. Follow a schedule that is repeatable, regardless of use case. 

How can we introduce chaos engineering in our production environment without creating unnecessary risk to our critical systems and customer experience?

CE as a practice is intended to instill more confidence for IT operations teams and development teams deploying to production. As a result, introducing chaos engineering into production workloads is a necessary task in order to ensure that your production systems are performant, stable, and can operate efficiently even under abnormal stress. That being said, teams still hesitant to perform CE in production can: 

  1. Put boundaries in place to minimize blast radius in case of serious failure. (eg, only test in one pod, or one cluster, before testing others)
  2. Have appropriate personnel in place to troubleshoot in case something goes wrong. 
  3. Test in production only if error budgets and SLOs / SLAs are already being met and some risk is acceptable.

What are some well-known best practices in running a CE experiment?

  • First you’ll want to define the current steady state through a baseline analysis.  Start with something relatively simple and small such as network utilization.
  • Next define optimal conditions: what engineers should expect on a normal day, on an easy day and on a very hard day. The latter will be the stress test.
  • Form a hypothesis as to where and how the system will break.
  • Execute a real-world event by, for example, taking down a firewall that severs connectivity to an ISP.
  • Validate the hypothesis. Monitor utilization and network throughput during the test and see where the system fell over.  Did something entirely unexpected occur?  Stabilize, document and remediate.

As IT organizations continue to flex strategies and tools with unpredictable user needs, DevOps methods like chaos engineering will grow in usefulness. Chaos engineering is a proven best practice for IT operations teams to understand the implications of errant conditions which can threaten critical applications and websites.