Site Reliability Engineering Is a Kind of Magic

    It takes a lot of operational wizardry to keep today’s modern, highly dynamic environments running smoothly—which is why Site Reliability Engineering (SRE) is becoming so vital. This post examines the changing nature of today’s environments, and the tough demands facing SREs.

    Site Reliability Engineering Teams: The Wizards of IT

    Site Reliability Engineering (SRE) teams can be considered the IT equivalent of wizards, or as Andrew Widdowson, an SRE at Google, described it, “Like being part of the world’s most intense pit crew… changing the tires of a race car as it’s going 100 mph.”

    So how are SREs different from traditional IT operations? Can a discipline originating in the world of web-scale, cloud-native unicorns ever apply to steady-as-she-goes state of enterprise IT? Yes, it can.

    The scale out way is really the new way of managing enterprise IT. The notion that enterprise IT exists behind closed walls doesn’t exist anymore. Now, the only way to create and conduct business at scale is through engineering reliability managed in an unprecedented manner. The demand for mobile experiences and the advent of complex cloud architectures has shifted the operational focus. It’s no longer about keeping the lights on. It’s instead about performance. Apps have to work well, the experience has to be great, and the infrastructure behind it needs continual monitoring.

    Like any feature, reliability isn’t something that’s retrofitted after deployment; it’s established and enhanced as software is developed, tested, and released. That means establishing a new discipline, which Ben Treynor—Google’s original SRE lead—describes as “what happens when a software engineer is tasked with what used to be called operations.”

    A Sobering Reality

    It’s easy to throw out yet another three-letter acronym and claim it’s a magical elixir for all the problems involved with running complex IT systems. In reality, engineering reliability into distributed systems with thousands of containerized applications and microservices is a tough gig. Not least because of all the moving parts, but also because any preconceived notions about predictable system behavior no longer apply.

    Take for example keeping watch over a modern software application. This might consist of business logic written in polyglot languages and linked to the legacy ERP system (custom built or packaged or both). There’ll also be a raft of databases—traditional relational for transactional support, yes, but more likely a smorgasbord of NoSQL data stores—be that in-memory, graphing, or document—perhaps fronted by recently adopted Node.js.

    Some of this componentry will be on-premises, some will be containerized and moved to the public cloud. That might mean Docker and Kubernetes on AWS, or maybe Azure and Mesos—heck, why not both for some hybrid-style resilience?

    But like the old Monty Python sketch, “you’ll be lucky” if this is all you ever have to manage. Depending on the nature of the business, there’ll also be a glut of third-party services—including payment processing and reconciliation. That’s not to mention all the new web and mobile apps interacting with the core business systems through an API gateway and possibly some analytics horsepower delivered by the likes of Hadoop and Elasticsearch. It’ll take a lot of operational wizardry to keep all that performant.

    Fortune Favors the Bold

    In a wonderful talk at SREcon, Julia Evans from Stripe described the realities of managing today’s complex distributed systems. What was refreshing about her presentation was the open admission that she often finds the work difficult and how there’s always a ton of new stuff to learn. As she says in her abstract, she doesn’t always feel like a wizard (echoing the protestations of Harry Potter).

    This honesty illustrates what’s exciting about being an SRE. With systems like the ones described above causing any number of thorny problems, it’ll be the inquisitive and brave that keep business on track. Being an SRE isn’t for the faint-hearted or those happy with a fire-fighting status-quo. It’s for those within our ranks who get bored easily—those super sleuths who keep asking reliability questions, crafting improvements, and learning as they go.

    So, if we consider a typical business-critical problem that could impact our modern application—let’s say some latency issue is causing an increasing number of mobile app users to abandon a booking service? How would teams address the issue? Problems like this might go unnoticed for some time, or there could be a deluge of alarms. Even when a problem is identified, where do teams find the root-cause? Is it a problem with a new code release or at the API gateway? Is it down to some weird microservices auto-scaling issue? Was that earlier CPU increase we thought was OK actually really bad?

    With an SRE-style approach, business-critical problems are never addressed in knee-jerk fashion. Using modern tooling in areas such as AIOps and application performance monitoring, SREs can observe the real-time behavior of applications, with systems collecting and correlating information from all related components. Rather than react after the fact, these solutions continuously identify anomalous patterns (like those mobile app abandonments) and compare them to historical trends—meaning SREs are alerted well before the business is impacted.

    But beyond exposing new normal application weirdness and “unknown-unknowns,” modern tools also encourage and stimulate more of the SRE detective work—the real valuable stuff. These tools won’t just detect anomalies and then leave teams scrambling to find the needle in a haystack of needles. Instead, they’ll analytically gather all the evidence and lead teams in fact-based fashion towards a solution. Like, for example, using an SRE-inspired monitoring service to detect a performance anomaly introduced with a new software build and then tracing to the actual code causing the problem.

    Like Harry Potter, operations professionals might have a hard time accepting they’re wizards. But ask yourself this—do you want to remain a silly muggle getting burnt out by constant fire-fighting? Of course not, it’s career limiting and sucks. Time then for some SRE magic—gaining the skills and tools needed to adopt new tech like containers and microservices, and becoming an essential part of future-proofing your business.

    Looking to AIOps for Trusted Guidance

    What if you could empower SREs with the insights needed to drive improvements? What if instead of the typical war rooms and on-call burn out, SREs had a trusted guide to quickly fix problems? Today, AIOps solutions can help. These solutions can help your SRE teams prevent alert fatigue by triaging, using a combination of notification rules, process changes, dashboards, and machine learning to proactively monitor SRE golden signals and measure what really matters for customer experience.