The Path to Stable DevOps Structures and Strategy: Evening Off Instead of Fire Alarm

Do you know that feeling? The popcorn is still warm, the trailer for your Friday night movie is just starting – your partner beside you. Perfect. Then the buzz. That all-too-familiar vibration in your pocket. As an on-call IT manager at a company running critical infrastructure, your phone is a lifeline – or sometimes a leash. Thirty minutes. That's how much time you have to be at your workplace, to mobilize the team. Another weekend emergency deployment begins.

This would be the third Friday in a row. A critical product release going live in two weeks is clearly taking its toll. With a sigh carrying the weight of interrupted family time and mounting pressure, you switch on your laptop – the clatter of the keyboard a harbinger that half the operations department will soon be woken up. In an environment like this, it becomes routine not to think about time zones anymore and to pull people out of bed at 1 a.m.

And in the back of your mind, the questions keep circling:

How do we break this cycle?

How can we roll out changes during business hours and handle incidents automatically, or at least without full alarm?

Our team often lands right in the middle of such fire-alarm situations. The first step is not to present a 100-slide deck on ideal DevOps. Instead: grab the virtual fire hose and help identify and put out the most urgent fires. This isn't just about immediate relief, but about winning back valuable time. Time to breathe, think, and jointly develop a real roadmap to stabilize the system long-term. At XALT, we believe in taking everyone on this journey. The team should not only understand the “why” but actively contribute to the “how.”

Authentication Platform in Active/Active Operation

Take, for example, a central platform for authentication and authorization at a critical institution. In our customer's case, it was a constant source of those incident-heavy weekends.

One non-negotiable requirement was: the platform had to run in active/active mode across two regions. Without that, other central services refused to migrate to it. This requirement came up in every conceivable internal meeting. This reinforcement strategy, based on principles of successful organizations (as described, for example, by Gene Kim in his books), proved decisive here.

Our experience shows: this consistent communication helps enormously in securing the budget to fix such core problems in the first place. Once the funds were approved, it was up to us to improve the situation. Taking into account the unique environment and constraints, we designed and implemented a tailored solution for robust active/active operation, finally removing the blocker for the dependent teams.

SRE Culture and Structured Operating Processes

But technology is only part of the equation. During the complex transition from active/standby to active/active, we worked closely with the operations team to further develop its SRE (Site Reliability Engineering) culture.

First, we created clear guides for all daily tasks. It was very important to us that these guides didn't just end up scattered across individual notes. We made sure they were all brought together in one central place in Confluence. That's how Confluence became the single reliable point of contact for all information.

As an Atlassian Platinum Partner, we know how much better teams share knowledge when Confluence is used in a structured way. And yes, we're happy to help with migrating Jira and Confluence to the cloud too!

In parallel, we established a blameless post-mortem culture. It's impressive how much you can learn when the focus is on system improvement instead of blame. To truly anchor this culture and prepare teams for every eventuality, we deliberately go beyond the usual practice of only conducting these reviews for major, revenue-critical incidents, as is standard at some financial companies.

By the way, we have a favorite template for post-incident reports that we like to use as a starting point – you can view it here for free and adapt it for your team.

To further improve platform stability and enable the team to have fast rollbacks and safe releases, we relied on a Blue/Green deployment strategy. That's often our preferred approach to ensure fast, safe rollbacks. Of course, canary releases work even better for progressive delivery, but sometimes applications simply aren't suited for it, especially when, for example, a central database is involved that makes partial deployments difficult.

For the Blue/Green setup, we used separate GKE clusters within their GCP environment. That was a real game-changer. It gave the team immense confidence. The end of stressful Friday and Sunday releases had come. The worst thing that could still happen now was a delayed release. A massive improvement over previous emergencies with forced “fix forward”.