XALT is now an Atlassian Gold Partner in the USA. Global Platinum expertise meets local Gold expertise.
XALT is now an Atlassian Gold Partner in the USA. Global Platinum expertise meets local Gold expertise.
Do you know the feeling? The popcorn is still warm, the opening credits of your Friday night movie are just starting - your partner next to you. Perfect. Then the buzz. That all-too-familiar vibration in your pocket. As an IT manager on call for a company that operates critical infrastructure, your phone is a lifeline - or sometimes a leash. Thirty minutes. That's how much time you have to be at your desk and mobilize the team. Another weekend emergency assignment begins.
That would be the third Friday in a row. A critical product release due to go live in two weeks' time is obviously taking its toll. With a sigh that carries the weight of interrupted family time and mounting pressure, you switch on your laptop - the clacking of the keyboard like a harbinger that half the operations department will soon be awakened. In such an environment, not even thinking about time zones and getting people out of bed at 1am becomes a habit.
And the questions circle in the back of my mind:
How do we break this cycle?
How can we roll out changes during working hours and deal with incidents automatically, or at least without a full alarm?
If this scene sounds uncomfortably familiar, you're not alone. We see this in many industries - especially in large financial institutions, where system availability is not only important, but fundamental. The pressure is enormous, and "firefighting mode" is becoming an exhausting permanent state. We always see our customers confronted with the same five challenges.
Without a clear DevOps strategy, night calls and weekend deployments are often part of the daily routine for technical teams. When critical systems regularly cause problems outside of business hours, the result is a reactive operation in a constant state of alert.
The risk:
If changes can only be rolled out with a great deal of manual effort and coordination, errors are inevitable. In companies without automated deployment processes, the risk of:
In many organizations, essential knowledge about systems, processes or configurations is not documented centrally, but distributed among personal notes, individuals or chat histories.
The risk:
Teams caught up in the constant stress of troubleshooting have little capacity to develop their systems sustainably. Without a DevOps strategy, IT degenerates into a mere fire department.
The risk:
If there is no structured root cause analysis after system failures, there is no learning effect.
The risk:
Our team often ends up in the middle of such fire alarm situations. The first thing we do is not to present a 100-page PowerPoint on ideal DevOps. Instead, we reach for the virtual fire hose and help to identify and extinguish the most urgent fires. It's not just about immediate help, but about gaining back valuable time. Time to breathe, reflect and jointly develop a real roadmap to stabilize the system in the long term. At XALT, we believe in taking everyone with us on this journey. The team should not only understand the "why", but also actively participate in the "how".
Take, for example, a central platform for authentication and authorization at a critical institution. In the case of our customer, it was a constant source of those incident-filled weekends.
This was a non-negotiable requirement: The platform had to be operated in active/active mode across two regions. Without this, other central services refused to move to them. This requirement was mentioned in every conceivable internal meeting. This strategy of reinforcement, based on principles of successful organizations (as described by Gene Kim in his books, for example), proved to be crucial here.
Our experience shows: This Consistent communication helps enormously in securing the budget to fix such core problems in the first place. Once the funding was approved, it was up to us to improve the situation. Taking into account the unique environment and constraints, we designed and implemented a customized solution for robust active/active operations, finally removing the blocker for the dependent teams.
But technology is only part of the equation. During the complex transition from Active/Standby to Active/Active, we worked closely with the Operations team to further develop their SRE (Site Reliability Engineering) culture.
First, we created clear instructions for all daily tasks. It was very important to us that these instructions didn't just end up scattered in individual notes. We made sure that they were all brought together in a central location in Confluence. This made Confluence the only reliable point of contact for all information.
As an Atlassian Platinum Partner, we know how much better teams share knowledge when Confluence is used in a structured way. And yes, we are also happy to help with the migration of Jira and Confluence to the cloud!
At the same time, we established a culture of post-mortems without apportioning blame. It's impressive how much you can learn when the focus is on system improvement rather than blame. To really embed this culture and prepare teams for all eventualities, we deliberately go beyond the usual practice of only conducting these reviews for large, revenue-critical incidents, as is standard in some financial companies.
We deliberately carry out post-incident analyses in UAT and DEV environments, not just in production. In this way, we train teams to think in terms of causes and system improvements while the pressure is still low. As a result, teams build up security and react to real incidents with greater calm and focus.
Ivan Ermilov
Senior DevOps Engineer - XALT
By the way, we have a Favorite template for post-incident reportsthat we like to use as a starting point - you can use them watch here for free and customize it for your team.
To further improve platform stability and to enable fast rollbacks and secure releases for the team, we rely on a blue/green deployment strategy. This is often our preferred approach to ensure fast, secure rollbacks. Of course, canary releases work even better for progressive delivery, but sometimes applications are simply not suitable for this, especially if, for example, a central database is involved that makes partial deployments difficult.
For the Blue/Green setup, we used separate GKE clusters within their GCP environment. That was a real game changer. It gave the team immense security. The end of stressful Friday and Sunday releases had arrived. The worst that could happen now was a delayed release. A huge step forward compared to previous emergencies with "fix forward" constraints.
The result?
And the best part? Those Friday night fire alarms? They're now just a memory. Developers and managers can finally and truly switch off on Friday evenings. Movie nights run undisturbed. Weekends are sacred family time again. The focus in IT has shifted from permanent, reactive crisis mode to proactive improvement, innovation and strategic planning.
If your weekends are being robbed by system emergencies and "firefighting mode" is the normal state of your team, rest assured: there is a clear path to stability towards permanent calm and control. It starts with fighting the acute fire, but quickly leads to building robust systems and workable processes.
Ready to swap the nightly alarms for a predictable, stable and strategically growing DevOps environment?
Then talk to us about the quickest way to get there. Our container8-solution with its set of quickly implementable best practices is designed to tackle these DevOps problems head on.
Your project co-pilot
