The Persistent Barriers to Agile Transformation—and What to Do About Them

We often talk about agile processes, but we should instead focus on agile values and principles—not just in our IT teams, but across the entire organization. This blog post examines some of the persistent obstacles to true agile transformation, and explores how BizOps can help.

Agile Transformation Hasn’t Happened for Many

As we embarked on our BizOps journey, we decided to commission Forrester Consulting to examine the current state of agile transformation and outline recommendations for executives.¹ Our key hypothesis was that a new CEO-CIO partnership was required and we wanted to gain insights into the products and practices needed to realize it.

There are many insights from the resulting survey, but to anyone who has been an agile advocate for the past several years, this picture will probably feel both shocking and surprisingly familiar.

Source: A commissioned study conducted by Forrester Consulting on behalf of Broadcom, August 2019. For additional survey findings, please read the study, "To Drive Great Business Results With Software, Close The CEO-CIO Gap."  

The fact that 72% of CIOs report that detailed requirements are compiled before software design and development starts might sound surprising, especially when it seems everybody across the industry claims they are “agile.” What’s even more surprising is that about the same percentage of CIOs we surveyed thought this current state was ideal. In reality therefore, the approaches being employed are largely what we’d refer to as “Water-Scrum-Fall.”

Contrast this with the second principle of the Agile Manifesto: “Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.”

How do we have such a disconnect between today’s realities and the desired state of embracing change and responding quickly to customer needs?

Why the “Way it’s Worked” Hasn’t Worked

IT has long been acting as a service provider to the business, and truth be told, this contract-centric culture and mentality is still pervasive. My own experience is that executives far too often demand “commitments” without understanding or embracing an agile mindset. To lead agile transformation, executives need to embrace change and uncertainty. This often requires a different leadership style, and maybe in some cases different personality types, than those that worked successfully in the past.

For the last decade, many IT leaders have reluctantly empowered agile teams by giving them “autonomy.” Often, this meant giving teams the ability to choose their own tools, to organize in teams following an agile process of their choosing, or maybe to select some cool new technology stack. But fundamentally, these leaders and their approaches did not change:

• They continued to act and behave with a traditional contract-based mentality and “timely delivery of content” as the primary measure of success.
• Work breakdown structures were lumped into sprints; gate-phased plans driving Water-Scrum-Fall created a false sense of predictability.
• Legacy KPIs were used to measure progress.
• Team “commitments” remained very much a reality.

These legacy approaches haven’t been working, and, in fact they are disastrous for organizations that need to pursue digital transformation. Here are a couple key reasons:

• Tool fragmentation. It has become extremely difficult for management to gain visibility into the status of software delivery.
• Lack of trust. Teams often don’t want to provide visibility because they don’t trust management. Engage with any team on the subject of predictability or throughput and watch the team react: You will systematically get some version of “It’s impossible to fully estimate.”

This lack of trust is particularly damaging. Gene Kim writes extensively about the concept of psychological safety in his book, “The Unicorn Project.” In particular, he uses Google as an example. A team at Google measured the extent to which members on a team felt safe to discuss problems, to say what they think without fear of being blamed or ridiculed. They found that this feeling of safety was the greatest predictor of performance. And as a result, this is integral to how they train leaders, and a critical part of the culture at Google.

How do you foster psychological safety? It starts with transparency and trust. In this effort, leaders need to go beyond paying agile lip service, and begin to truly embrace this approach.

Scrum Approaches: Are We There Yet?

Back in 1995, Jeff Sutherland and Ken Schwaber jointly presented an introduction to Scrum at an Object-Oriented Programming, Systems, Languages & Applications (OOPSLA) conference. Their key realization was that for most systems we build:

• The requirements and key results that define success are either unknown or loosely defined at the beginning of a project.
• We don’t understand the full extent of what needs to be built nor do we know all the tasks at the onset of a project.
• Assessing the quality of a system is often challenging as it may require experimentation with end customers or users.

These realities underscore how, as a leader, you have to accept and embrace uncertainty, especially when it comes to timelines and content. Sutherland outlined how Scrum is an approach that “increases flexibility and ability to deal with complexity, and produces a system that is responsive to both initial and additionally occurring requirements.”²

Through this approach, teams focus on product increments, seeking to deliver value by constraining a product’s scope to a horizon that is more manageable and predictable. For each increment, it is important to underscore that teams have total autonomy.

Unfortunately, while this approach was unveiled around 25 years ago, it’s still not a reality in most organizations. In fact, this is a far cry from the Water-Scrum-Fall pattern we continue to observe in the enterprise. It is only by rejecting this pattern and truly undergoing agile transformation that teams can position themselves for success. As outlined in the fifth principle of the Agile Manifesto: “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

Agile Transformation: How BizOps Can Help

While for many, agile transformation hasn’t become a reality to date, the urgency to make it happen continues to escalate. It is in part for these reasons that BizOps is emerging as such an important concept. In many ways, BizOps is about fostering this kind of agile mindset across the whole organization by creating a new level of transparency and trust. While BizOps is often associated with data-driven decisions, it is equally about promoting the kind of empathy that can truly transform organizational culture.

Empowering teams is not just about giving them the ability to choose their tools, but first and foremost about relentless focus on engaging on the why and the what. BizOps can be instrumental in providing this context. Through BizOps, teams can put business outcomes at the center of everything, from value management to software development to IT operations.

Don’t be afraid to engage in backlog grooming as a key tenet to gain alignment and motivate your teams by giving them context to what they are doing. Problem solving with teams is also a powerful way to build trust across organization layers.

By embracing BizOps and agile transformation, leaders can secure the kind of meaningful feedback loop that will give them the confidence that teams are progressing in the right direction. At the same time, this will empower development teams with the kind of freedom and psychological safety they need.