How Enhanced DevOps Practices are Fueling Benefits at BT Group

    BT Group is one of the leading communications services companies. By coupling optimized DevOps practices with advanced requirements design, automation, and AIOps, the team at BT has been able to speed delivery, enhance quality, and provide increased customer value. This post from Glyn Martin, Head of QA Transformation at BT, offers an insightful look at their continuous testing and automation journey, and some of the key strategies that have fueled their success.

    Accelerating Transformation

    2020 was a difficult year for all of us. In most industries, organizations’ revenues and profits have taken a hit from the COVID-19 pandemic. Working within BT, a global communications provider, we’ve been challenged by the pandemic, but we’re relatively resilient.

    If anything, the pandemic has caused us to accelerate our transformation journey. That’s meant we’ve had to make changes quickly. We’ve already been on a journey to agile for some time, but that journey is now more important than ever.

    While we want to deliver changes to market quickly, we also want to do so sustainably, and that’s true whether or not there’s an emergency. In our organization, the implications of services not working can be terrible. For example, we’ve been supporting testing centers and new hospitals created to treat COVID patients, so we need to get it right.

    Factors like our coverage, our service quality, and how quickly we can innovate have taken on a new scale of importance, especially in a telco market that was already very competitive. Like a lot of other IT organizations, we have to contend with this paradox, one in which we are under pressure to deliver more value, improve quality, and do so more quickly, all while contending with reduced budgets. Consequently, we need flexible cost models that support increasing scale, and our ability to demonstrate value, which is key to our ability to sustain growth.

    Overcoming Silos and Complexity to Pivot Quickly

    In many ways, telecoms aren’t that different from other enterprises, such as insurance or financial services firms, when you consider the complexity of products and services. In our case, we have hundreds of thousands of product permutations. As a business, we’re trying to simplify and do so in an agile way.

    BT, like most companies, is very siloed. We’re still trying to work to become closer as a company. While we’ve been pursuing DevOps practices, we’ve still got different elements and organizations. Some of these teams are doing similar things, but they’re working in silos.

    People are starting to realize that teams need to work better together, particularly in terms of the coupling between business and IT. Especially as we move to more SaaS-based and low-code solutions, there’s not such a big gap as there was in the past. Increasingly, we can expect many of our business partners to be much more tech savvy.

    There are also a lot of challenges in terms of how you integrate with other tools and technologies. At BT, we have 58 different IT stacks; that’s not systems, that’s entire stacks, with each stack comprised of hundreds of systems. At this time, we’re engaged in a simplification program, looking to reduce that number to 14. Even then, there will be complexity behind the scenes that will present increasing challenges moving forward.

    How do we as an IT organization get leaner, even when we still have some legacy systems, which we’ll never fully get rid of. That’s the kind of trade off we have to make; how do we actually deal with that complexity and hide it from users. We need to pursue a number of programs so we can accelerate change, and reduce the cost and waste of legacy systems.

    Human-Centered Design

    Within the IT organization at BT, we have taken a human-centered design approach as a core principle. That means we don’t start with technology. We’re not trying to impose something, simply mandating that there’s a technology they need to use. The person may not understand the technology or how it relates to their problems.

    By taking a human-centered approach, we talk to customers, understand their paint points, and map out their current processes. We also try to understand their aspirations, such as where they want to be in six months. We’ll look at why a given process is not running well, and see what solutions can be applied. By starting with an understanding of the customer and their pain points, we can be better equipped to determine, and deliver, the best solution.

    The Move to Realize Real Speed Improvements

    To deliver the service levels customers want and expect, quality has to be ingrained throughout the life cycle. To do so, we need to instill a commitment to quality across the organization. As part of this, we wanted to shift testing left. We consequently needed shared capabilities that span the lifecycle and help drive automation and improvements.

    I always say, “you’re only as good as your lowest common denominator.” What we were finding was that we’d be trying to do certain things quickly, but the ultimate results were limited. For example, we had automated build and test capabilities. But if it took us weeks to create test scripts or manually curate test data, we weren’t going to meet our overall speed objectives. Plus, with these manual efforts, we were finding that our coverage was ultimately still poor and that led to defects making it later into the lifecycle, or even into production, and we simply can’t afford to let that happen.

    Enter Continuous Testing

    Focusing on continuous testing over the last year has really given us the ability to deliver quickly across the whole lifecycle. We were able to go from doing what I’d refer to as a semi-agile approach, where we did user stories and agile ceremonies, but we weren’t really deploying any faster to production. That’s because our stakeholders were scared that we didn’t have the same level of control as before, when we were doing waterfall releases. We’ve done a lot of work on every aspect, especially from a testing point of view. Rather than just looking at automated testing, we’re looking at moving test creation and execution, including security and performance testing, earlier in the lifecycle. This is really a key in our ability to have continuous testing fuel enhanced DevOps practices.

    Eliminating Customer Value Gaps: Requirements Design

    We want to work at a fast pace and really deliver value. What we’d find is that there’s often a gap between defining value and the customer realizing value. We’d start with requirements on one end, but at the other end, what we’d deliver to market was very different. We think testing or delivery teams have done a great job, met acceptance criteria, but then the customer uses the feature and it doesn’t meet their needs. Or, in other cases, maybe there’s a great idea that arose 12 months ago. We end up deploying as a new feature, and six months after rolling it out, see it hasn’t been used at all. Maybe the offering didn’t accurately map to customer requirements, or maybe the idea wasn’t executed in time to be useful. Either way, all the work invested in designing, building, testing, and deploying that feature is wasted.

    We certainly can’t afford that kind of waste. We need to eliminate that waste and clearly understand the drivers and requirements for the changes we are making. That’s the only way we can ensure that what we produce actually delivers value. That’s why we implemented advanced requirements design capabilities.

    Once we started rolling out more requirements design capabilities, our customers were really enthusiastic and wanted more. We continue to want to have more projects driven by these sophisticated requirements gathering processes. This allows us to collaborate effectively with business stakeholders. With these advanced capabilities, we’re able to more effectively assess requirements, tease out requirements we hadn’t thought of, and ensure high levels of test coverage.

    As a result of all this, not only have we been able to generate tests more quickly, but we’ve been able to achieve better test coverage. By using artificial intelligence (AI) within our pipeline tools, we can actually employ a risk-based testing approach and ensure we’re doing the right tests. We’ve made a lot of great progress in this area.

    AIOps and the Move to Automation

    We started on a continuous testing journey, but that’s just the start. In a world where the pace of change is always increasing and demands on the team continue to grow, we need to do things in the leanest, most efficient manner possible.

    In improving our DevOps practices, we’re looking at every aspect from a quality assurance point of view. We’re focusing on things like flow management, which will become increasingly important. We’re seeking to better understand what pieces of our processes are taking too long. That involves looking at every aspect of the software delivery cycle.

    In this paradigm, how do we get the capabilities to move to becoming centers of excellence, rather than doing massive amounts of work? Instead of dedicating a lot of effort on doing testing, for example, we’re looking to get to where we’re focusing on helping teams to automate their testing. That’s the challenge going forward.

    We need to think about every part of the process, and how to put the stepping stones in place to move to a more automated future. For that reason, we’ve started to move to AIOps. That represents a stepping stone to automation, which is the long-term destination. We’re employing AIOps (which stands for artificial intelligence for IT operations) so we can monitor systems and see what’s actually happening in production. We’re working to determine how to best feed that intelligence back into the lifecycle.

    The Vision: End-to-End Automation of Delivery Pipelines

    We’ve started creating continuous delivery and DevOps pipelines. However, there are still manual aspects to that lifecycle currently. I’ve put a challenge to my team, asking them to help get us to where we can establish a completely automated cycle. How do we get to the point at which I can put a requirement into our project tracking software, then click a button, and, with zero touches, put that change into production? How can we have confidence that the change has been done safely, that it works, or that, if it doesn’t work, that it’s handled immediately? That’s going to be our focus in the coming months.

    Unified, AI-Powered Insights

    We want to continue to be more agile. We’re early in our cloud journey. We need to make sure the different technologies in place, including legacy systems, work with our evolving cloud implementations. We need to bring it all together, and have a full pipeline that everyone can view, so they can manage their work and make informed decisions.

    At the heart of it, it’s about looking at data, looking at processes, identifying where the waste is. In this area, AI is becoming more prevalent in the way we work, helping us to be more effective, faster, and more accurate. AI is enabling us to find defects quickly, ensure we’re running the right tests, and anticipate problems before they happen in a production environment.