With the rise of digitalization, software quality has emerged one of the major competitive differentiators for businesses. Quite simply, the ability of the development organization to deliver the highest quality software has become foundational to business success. Within most organizations, teams view quality as being driven by engineering, focusing on the question "Did we build things right?"
Approaches like continuous integration, continuous deployment, and continuous testing are becoming pervasive, but these disciplines all tend to be focused on the implementation stage of the development lifecycle, and on answering, “Did we build the right things?”
While these concepts are invaluable, they need to be augmented by another key concept: continuous validation. In order to understand what continuous validation is about, it is helpful to return to the origins of quality as stated by Gerald Weinberg:
"Quality is some value to some person."
This definition helps to underscore the integral connection between quality and value, and within this context, customer value specifically. Starting with this as a perspective, we look at the question “Did we build the right things?” and enhance it by asking “Did we build the things right?”
When discussing user journeys with customers, teams rarely focus on verifying quality. This remains a persistent gap between engineering and the business. Ultimately, the risk is that engineering loses the critical context: Why should a certain feature or product even be built for the customer?
Aiming for an extended approach bridging the gap between quality engineering and business, continuous validation addresses the following concepts:
- Establish awareness of the value delivered to the customer
- Focus on epic completion instead of individual user stories
- Foster organizational improvement as part of the business idea
- Enhance trust between all parties when delivering value
The basic idea is to establish feedback loops along the entire development path, from gathering the initial idea until final delivery to production. Throughout, we’re validating that we still deliver value to some person. This feedback is only valuable if it is received in a timely manner and it can be related to a specific value being delivered. This is aligned with the fail-fast approach, which is well known among engineers managing CI/CD pipelines.
Continuous Validation: Feedback Loops Across the Development Journey
Let's have a closer look at what it actually means to establish feedback loops along the development journey.
Using Scaled Agile Framework (SAFe) as a foundation for the development journey stages, continuous validation proposes feedback measures for each stage. It is key to ensure that the context about why we build something of value is kept in mind and regularly validated to ensure it is still of value to some person. Following is information on each stage:
- Roadmap. The agile toolbox comes with plenty of tools, including Value Proposition Kanvas and customer reviews of the first mockups. With these tools, we can check if we captured the idea and business needs correctly and ensure we reflect that in our product roadmap.
- Release planning. Adding context relating to time is essential in creating a release plan. This effort requires close cooperation with the customer to ensure that the product still delivers a value at a given time. (Critically, you want to ensure the product isn’t going to be delivered too late.)
- Program increment (PI) planning. At this point, we require clear ideas of the end-to-end test scenarios that guide engineering, including concrete examples that offer input for upcoming PI planning. Acceptance criteria have to relate to the delivered value and not only to an isolated user story. In addition, you should keep in mind how to demonstrate this value to the customer. These end-to-end views are key in order to retain business context when breaking down the features into the various user stories. Story mapping, which I personally like a lot, helps to visualize the priorities, customer needs, user stories, and releases, as well as how this is all connected. The closer we get to effective implementation, the more familiar the methodologies seem to be, since these efforts increasingly focus on how to “build things right.”
- Sprint planning refinement. Teams tend to think about demonstrating stories only when they are done and create a demo based on the implementation rather than preparing already a demo based on customer value. That’s why this phase of refinement is so vital.
- Sprint realization, coding, testing, demo. At this stage, there’s the risk that teams lose the value context and is essential to strive again for a connection to the customer value. Invite customers for extended sprint demos and do guided user experience tests to validate features.
- PI system demo. Receiving guided customer feedback, like guided UX testing, after a few implementation steps is often more beneficial than many complicated testing procedures. Don’t get me wrong, however: For agile quality engineering, it is mandatory for the team to be having discussions around which quality aspects the testing should focus on in the upcoming cycle. It is also critical to determine which tests should be automated.
- Release and run production. Continuous validation simply extends the view on quality beyond the pure engineering view. All of these feedback loops help to evaluate the customer value of the new feature, at different stages along the development cycle. We can close the loop by incorporating observability into our toolbox. Through observability, we can validate the feature in production. Collecting data about usage, accessibility, and various other metrics allows us to gain direct feedback on the customer perception. We can do canary releasing or defining customer validation programs. In addition, many organizations already support direct defect ticketing as part of the product itself to enable further customer feedback. Hopefully, all these measures will provide a clear answer as to whether we built the right things and we built them right.
Continuous Validation: Key Questions to Get Started
Multiple measures have been mentioned and each team has to figure out which of the continuous validation measures are appropriate for their environment. In order to know where to start, you only need to ask a few questions:
- Do we (as a team) know why the customer needs a specific feature? What is the value they’re looking to receive?
- Do we have an end-to-end test scenario example in order to ensure we build the right feature?
- How should we demonstrate the feature?
- When do we get the first round of feedback from the customer?
Answering these questions helps guide you to the corresponding stages mentioned above. Any of these activities help you to get closer to delivering customer value and therefore improves the day-one quality perceived by the customer.
Conclusion: Key Takeaways
Following are the key takeaways for successfully pursuing continuous validation:
- Do not focus solely on engineering methodologies; extend the view to the customer
- Establish fast feedback loops to validate customer value at different stages, and make them easily visible to the team
- This journey starts with the first business idea and lasts until the feature is used by the customer in production
- Incorporate organizational improvements as part of the business idea to deliver more value to the customer in the future
In order to establish basic continuous validation, one option would be to consistently implement behavior-driven development (BDD) within the development path. This will be a topic we’ll dive into in an upcoming blog post.
