What is continuous testing? It’s about making testing an ongoing aspect of every activity though the software delivery cycle. This post examines why continuous testing is such a critical component of Agile development environments.
Agile, DevOps, continuous integration, and continuous delivery are part of every techie’s vocabulary today. We all want to be Agile, release software faster, and create innovative products like the Silicon Valley giants and startups. Many aspects of this transformation are very well understood, and to an extent successfully adopted. However, one area where there is still considerable confusion and adoption challenges is that of testing. When the software release cycle shrinks from years or months to weeks, days, or even faster, how do we reshape our testing practices to ensure the software released in production does not break and end up frustrating users, rather than delighting them?
Clearly, this is a common challenge, given that most DevOps shops still consider testing as the most frustrating bottleneck in software development.
Continuous, automated testing is the answer, but what is continuous testing and how do you go about implementing it? Wikipedia defines continuous testing as “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” But this definition misses the essence, nature, and magnitude of the transformation that continuous testing signifies. While automation is certainly a big part, continuous testing is fundamentally about transforming testing from being a time-boxed event in a linear process to embedding it as a fundamental and ongoing aspect of every activity though the software delivery cycle.
The goal of continuous testing in Agile environments should be “in-sprint testing.” Regardless of whether your sprint is two weeks or four weeks, the goal should be to complete all types of testing within that sprint so that every sprint ends with fully tested, ready-to-ship software. And if you think that your sprint is too short to allow for comprehensive testing, most likely you are thinking about it wrong.
Some organizations default to “running all the tests all the time” to ensure code coverage. This wastes resources and lengthens test cycles, without actually ensuring adequate code coverage. Save time, money, and resources by testing only what you need to test. Visual models allow various paths to be explored and optimized such that only handful of tests provide maximum coverage. It is also possible to import unit tests and test cases, remove duplicates and distribute and allocate optimized test cases in existing tools.
3. Employ Shift-Left Testing
To achieve “in-sprint” testing, shift testing left—so tests can be run earlier in the development cycle. Developers test as they go; the CoE provides expertise, specialized systems, and services. Test automation spans UI, functional, performance, and security. Teams work together and are focused on business value delivered to the customer. This requires developer-friendly tools as well as a culture shift.
4. Provide Complete Test Environments
The ability to provide test environments is critical to achieving continuous testing. Eliminate blocks and reduce wait times by providing complete test environments on demand with dev-friendly tools (as code, CI/CD integrations, supported open source). These environments should include:
Virtual services—provide robust simulation for unavailable, inaccessible, or still-in-development services. Developers and testers can continue to work in parallel by using the virtual service to simulate the response from the actual service.
Test data on demand—helps ensure teams can perform comprehensive tests with production-like data.
Ephemeral environments—ready on demand, decommissioned after use.
5. Get the Right Test Data
The inability to get robust test data causes significant delays in many application release cycles. To accurately test new functionality, test data should be as close as possible to what the application will encounter in production. If the test data lacks certain real-world characteristics (such as actual fields, data specifications, negative scenarios, and so on), the tests are unlikely to find many potential issues or break the application where there are weak points.
Understanding data models in order to extract the right data is a special skill in and of itself. And although production data is clearly the most realistic data to use for testing, data privacy regulations often restrict its use. The answer to this dilemma is a robust test data management tool, which can let you copy production data and mask it to protect sensitive information, while maintaining the characteristics that make production data desirable for testing: realistic and referentially intact data across rows. These tools can also generate test data synthetically from scratch when production data simply cannot be used.
6. Don’t Forget to Shift Right
Shift right to use data from the dev cycle and from production to optimize test cycles, fine tune tests, and build optimal regression suites. Shift-right techniques include real and synthetic user monitoring, canary deployments, A/B testing, chaos engineering, etc. For example, by shifting right, you can identify which features are being used in production and ensure your regression tests cover those features. In the same way, you can release a new feature to a small group of users (internal or external), understand the impact of that feature in production, and adjust as needed.
Many of the nimblest companies like Facebook and Netflix rely heavily on shift-right testing. Gartner recently released a report on shift-right testing, calling it a “must” for adopting DevOps practices.
7. Use Data and Metrics to Continuously Improve
Build cross-team collaboration and continuously improve with actionable analytics and feedback loops. Continuous delivery and continuous testing are journeys. Make sure all your teams have KPIs and access to data that will inform additional optimization. Don’t just collect data that is easy to collect, collect and report on the data that will provide actual insights to continuous improvement. Look to release orchestration applications to deliver this insight and a “single source of truth” for all teams across the application delivery lifecycle.
Sushil Kumar leads the DevOps business at Broadcom and is responsible for continuous delivery, continuous testing and API lifecycle management products. Sushil has over 25 years of industry experience in wide ranging roles spanning engineering, product management, marketing, professional services, and IT ops. Prior to joining CA/Broadcom, Sushil worked with leading tech companies such as Oracle driving key initiative such as Cloud, AI, and data analytics.