The use of artificial intelligence (AI) systems can elicit images of robots taking over the world, machines doing seemingly impossible things with absolute precision, and devices powered by software doing unimaginable things. A crucial element in all of this is what we as humans experience, and we are wowed or have a feeling of “eh.” There is a lot of thought, hard work, and research that goes into producing the wowed experience, effort that needs to constantly evolve. Following are some key principles to focus on:
Start with a “people-focused” design. Most AI systems are used by people and their experience is critical in evaluating the impact of these systems. Having a design that provides three options, with good, better, and best answers will help users make an impactful decision. Even technically, it is easier to achieve good precision at a few answers than a single answer and limiting it to three makes it user friendly. Get early feedback and validation on the design from your actual users (preferably a diverse set of users) and ask them to test the live system in an iterative mode for better results.
Focus on key metrics. Work with your stakeholders to define multiple metrics that you would like to use to measure “what success looks like.” This will help you understand the tradeoffs between different types of issues and user experiences. Incorporate feedback from user surveys, as well as false positive and false negative results, and ensure that the key metrics are relevant to your AI system goals and objectives.
Understand the raw data. Losing focus on the raw data can lead to skewed results and loss of confidence on the machine learning models. If you do not understand the data, get help from business experts to gain a full understanding of the story the raw data is telling you. Analyze it to ensure that there are no missing values, incorrect labels, or typos and check that the sample contains the full spectrum of all users that you wish to analyze. Also, consider the relationship between data labels and values that you are trying to predict based on dependent data and ensure that there is no biased data (data favoring a particular result). While analyzing the raw data, you will get an understanding of the limitations of your data set and model. This will help you communicate the scope and limitations of your predictions based on the pattern of the data to your stakeholders.
Test it to the end of the world. Testing the model and its predictions thoroughly will help you clean up the chinks in the armor. Keep a clean data set as a master that can be reused for every round of testing and incorporate new and changing use cases on this data set. Engage users for iterative user testing during the development cycle and don’t forget to continue to monitor the model for issues. Also, factor in time for addressing the issues, both in the short term and long term.
I hope that you find these best practices useful in your journey with your AI system and machine learning.
Deven Shah has led large IT Application Development, QA, Automation and Engineering teams along with championing IT Strategy and Operations. He is an innovator at heart and dabbles in new technologies. He is an avid runner, biker, golfer and loves to paint. After the craziness of work, he volunteers at the Civil Air Patrol teaching cybersecurity to middle school and high school cadets.
Connect with the author
More About BizOps
If you’d like to learn more about Broadcom, and how its BizOps solutions can help your teams address the urgent imperatives you face today, be sure to visit Broadcom.
Reposted from my original post on LinkedIn dated 4/22/19.