As the use of machine learning systems for decision-making support becomes mainstream, it has become more important to explain the outputs of these systems. ML models can be tasked with relatively low-risk tasks like product recommendations and churn prediction, as well as critical tasks like ballistics trajectory modeling and cancer screening. Many of these highly accurate ML systems have been constructed as a ‘black box’—that is, the internal logic is hidden from the developers of the system.
End users, business stakeholders, and governing bodies all need transparency into the results of machine learning model outputs. With so much potentially at stake—whether it’s a patient who is being screened for disease, a passenger in an autonomous vehicle or a small business owner applying for a loan—organizations adopting AI are under more pressure to lift the veil on how the AI models under their management make decisions.
Explainable AI (XAI) is a set of tools and methods that attempt to help humans understand the outputs of machine learning models.
For an example of how obscure a machine learning prediction can be, consider how deep learning works under the hood. AI systems that involve computer vision, NLP and robotics typically use deep learning in which the model learns to identify patterns based on very large training data sets. Similar to how a human brain processes thoughts, it can be difficult or impossible to determine how a deep learning algorithm makes a prediction.
Whether in a critical environments like healthcare or military, or a business setting like finance, users, stakeholders and governing bodies need to have trust in AI systems.
XAI is a very active field of study, due in part to the tremendous success with deep learning models for various tasks. There are many, many approaches to AI explainability, all with various dependencies and limitations.
Most explainable AI methods fall into one of two categories:
Local interpretability, or individual decision-making, is the most well understood area of XAI. This consists of providing an explanation when an individual prediction has been generated. LIME and SHAP provide mathematical answers to that question, and those answers can be presented to data scientists, end users and other stakeholders, which is an important requirement for organizations implementing explainable AI principles.
LIME (Local Interpretable Model-agnostic Explanations) is a popular open source API, and available in R and Python. LIME generates an explanation by “approximating the underlying model by an interpretable one, learned on perturbations of the original instance (e.g., removing words or hiding parts of the image”.
SHAP is a popular open source framework based on a game theoretic approach that connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions. Like LIME, SHAP is model agnostic. It can also be implemented with NVIDIA GPUs using RAPIDS, which is useful for deep learning use cases.
ML/DL models learn from the past, in much the same way humans do. The “past” for ML/DL models is the labeled data: that is, the data on which it has been trained. If the data is big enough to cover all the scenarios, and it is learnable, the model will learn from it and hopefully make accurate predictions in future situations.
Being able to analyze the data before training, and track the training experiments is very important for explainability purposes, and a must-have tool in every data scientist’s arsenal.
Iguazio uses MLRun, an open source MLOps orchestration framework, to analyze, explore and track datasets and experiments to yield the best model possible, so even if the model is a black box, the road to building it won’t be as dark and may even open the box a little bit. And as an open platform, MLRun users can use SHAP and LIME, and store results as an artifact in the system.
