Explainable AI (XAI) refers to methods and design choices that help people understand why an AI system produced a particular output, such as a credit score, a fraud flag or a recommended next step. An explanation might show which inputs mattered most, which past examples the decision resembles, or what change in the inputs would have changed the result. The aim is for the people who use, oversee or are affected by an AI system to be able to check it, challenge it and trust it to the right degree.
At a glance
- XAI covers techniques that make a model’s outputs understandable to people, either by using simpler models or by explaining complex ones after the fact.
- Explanations range from simple models whose factors can be read directly to tools that estimate which inputs drove a particular result.
- Explanations for large language models are partial with current methods; a model’s stated reasoning may not match how it actually reached its answer, and showing its sources lets people check the evidence without explaining the decision.
- Explainability supports AI governance, bias testing, audits and some legal duties on automated decisions.
- It matters most where AI decisions affect people’s money, jobs, health or legal rights.
What problem it solves
Many AI models, especially those built with machine learning (ML), learn patterns from data that no person wrote down. They can be accurate yet opaque: the model says “decline” or “high risk” without saying why. That creates several problems for a business.
Staff may ignore a tool they can’t check, or follow it blindly. Customers and employees affected by a decision may be entitled to an explanation. Regulators, auditors and insurers increasingly ask how automated decisions are made and how bias is controlled. When a model is wrong, the team running it needs to see why in order to fix it. Explainable AI gives people a way to look inside the decision, or at least around it, so they can catch errors, spot unfair patterns and stand behind the outcome.
How it works
Interpretable models. For some tasks, the simplest route is a model that is understandable by nature, such as a decision tree, a scorecard or a linear model, where each factor’s contribution is visible. These can be less accurate than complex models on some problems, so the choice is a trade-off.
Post-hoc explanations. For complex models, separate techniques estimate which inputs most influenced a given output. Common approaches include feature attribution, which estimates how much each input moved one decision (for example, “income and payment history moved this score most”; SHAP and LIME are widely used methods of this kind), counterfactuals (“the result would change if X were different”), and highlighting the parts of an image or text that mattered. These are approximations of the model’s behaviour, not a full account of it, and different methods can disagree, so explanation methods should themselves be tested, for example by checking that they are stable and consistent with known model behaviour.
Explanations for generative AI. Explaining how a large language model (LLM) reaches a given answer is still largely a research problem. Asking a model to explain its reasoning can help a reviewer, but the explanation is itself generated text and may not reflect the model’s actual process. Many products also show the source documents an answer drew on. That is provenance, not explainability: it lets a person check the evidence and supports audit, but it does not explain the model’s decision process, and it does not prevent an AI hallucination.
Documentation and monitoring. Explainability also includes records about the model: what data trained it, what it is meant for, its known limits and how its performance and fairness are tracked over time.
When it matters for buyers
- When AI makes or shapes decisions about people. Lending, hiring, pricing, insurance, healthcare and access decisions draw the most scrutiny.
- When regulations apply. Frameworks such as the EU AI Act and the NIST AI Risk Management Framework (AI RMF) treat transparency and explainability as part of trustworthy AI; specific obligations depend on jurisdiction and use.
- When adoption stalls. Users who can see why a recommendation was made are more likely to use it well.
- When something goes wrong. Explanations and logs help you investigate a bad outcome and show what you did about it.
Explainability requirements usually sit inside a wider risk and compliance program; see our governance, risk and compliance options.
Questions to ask vendors
- What explanation does the product give for each output, and who can see it: administrators, end users, affected individuals?
- Are explanations built into the model or estimated afterward, and how reliable are they?
- For generative AI features, what can you show about how an output was produced, and, separately, does the product show the sources it used so users can check them?
- What documentation do you provide on training data, intended use and known limitations?
- How do you test for and report bias, and can we run our own tests?
- Can we export decision logs and explanations for audits or disputes?
- How do you support customers’ legal obligations on automated decisions in the jurisdictions where we operate?
How it differs from AI governance
AI governance is the organization-wide set of policies, roles and controls for how AI is chosen, used and monitored. Explainable AI is one technical capability that governance relies on: it supplies the evidence of why a model behaves as it does. A company can have governance policies that require explanations, but without explainable systems those policies are hard to meet. Explainability alone does not make AI use responsible; it has to be paired with oversight, testing and accountability.