Big question: What must an explanation let this person do?
Research lock: 2026-08-28
Why this week matters
Explanation is not a single output. A developer debugging a model, a regulator evaluating compliance, a clinician calibrating reliance, and an applicant contesting a denial need different evidence. A plausible heatmap can be unfaithful; a faithful global model description can be useless to an affected person. Start with audience and action, then select and test the explanation.
Deeper teaching spine
1. Separate four ideas
- Transparency: what exists, who made it, what data/process/version was used, and what happened.
- Interpretability: how a person can understand the model or relationship itself.
- Explainability: reasons or evidence accompanying an output or process.
- Contestability: information and procedure that let someone challenge and change a consequential result.
These support one another but are not synonyms.
2. Use NIST's four principles as a test
NIST proposes explanation, meaningfulness to the intended consumer, explanation accuracy, and knowledge limits. Apply all four to every technique. An explanation that looks intuitive but does not reflect the model fails accuracy; a mathematically faithful explanation that the recipient cannot act on fails meaningfulness.
3. Prefer interpretable models where stakes justify it
Rudin argues that high-stakes decisions should often use inherently interpretable models rather than explaining black boxes after the fact. Treat “accuracy requires complexity” as an empirical claim requiring comparison, not a law. Students should demand a benchmark between an interpretable candidate and a black box under the actual task and constraints.
4. Understand what local attributions say
LIME approximates local behaviour around an input; SHAP assigns additive feature contributions relative to a background expectation. Results depend on perturbations, feature dependence, background data, model access, and implementation. They are explanations of a model prediction under assumptions—not causal reasons in the world.
5. Test explanations adversarially
Adebayo et al. showed that some saliency methods can remain visually similar after model parameters or labels are randomized. Slack et al. demonstrated adversarial models that can fool perturbation-based LIME and SHAP explanations. Explanation systems therefore need validation: sensitivity, stability, faithfulness, robustness, human comprehension, and decision utility.
6. Counterfactuals need feasibility and recourse
“If income were $5,000 higher, approval would change” may be understandable but not actionable, causal, stable, or fair. A recourse design should consider immutable or protected attributes, feature dependencies, cost, time, uncertainty, repeated use, and whether the institution can change its own policy rather than shifting the burden to the person.
7. Mechanistic interpretability changes the object
Anthropic's circuit-tracing work uses attribution graphs to propose computational structures inside language models. This may generate testable hypotheses about model behaviour. It does not provide a complete causal account, guarantee coverage, or directly satisfy an affected person's need for recourse. Compare it with output-level, procedural, and institutional explanations.
Case-study dossier
Wolf versus husky: a useful local clue
The classic LIME example suggested a classifier relied on snowy backgrounds. Ask what this establishes: a local surrogate found evidence consistent with a spurious feature. Ask what it does not: global behaviour, causality in the world, fairness, or field reliability. Design a follow-up test set that manipulates snow and animal independently.
Saliency sanity checks: visual appeal is not faithfulness
Show original and randomized-model saliency maps without labels. If students cannot distinguish them, ask whether the explanation reflects learned parameters. This is a vivid example of evaluating the explainer as a model in its own right.
Credit denial: explanation versus recourse
The U.S. CFPB has stated that creditors using complex algorithms still must provide specific, accurate adverse-action reasons. Students compare a feature-attribution chart, a reason code, and a feasible counterfactual. Which satisfies legal notice, understanding, and ability to contest?
Explanation selection matrix
| Audience | Purpose | Candidate artifact | Core validation |
|---|---|---|---|
| Developer | Debug spurious behaviour | counterfactual test suite, slice errors, local/global attribution | changes when model/data changes |
| Regulator/auditor | Assess process and compliance | documentation, logs, subgroup results, decision rules | traceable and reproducible |
| Domain professional | Calibrate reliance | confidence, similar cases, limits, actionable factors | improves decisions without over-trust |
| Affected person | Understand and contest | specific reasons, evidence, feasible recourse, appeal | understandable and can change outcome |
| Executive | Accept residual risk | system boundary, scenarios, control evidence | calibrated aggregation, no hidden tails |
Visual evidence plan
| Visual | Capture target | Teaching use | Guardrail |
|---|---|---|---|
| NIST four principles | NISTIR 8312 | Reveal one principle at a time and test an explanation. | Principles are a framework, not a certification. |
| LIME wolf/husky | LIME paper | Crop the example and design a falsification test. | A local surrogate is not a causal proof. |
| SHAP attribution diagram | NeurIPS paper | Show base value → feature contributions → prediction. | State the background and feature-dependence assumptions. |
| Saliency randomization | NeurIPS 2018 paper PDF | Compare explanations before/after parameter randomization. | A failed necessary test does not rank every alternative. |
| Circuit tracing | Anthropic methods paper | Contrast internal computational hypotheses with user-facing explanation. | Treat graphs as partial, method-dependent evidence. |
| Adverse-action requirement | CFPB Circular 2022-03 | Crop the requirement for specific and accurate reasons. | Explain U.S. credit-law scope. |
Reading and citation ledger
- NIST, Four Principles of Explainable Artificial Intelligence.
- Rudin, “Stop explaining black box machine learning models for high stakes decisions”.
- Ribeiro, Singh & Guestrin, LIME.
- Lundberg & Lee, SHAP.
- Wachter, Mittelstadt & Russell, counterfactual explanations.
- Adebayo et al., “Sanity Checks for Saliency Maps”.
- Slack et al., “Fooling LIME and SHAP”.
- Anthropic, Circuit Tracing methods.
- CFPB Circular 2022-03.
Watch list
- Never call an attribution a cause without a causal design.
- Evaluate the explanation method on the actual model and use case.
- Ask what action the explanation enables and whether the institution offers real recourse.