Big question: If the model is a mirror of the data, who curated the reflection?
Research lock: 2026-08-28
Why this week matters
“Biased data” is too vague to diagnose or repair anything. Harm can originate in history, problem formulation, sampling, measurement, labels, aggregation, evaluation, deployment, or a feedback loop. The same observed disparity can require a new target, a new collection process, subgroup evaluation, workflow change, compensation, deletion, or abandoning the system. Students need a causal account before choosing a mitigation.
Deeper teaching spine
1. Start before collection
Ask what construct the organization actually cares about and what proxy it can cheaply observe. “Health need” becomes past healthcare spending; “job performance” becomes manager ratings; “risk” becomes an arrest or default label. A model can accurately predict a proxy and still be invalid for the intended decision.
2. Categories are interventions
Datasets do not discover a natural inventory of the world. Designers choose categories, sources, annotation rules, disagreement procedures, and exclusions. Excavating AI is useful because it connects ImageNet's person labels to an inherited classification genealogy. The ImageNet remediation statement then turns criticism into an operational question: what can be removed, relabelled, audited, or retired after downstream copies already exist?
3. Consent is contextual, not merely public/private
Public availability does not automatically authorize biometric identification, model training, or unlimited reuse. Contextual integrity asks whether the actors, attributes, purposes, and transmission principles fit the social context. Clearview AI makes the difference concrete: billions of publicly accessible images were reorganized into a searchable biometric capability that the original contexts did not provide.
4. Documentation should create boundaries
A datasheet is useful only when it changes behaviour: approved uses, prohibited uses, population limits, known gaps, provenance, consent, collection period, labelling process, maintenance owner, deletion route, and triggers for re-evaluation. Have students reject a deployment that violates their own datasheet rather than treating documentation as a compliance souvenir.
5. Remedy must follow derivatives
The FTC's Everalbum settlement is the sharpest teaching case: remedy reached not only improperly retained photos and videos, but models or algorithms developed with them. It forces the class to ask when deletion of source data is insufficient, and what technical evidence could establish that a derivative has been retrained or destroyed.
Case-study dossier
Microsoft Tay: interaction data is an attack surface
Microsoft's first-party postmortem identifies coordinated abuse and the need for safeguards. Decompose the failure into training data, online learning or response generation, adversarial interaction, product constraints, monitoring, escalation, and shutdown authority. “The internet taught it racism” is not an adequate causal explanation.
ImageNet person categories: taxonomy as power
Students compare the critical visual investigation with ImageNet's remediation. Ask who had standing to object, which categories were unsafe, what happened to models trained on prior versions, and whether removal erases research value or prevents continuing harm.
Rite Aid: the deployment completed the harm
The FTC alleged that facial-recognition alerts produced false matches and that inadequate testing, monitoring, notice, staff practices, and vendor oversight exposed consumers to accusation and humiliation. Build a timeline from image capture to employee action. The lesson is that a false match becomes consequential through a workflow.
Clearview AI: a new capability from old data
The Canadian privacy regulators' joint investigation is a strong local record for consent, appropriate purpose, scraping, biometric sensitivity, and deletion. Ask whether the relevant data object is a photo, a face template, an identity link, the search index, or all four.
Worked examples
Proxy audit: “employee potential”
- Define the construct in plain language.
- List observable proxies: promotion, manager rating, retention, sales, peer feedback.
- Identify institutional processes shaping each proxy.
- Name groups for whom the proxy–construct relationship may differ.
- State what decision the prediction will trigger.
- Decide whether a better measurement, a narrower claim, or no model is warranted.
Minimal datasheet exercise
Teams receive an imagined customer-support dataset. Require: origin, consent, unit of observation, time window, language distribution, sensitive fields, annotation instructions, disagreement rate, missingness, intended use, prohibited use, maintenance owner, deletion procedure, and a “do not deploy if” condition.
Remedy ladder
Order possible remedies from least to most structural: disclosure, monitoring, threshold change, relabelling, recollection, target change, workflow redesign, access/appeal, compensation, source deletion, derivative-model deletion, and system retirement. Match each to a causal mechanism.
Visual evidence plan
| Visual | Capture or local asset | Teaching use | Guardrail |
|---|---|---|---|
| Lifecycle harms taxonomy | public/courses/mai-105/evidence/lifecycle-harms.jpg | Mark where each harm enters and which intervention can reach it. | Categories overlap; do not force one cause per incident. |
| Dataset datasheet | public/courses/mai-105/evidence/datasheets-for-datasets.jpg | Turn a paper into a deploy/no-deploy artifact. | Documentation quality is not data quality. |
| Excavating AI | public/courses/mai-105/evidence/excavating-ai.jpg | Reveal the institutional history behind labels. | Use sensitive examples sparingly and with purpose. |
| ImageNet remediation | public/courses/mai-105/evidence/imagenet-remediation.jpg | Compare criticism with a first-party repair statement. | Remediation does not establish that downstream harms vanished. |
| FTC Everalbum order | public/courses/mai-105/evidence/everalbum-ftc.jpg | Highlight deletion reaching derived models. | Distinguish settlement obligations from a universal legal rule. |
| Data-to-decision pipeline | public/courses/mai-105/week-2/data-decision-pipeline.webp | Let students attach an owner, evidence, and remedy at every handoff. | Add the institution's incentive and affected person's appeal path. |
Reading and citation ledger
- Suresh & Guttag, “A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle”.
- Gebru et al., “Datasheets for Datasets”.
- NIST SP 1270, Managing Bias in AI — socio-technical rather than dataset-only framing.
- Crawford & Paglen, Excavating AI and ImageNet's 2019 remediation statement.
- Microsoft, “Learning from Tay's introduction” — first-party postmortem; supplement with independent analysis if making broader claims.
- FTC, Everalbum settlement.
- FTC, Rite Aid facial-recognition enforcement.
- Office of the Privacy Commissioner of Canada, joint Clearview AI findings.
- Mitchell et al., “Model Cards for Model Reporting” — pair dataset documentation with model-level intended use, performance, and limitations.
Watch list
- Check whether a data-remedy case is an allegation, proposed order, settlement, or final judgment before describing its force.
- For synthetic data, demand a provenance account. Generation can reproduce upstream categories while making lineage less legible.
- “Publicly available” is a factual description of access, not a complete ethical or legal consent analysis.