Week 06 — Bias in AI

Big question: Which intervention reaches the mechanism that produced the disparity?
Research lock: 2026-08-28

Why this week matters

Week 2 located bias across data work; Week 3 showed that metric selection is normative. This week integrates both with privacy and accountability. The aim is not another catalogue of biased systems. Students should learn to distinguish a disparity from its mechanism, select a mitigation that can reach that mechanism, and define evidence that the mitigation worked without creating a new harm.

Deeper teaching spine

1. Diagnose five mechanisms

• Structural/historical: the target reflects unequal access, enforcement, wealth, or opportunity.
• Representation: relevant people or conditions are missing or under-sampled.
• Measurement: labels or features mean different things across groups.
• Aggregation/model: one relationship or threshold is imposed where subgroup patterns differ.
• Deployment: interface, staff action, incentives, environment, or feedback changes the effect.

A sixth category—evaluation bias—appears when the test set, metric, subgroup definition, or time window cannot reveal the harm.

2. Match mitigation to mechanism

• Pre-processing: recollection, relabelling, reweighting, feature review, target redesign.
• In-processing: constraints, robust objectives, subgroup-aware learning.
• Post-processing: group thresholds, calibration, abstention, uncertainty routing.
• Workflow/institution: narrower use, human authority, appeal, resource change, vendor governance, compensation, or non-use.

Post-processing parity cannot repair a target that encodes unequal access. More representative data cannot justify an illegitimate use. Removing a sensitive attribute cannot remove proxies or structural pathways.

3. Intersectionality changes the test plan

Gender Shades is a model for evaluation design: aggregate performance and single-axis slices can hide the worst-served intersection. Students should pre-specify intersections connected to plausible harms, report denominators and uncertainty, and avoid an uncontrolled fishing expedition that produces unstable claims for tiny samples.

4. Speech recognition makes performance tangible

Koenecke et al. audited five commercial speech-recognition systems and reported substantially higher average word-error rates for Black speakers than white speakers in their study data. The case lets students examine dialect, geography, recording conditions, vendor comparison, and downstream stakes. A transcription error in casual dictation differs from one in employment, healthcare, education, or legal evidence.

5. Evaluate the intervention, not only the model

Define a causal prediction: “If we change X, metric Y for population Z should improve by amount/range R without degrading safety measure Q over window T.” Add process evidence: was the new workflow used, did people appeal, did staff override appropriately, and were burdens shifted elsewhere?

Case-study dossier

Obermeyer: change the target

The algorithm predicted cost, not illness. Unequal access made spending a systematically distorted proxy for need. Ask students to compare removing race, balancing samples, adding clinical variables, changing the target, or changing the resource-allocation policy. Which option reaches the mechanism?

Gender Shades: change the evaluation population

The audit's intervention was epistemic: build a dataset and intersectional analysis capable of showing what aggregate benchmarks concealed. Then ask what evidence would be needed to conclude a vendor fixed the problem in the field.

SafeRent: bias across a vendor–deployer workflow

Map applicant data, scoring logic, omitted voucher information, landlord decision, notice, and appeal. A mitigation could sit at multiple points. Students must select an owner and explain why their change would affect housing outcomes rather than dashboard parity alone.

Automated speech recognition: errors become institutional

Provide two hypothetical transcripts with the same semantic content but different error rates. Teams decide whether to use the system for meeting notes, call-centre performance scoring, clinical documentation, or courtroom captions. Require stakes-proportionate safeguards.

Mitigation worksheet

For any observed disparity, answer in order:

1. What exact outcome differs, with denominator and uncertainty?
2. Is the underlying construct valid for the decision?
3. Which causal pathways could produce the difference?
4. What evidence distinguishes those pathways?
5. Which actor controls the most direct intervention?
6. What new privacy, safety, or fairness risk does the intervention create?
7. What outcome and process evidence would count as improvement?
8. What threshold triggers pause or retirement?

Seminar activities

1. Mitigation auction: teams receive a limited budget and bid on data, target, model, evaluation, workflow, or appeal changes. They must defend causal leverage.
2. Aggregate illusion: reveal overall accuracy, then single-axis slices, then intersections. Students record when their deployment decision changes.
3. Sensitive attribute paradox: compare fairness measurement, model use, and prohibited decision use of protected attributes. Show why “we never collect it” can make disparity invisible.
4. Failure transfer: every proposed fix must name a group or value that could be made worse.

Visual evidence plan

Visual	Capture or local asset	Teaching use	Guardrail
Obermeyer result	public/courses/mai-105/evidence/obermeyer-health.jpg	Draw access → spending → target → allocation.	Race is not the causal biological variable.
Gender Shades	public/courses/mai-105/evidence/gender-shades.jpg	Reveal aggregate, gender, skin type, then intersectional results.	Historic vendor results are not current performance claims.
SafeRent	public/courses/mai-105/evidence/saferent-doj.jpg	Build the vendor–landlord–applicant decision chain.	Label allegations and settlement posture.
Lifecycle taxonomy	public/courses/mai-105/evidence/lifecycle-harms.jpg	Attach each proposed mitigation to a specific mechanism.	More than one mechanism can operate at once.
Speech-recognition audit	PNAS/PubMed record	Capture the study's group word-error rates and sample/method.	Do not generalize beyond systems, speakers, and conditions studied.

Reading and citation ledger

1. Obermeyer et al., “Dissecting racial bias in an algorithm used to manage the health of populations”.
2. Buolamwini & Gebru, Gender Shades.
3. Suresh & Guttag, lifecycle sources of harm.
4. NIST SP 1270, identifying and managing AI bias.
5. Koenecke et al., “Racial disparities in automated speech recognition”.
6. DOJ, Louis et al. v. SafeRent Solutions.
7. FTC, Rite Aid facial-recognition enforcement — a deployment and monitoring counterexample.

Watch list

• Require current re-testing before making claims about a vendor named in an older audit.
• Do not recommend group-specific processing without legal and domain review.
• Fairness improvement in a benchmark is not evidence of improved lived outcomes.