Big question: When fairness metrics disagree, who chooses the error that matters?
Research lock: 2026-08-28
Why this week matters
Fairness is not a property a model simply has. It is a relationship among a decision, groups, outcomes, measurement choices, time, and institutional duties. Several desirable statistical criteria cannot generally be satisfied together when base rates differ. That is not a reason to abandon measurement; it is a reason to make the normative choice explicit and design recourse around it.
Deeper teaching spine
1. Build one confusion matrix before naming metrics
Use a lending, screening, or health example with 100 people. Mark true positives, false positives, true negatives, and false negatives for two groups. Students should physically identify who experiences each cell before calculating:
- Demographic parity: selection rates are equal.
- Equal opportunity: true-positive rates are equal.
- Equalized odds: both true-positive and false-positive rates are equal.
- Predictive parity: positive predictions have equal precision.
- Calibration: among people with the same score, outcome frequencies are similar.
Then ask which human harm each metric makes salient. A false positive in criminal risk, a false negative in disease screening, and a rejected qualified applicant are not interchangeable.
2. Impossibility is conditional, not mystical
Chouldechova and Kleinberg, Mullainathan, and Raghavan show related incompatibilities among calibration and error-rate balance under unequal base rates, except in special cases. Students should state the assumptions and avoid the lazy conclusion that “fairness is impossible.” Some combinations are incompatible; institutions still must choose a decision rule, justify it, monitor it, and provide remedy.
3. Measurement can dominate optimization
Obermeyer et al. found a widely used population-health algorithm exhibited racial bias because cost was used as a proxy for health need. Black patients with the same cost were often sicker because unequal access shaped spending. Removing race would not repair that target. Changing what the system predicts is more causally direct than optimizing parity on a broken label.
4. A metric needs a decision process
A defensible system includes the use purpose, metric choice, threshold, subgroup definitions, uncertainty, data period, appeal route, human authority, monitoring cadence, and change trigger. Metric dashboards without a process can turn contested choices into technical décor.
Case-study dossier
COMPAS: both statistics and stakes
Reproduce a small version of the dispute. One group argues from error-rate disparity; another from calibration. A third represents defendants and asks whether the score should influence liberty at all. The goal is to show that a statistically coherent defense does not automatically justify the use.
Healthcare cost proxy: prediction can be accurate and wrong for purpose
Draw a causal diagram: race and structural inequality influence access; access influences spending; spending becomes the target; the system allocates extra care. Ask students where an intervention changes the causal mechanism instead of the surface metric.
SafeRent: omitted information and housing access
Use the DOJ case page and settlement materials to examine tenant-screening scores, voucher income, risk proxies, vendor–landlord roles, notice, and recourse. Carefully distinguish plaintiffs' allegations, the government's participation, and resolved obligations.
Workday: vendor participation at scale
The 2025 preliminary collective-certification order is not a final liability ruling. Its teaching value is the agency question: when a vendor's system participates in repeated employment decisions, who is acting for whom? Pair the court record with the EEOC's amicus position and ask what logs and audit rights a deployer would need.
Worked example
Two groups each have 100 applicants. Group A has 60 qualified applicants; Group B has 30. A model approves 48 qualified and 10 unqualified people in A, and 24 qualified and 5 unqualified people in B.
- Both groups have an 80% true-positive rate: equal opportunity holds in this sample.
- Approval rates differ: 58% versus 29%, so demographic parity does not hold.
- Precision is identical: 48/58 and 24/29 are both about 82.8%.
Now change only Group B's false positives from 5 to 15. Equal opportunity still holds, but precision and false-positive rates change. Ask which error deserves priority for this particular decision and why. Then require an appeal design, not just a metric.
Seminar activities
- Metric hearing: groups represent affected people, deployer, regulator, and auditor. Each may nominate one primary metric and one constraint.
- Fairness specification: write a one-page decision policy with target, population, metric, threshold, measurement window, owner, and remedy.
- Uncertainty label: add confidence intervals and small-subgroup warnings to a polished dashboard. Ask whether the decision should pause.
- Counterfactual repair: compare changing the target, adding data, reweighting, thresholding, workflow redesign, and stopping the use.
Visual evidence plan
| Visual | Capture or local asset | Teaching use | Guardrail |
|---|---|---|---|
| COMPAS investigation | public/courses/mai-105/evidence/propublica-machine-bias.jpg and compas-methodology.jpg | Separate the claim from cohort construction and calculation. | Include Northpointe's calibration counterargument in discussion. |
| Fairness trade-offs | public/courses/mai-105/evidence/fair-prediction.jpg and kleinberg-fairness.jpg | Highlight the conditions, not merely the phrase “impossibility theorem.” | Do not infer that every fairness goal is incompatible. |
| Equalized odds paper | public/courses/mai-105/evidence/equalized-odds.jpg | Connect a formula to the false-positive and false-negative experiences. | A metric is not a complete ethical policy. |
| Healthcare proxy result | public/courses/mai-105/evidence/obermeyer-health.jpg | Annotate the difference between spending and need. | Avoid presenting race as a biological cause. |
| SafeRent record | public/courses/mai-105/evidence/saferent-doj.jpg | Trace score, landlord action, notice, and remedy. | Label procedural status and allegations accurately. |
| Workday records | public/courses/mai-105/evidence/mobley-workday.jpg and workday-eeoc.jpg | Compare court posture with the regulator's agency argument. | Preliminary certification is not a merits judgment. |
Reading and citation ledger
- Hardt, Price & Srebro, “Equality of Opportunity in Supervised Learning”.
- Chouldechova, “Fair prediction with disparate impact”.
- Kleinberg, Mullainathan & Raghavan, “Inherent Trade-Offs in the Fair Determination of Risk Scores”.
- ProPublica COMPAS methodology.
- Obermeyer et al., racial bias in population-health management.
- Buolamwini & Gebru, Gender Shades.
- DOJ, Louis et al. v. SafeRent Solutions.
- Mobley v. Workday, 2025 preliminary certification order and EEOC amicus brief.
- NYC Department of Consumer and Worker Protection, Local Law 144 / automated employment decision tools — a concrete audit-and-notice regime to critique.
Watch list
- Update court and enforcement status immediately before teaching active cases.
- Never compare raw rates without checking denominators and data windows.
- Ask whether protected-group categories themselves are well measured, lawful to use, and meaningful in context.