Big question: When an AI system causes harm, who knew, who controlled, who owed a duty, and who can repair it?
Research lock: 2026-08-28
Why this week matters
“The algorithm did it” hides a supply chain. “A human was in the loop” can hide a powerless operator. Responsibility becomes tractable when students separate causal contribution, knowledge, control, assigned role, moral duty, legal liability, and capacity to provide remedy. Those dimensions may point to different actors.
Deeper teaching spine
1. Build a responsibility map, not a blame list
For every actor—data supplier, model provider, integrator, deployer, operator, executive, auditor, regulator—record:
- What did they contribute causally?
- What did they know or reasonably should have known?
- What could they change, stop, or escalate?
- What duty came with their role?
- What evidence did they preserve?
- What remedy can they provide?
The map can assign overlapping responsibility without pretending every actor is equally responsible.
2. Detect moral crumple zones
Madeleine Elish's term describes situations in which a proximate human absorbs blame for a complex automated system despite limited meaningful control. An operator facing high alert volume, poor interface design, production pressure, and disabled safety functions may be causally involved without being the best prevention target.
3. Human oversight needs four properties
A meaningful overseer needs information, time, competence, and authority. Add incentives and an available alternative. A rubber-stamp screen or impossible alert queue does not transfer responsibility to a human simply because a button exists.
4. Accountability is an evidence system
Before deployment, define decision rights, logging, versioning, incident taxonomy, escalation, stop conditions, notification, investigation, correction, compensation, and recurrence prevention. Auditability after harm depends on choices made before harm.
5. Liability is not identical to ethics
Court decisions, product-liability rules, contracts, employment law, and regulator orders allocate legal duties differently. The EU Product Liability Directive explicitly brings software and AI within product concepts, but territorial and implementation details matter. Legal compliance may set a floor while ethical responsibility can extend further.
Case-study dossier
Uber ATG: safety authority before the crash
Use the NTSB record to reconstruct the chain: system classification, emergency-braking configuration, operator monitoring, safety-driver design, testing programme, safety culture, and oversight. Ask what change would have had the greatest preventative leverage one week before the crash.
Air Canada chatbot: channel ownership
The tribunal held the company responsible for information on its website. Students should design content ownership, knowledge-base updates, uncertainty handling, escalation, monitoring, and customer remedy for a travel chatbot. The lesson is not merely “chatbots can hallucinate”; it is that deployers create and govern an authoritative channel.
Workday: vendor or agent?
Use the preliminary court order and EEOC brief to examine how a vendor can participate in employment screening at scale. Assign contracting terms: permitted use, audit access, protected-group testing, logs, model-change notice, incident cooperation, appeal data, and indemnity. Then ask which duties cannot be outsourced by contract.
Tesla Autopilot recall remedy investigation
NHTSA opened a recall query to examine whether a software remedy was adequate. This is useful for distinguishing a disclosed fix from evidence that the fix changes field behaviour. Require a post-remedy evaluation plan: exposure, misuse, alerts, driver response, disengagement, crashes, and stop criteria.
Seminar activities
- Blame stack: students receive only a harmful output, then progressively reveal contracts, logs, warnings, and authority. They must revise assignments with evidence.
- Oversight stress test: cut the operator's time, information, authority, or alternative. Identify the point at which “human oversight” becomes ceremonial.
- Incident tabletop: a model update causes false denials. Teams act as vendor, deployer, regulator, and affected-person advocate during the first 24 hours.
- Remedy design: require individual correction, group-level review, compensation, notification, model/workflow repair, and evidence of non-recurrence.
Visual evidence plan
| Visual | Capture target | Teaching use | Guardrail |
|---|---|---|---|
| NTSB Uber findings | public/courses/mai-105/evidence/ntsb-uber.jpg | Colour-code technical, operational, organizational, and regulatory causes. | Do not collapse probable cause into a single operator action. |
| Air Canada tribunal paragraph | CanLII | Show deployer responsibility for its website channel. | It is a small-claims tribunal decision in a specific jurisdiction. |
| Moral crumple zones paper | DOI record | Pair the concept with a responsibility map. | The concept does not absolve proximate humans automatically. |
| Workday procedural record | public/courses/mai-105/evidence/mobley-workday.jpg | Mark allegations, procedural posture, and agency question. | Preliminary certification is not final liability. |
| NHTSA recall query | official PDF | Capture the investigation purpose and remedy-evaluation question. | Opening an investigation is not a final defect finding. |
| EU product-liability text | Directive (EU) 2024/2853 | Highlight software/AI treatment and responsible economic actors. | Explain transposition and territorial scope. |
Reading and citation ledger
- Madeleine Clare Elish, “Moral Crumple Zones”.
- NTSB, Tempe automated-vehicle crash investigation.
- Moffatt v. Air Canada, 2024 BCCRT 149.
- Mobley v. Workday, 2025 order and EEOC amicus brief.
- NHTSA, Recall Query RQ24-009.
- Directive (EU) 2024/2853 on liability for defective products.
- Canada, Directive on Automated Decision-Making and Algorithmic Impact Assessment tool — a concrete model of pre-deployment impact tiering and obligations.
Watch list
- Always label procedural posture: allegation, investigation, preliminary order, settlement, final judgment, recall, and remedy are different.
- Contract allocation does not necessarily eliminate statutory or moral duties.
- Incident response should include affected-person communication and correction, not only engineering rollback.