Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction
As debates about the policy and ethical implications of AI systems grow, it will be increasingly important to accurately locate who is responsible when agency is distributed in a system and control over an action is mediated through time and space. Analyzing several high-profile accidents involving complex and automated socio-technical systems and the media coverage that surrounded them, I introduce the concept of a moral crumple zone to describe how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system. Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions. While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator. The concept is both a challenge to and an opportunity for the design and regulation of human-robot systems. At stake in articulating moral crumple zones is not only the misattribution of responsibility but also the ways in which new forms of consumer and worker harm may develop in new complex, automated, or purported autonomous technologies.
ABC 15 staff. 2018. “Self-driving Uber car hits, kills pedestrian in Tempe.” ABC 15 Arizona, Mar 19. Accessed 1 July 2018. https://www.abc15.com/news/region-southeast-valley/tempe/tempe-police-investigating-self-driving-uber-car-involved-in-crash-overnight.
AP, 1986. “Fatal Radiation Dose in Therapy Attributed to Computer Mistake.” New York Times 21 June, pg. 50. Accessed 1 July 2018. https://www.nytimes.com/1986/06/21/us/fatal-radiation-dose-in-therapy-attributed-to-computer-mistake.html.
Bainbridge, L. 1983. “Ironies of Automation.” Automatica 19:775-779.
Barnaby, K.C. 1968. Some Ship Disasters and Their Causes. Cranbury N.J.: A.S. Barnes.
BEA, 2012a. Final Report on the Accident on 1st June 2009. Accessed 1 July 2018. https://www.bea.aero/en/investigation-reports/notified-events/detail/event/accident-de-lairbus-a330-203-immatricule-f-gzcp-et-exploite-par-air-france-survenu-le-01062009-da/.
BEA. 2012b. Final Report on the Accident on 1st June 2009: Appendix 1:CVR Transcript (EN). Accessed 1 July 2018. https://www.bea.aero/uploads/tx_elyextendttnews/annexe.01.en_03.pdf.
Behr, P., 2009. "Three Mile Island Still Haunts U.S. Nuclear Power Industry," New York Times, March 27. Accessed 1 July 2018. http://www.nytimes.com/gwire/2009/03/27/27greenwire-three-mile-island-still-haunts-us-reactor-indu-10327.html.
Benjamin, S., 1979. “Human Error Cited in 3-mile Accident.” Boston Globe. May 12, pg 5.
Bryson, J. J., M. E. Diamantis, T.D. Grant. 2017. “Of, for, and by the people: the legal lacuna of synthetic persons.” Artificial Intelligence Law 25: 273-291.
Calo, R 2016. Robots in American Law. We Robot 2016 presentation. University of Washington School of Law Research Paper 2016-04. Accessed 17 Feb 2019. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2737598.
Calo, R. A. Michael Froomkin, and Ian Kerr. eds. 2016. Robot Law. Northampton: MA: Elgar Publishing.
Casner, S.M. and J. Schooler, 2014. “Thoughts in Flight: Automation Use and Pilots’ Task-Related and Task-Unrelated Thought.” Human Factors 56(3,):433-422.
Chui, M. J. Manyika and M. Miremadi, 2015. “Four Fundamentals of Workplace Automation.” McKinsey Quarterly November. Accessed 1 July 2018. http://www.mckinsey.com/business-functions/business-technology/our-insights/four-fundamentals-of-workplace-automation (accessed 1/2/2016);
CNN, 2012. “Final Air France crash report says pilots failed to react swiftly” CNN.com July 5. Accessed 1 July 2018. http://www.cnn.com/2012/07/05/world/europe/france-air-crash-report/.
Code of Federal Regulations, 14 CFR 91.3.
Coeckelbergh, M., 2011. “Moral Responsibility, Technology, and Experiences of the Tragic: From Kierkegaard to Offshore Engineering.” Science and Engineering Ethics 18(1):35-48.
Cooling, J.E. and P. V. Herbers. 1983. “Considerations in Autopilot Litigation.” Journal of Air Law and Commerce 48: 693-723.
Christiansen, D. 1979. “TMI and the press.” IEEE Spectrum November: 92-95.
Cummings, M.L. 2006. “Automation and accountability in decision support system interface design.” Journal of Technology Studies 32: 23-31.
Davenport, T. and Kirby, J. 2015. “Beyond Automation.” Harvard Business Review 93 (6). Accessed 1 July 2018. https://hbr.org/2015/06/beyond-automation.
Davies, Alex. 2016. “Google’s self-driving car caused its first crash.”
Wired Feb 29. Accessed 17 Feb 2019 https://www.wired.com/2016/02/googles-self-driving-car-may-caused-first-crash/
Draper, C.S, 1955. “Flight Control” 43rd Wilbur Wright Memorial Lecture. Journal of the Royal Aeronautical Society 59 July: 451-478
Elish, M.C. and Tim Hwang. 2015. “Praise the Machine! Punish the Human! The Contradictory History of Accountability in Automated
Aviation.” May 18. Data & Society Working Paper. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2720477
FAA. 2011. AC 23-17C - Systems and Equipment Guide for Certification of Part 23 Airplanes and Airships. Accessed 1 July 2018. https://www.faa.gov/regulations_policies/advisory_circulars/index.cfm/go/document.information/documentID/1019689 .
Ford, D.,1981. “A Reporter at Large: Three Mile Island.” The New Yorker Apr 6: 49-120.
Friedman, Batya, ed. Human Values and the Design of Computer Technology. New York: University of Cambridge Press.
Froomkin, M., 1995. “The Metaphor is the Key: Cryptography, the Clipper Chip, and the Constitution.” U. PA. L. Rev. 143(3): 709-862.
Graham, K. “Of Frightened Horses and Autonomous Vehicles: Tort Law and its Assimilation of Innovations,” Santa Clara L. Rev. 52: 1241.
Jones, M. L. 2015. “The Ironies of Automation Law: Tying Policy Knots with Fair Automation Practice Principles. Vand. J. Ent. & Tech. 18: 77-134.
Kovach, S.. 2017. “Google quietly stopped publishing monthly accident reports for its self driving cars.” Business Insider Jan 18. Access 17 Feb 2019. https://www.businessinsider.com/waymo-ends-publishing-self-driving-car-accident-reports-website-2017-1
Kalra, N. and S.M. Paddock. 2016. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability. RAND Corporation, Santa Monica, CA. Accessed 1 July 2018. www.rand.org/t/RR1478.
Kemeny, J.G. et al., 1979. "Report of the President's Commission on the Accident at Three Mile Island," U.S. Government Printing Office, 0-303-300, October.
Langewiesche, W., 2014. “The Human Factor.” Vanity Fair September. Accessed 1 July 2018. http://www.vanityfair.com/news/business/2014/10/air-france-flight-447-crash.
Lederer, J. 1974. “Human Factors & Pilot Error,” Air Line Pilot July:13-14.
Lee, T. 2018. “NTSB: Uber’s sensors worked; its software utterly failed in fatal crash.” ArtsTechnica, June 24.
https://arstechnica.com/cars/2018/05/emergency-brakes-were-disabled-by-ubers-self-driving-software-ntsb-says/ (accessed 7/1/2018).
Leveen, S.A., 1982. “Cockpit Controversy: The Social Context of Automation in Modern Airlines.” Ph.D. Dissertation, Department of Science and Technology Studies, Cornell University.
Leveson, N.G. and C.S. Turner, 1993. “An Investigation of the Therac-25 Accidents,” IEEE, July:18-41.
Leveson, N. G.. 2011. Engineering a Safer World: Systems Thinking Applied to Safety. Cambridge, MA: MIT Press.
Lin, P., K. Abney, and G. A. Bekey, eds. 2014. Robot ethics: The ethical and social implications of robotics. Cambridge, MA: MIT Press.
Markoff, J., 2016. “Google Car Exposes Regulatory Divide on Computers as Drivers.” New York Times, Feb10. http://www.nytimes.com/2016/02/11/technology/nhtsa-blurs-the-line-between-human-and-computer-drivers.html?smid=tw-share&_r=0 (accessed 2/11/2016).
Mindell, D. A. 2015. Our Robots, Ourselves. New York: Viking.
Nissenbaum. H., 1996. “Accountability in a Computerized Society.”Science and Engineering Ethics 2(1):25-42.
NTSB. 2018. HWY18MH010 – Preliminary Report. National Transportation Safety Board (NTSB), May 24. Accessed 1 July 2018. https://goo.gl/2C6ZCH.
O’Brien, S.A. 2018. “Uber operator in fatal self-driving vehicle crash was likely streaming ‘The Voice.” CNN, Jun 22. Accessed 1 July 2018. http://money.cnn.com/2018/06/22/technology/uber-self-driving-crash-police-report/index.html.
Omang, J. 1979. “’Nuggets’: A Collection of Nuclear Glitches.” Washington Post, Feb 10. Accessed 17 Feb 2019. https://www.washingtonpost.com/archive/politics/1979/02/10/nuggets-a-collection-of-nuclear-glitches/a48dbfae-d6d8-45ef-964c-a433a5f6bdf6/.
Oslund, J., 1986. “NWA Airbus 320s to be most advanced jets ever.” Minneapolis Star Tribune. 9 Oct.
Parasuraman, R. and V. Riley, 1997. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors June 39(2): 230-253.
Parasuraman, R., T.B. Sheridan, C.D. Wickens. 2000, “A Model for Types and Levels of Human Interaction with Automation.” IEEE Transactions on Systems, Man and Cybernetics, volume 30, issue 3 (May) pp. 286-297.
Perrow, C.1984. Normal Accidents: Living with High-Risk Technologies. New York: Basic Books.
PBS, 1999. “Meltdown at Three Mile Island.” PBS American Experience. Chana Gazit, dir. WGBH Boston.
Ramsey, M. 2015. “Who’s Responsible When a Driverless Car Crashes? Tesla’s Got an Idea.” Wall Street Journal, May 13. Accessed 1 July 2018. http://www.wsj.com/articles/tesla-electric-cars-soon-to-sport-autopilot-functions-such-as-passing-other-vehicles-1431532720.
Roscoe, A.H., 1992. Workload in the glass cockpit. Flight safety digest. Alexandria, VA: Flight Safety Foundation
Rubinstein, E. 1979. “The accident that shouldn’t have happened.” IEEE Spectrum November: 33-57.
Said, C. 2018. “Exclusive: Temple Police chief says early probe shows no fault by Uber.” San Francisco Chronicle March 26. Accessed 1 July 2018. https://www.sfchronicle.com/business/article/Exclusive-Tempe-police-chief-says-early-probe-12765481.php.
Sarter, N.B., D.D. Woods, and C.E. Billings, 1997. “Automation Surprises,” In: G. Salvendy, ed. Handbook of Human Factors & Ergonomics 2nd ed, New York: Wiley.
Schoettle, B. and M. Sivak, 2015. A Preliminary Analysis of Real-World Crashes Involving Self-Driving Vehicles. October. University of Michigan: Transportation Research Institute. UMTRI-2015-34. Accessed 1 July 2018. http://www.umich.edu/~umtriswt/PDF/UMTRI-2015-34_Abstract_English.pdf.
Sheridan, T. 1992. Telerobotics, automation and supervisory control. Cambridge, MA: MIT Press.
Sheridan, T. B. and R. Parasuraman. “Human-Automation Interaction.” Review of Human Factors and Ergonomics 1(1): 89-129.
Somerville, H. and D. Shepard. 2018. “Uber car’s ‘safety’ driver streamed TV show before fatal crash: police.” Reuters, June 22. Accessed 1 July 2018. https://www.reuters.com/article/us-uber-selfdriving-crash/uber-cars-safety-driver-streamed-tv-show-before-fatal-crash-police-idUSKBN1JI0LB.
Sorokanich, B. 2015. “Tesla Autopilot First Ride.” Road & Track, Oct 14. Accessed 1 July 2018. http://www.roadandtrack.com/new-cars/car-technology/news/a27044/tesla-autopilot-first-ride-almost-as-good-as-a-new-york-driver/.
Stilgoe, J. 2017. “Machine learning, social learning and the governance of self-driving cars.” Social Studies of Science 48(1), 25–56. https://doi.org/10.1177/0306312717741687
Toth, R. 1979. “Nuclear Accident Blamed Primarily on Human Error.” Los Angeles Times Apr 11, pg. 1
Umson, C., 2016. Letter. NHTSA. Accessed 1 July 2018. http://isearch.nhtsa.gov/files/Google%20--%20compiled%20response%20to%2012%20Nov%20%2015%20interp%20request%20--%204%20Feb%2016%20final.htm
Vaughan, D. 1996. The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA. Chicago, IL: University of Chicago Press.
Wakabayashi, D. 2018. “Self-Driving Uber Car Kills Pedestrian in Arizona, Where Robots Roam.” The New York Times, Mar 19. Accessed 1 July 2018.
Walker, J. S. 2004. Three Mile Island: A Nuclear Crisis in Historical Perspective. Berkeley, CA: University of California Press.
Walker, J. S. 2009. “Documenting Three Mile Island.” Bulletin of the Atomic Scientists. March 18.
https://thebulletin.org/2009/03/documenting-three-mile-island-2/ Accessed Feb 17 2019.
Weiner, E.L., 1989. Human factors of advanced technology (“glass cockpit”) transport aircraft (NASA Contractor Report 177528). Moffett Field, CA: NASA Ames Research Center.
Copyright (c) 2019 Madeleine Clare Elish
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors of all content published in ESTS retain the copyright to their work, and agree to license them under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. Please read our open access policy for more information.