download pdf

Self-driving vehicles: Safe, but not 100%
Autonomous, self-driving cars are being tested and trained extensively and have already covered thousands of miles of real road driving. Incidents are remarkably rare. However, any accidents – especially if they involve fatalities – are covered broadly in media all over the world, and consumers   wonder whether autonomous vehicles (AVs) are actually safe, and whether they should ever trust them. Experts agree that AVs do have the potential to benefit the world by increasing traffic efficiency, reducing pollution and eliminating up to 90% of traffic accidents – those that are caused by driver error, tiredness, drunkenness or other human factors. Though safety is constantly improving and injuries and deaths might be significantly reduced, crashes will never be completely avoidable. And any imminent crashes will require AVs to make difficult decisions.

How to react when a crash is imminent?
Imagine, as an example, situations as depicted in Figure 1. The AV may avoid harming several pedestrians by swerving and sacrificing a passerby (A), or the AV may be faced with the choice of sacrificing its own passenger to save one (B) or more (C) pedestrians.
Although these scenarios appear unlikely, even low-probability events are bound to occur with millions of AVs on the road. Furthermore, the tradeoffs involved in these scenarios will occur in much more frequent, but less extreme scenarios: instead of choosing between certain death, the car will need to choose between slightly increasing the risk toward one group rather than toward another. AV programming must include decision rules about what to do when these situations occur.

While a human driver has to spontaneously react in a split second, an autonomous vehicle needs to be programmed deliberately; someone needs to define the rules, well before AVs become a global commodity. The algorithms that control AVs will need to embed moral principles guiding their decisions in situations of unavoidable harm. But what is the right moral decision in such a case? How should AI be programmed to react in such an instant?  Manufacturers and regulators will need to accomplish three potentially incompatible objectives: consistency, avoiding public outrage, and not discouraging buyers. One step toward solving this problem is trying to learn how people feel about alternative decisions that self-driving vehicles’ AI might have to make. This was the purpose of our Moral Machine study (see Box 1).

Saving passengers or pedestrians?
Another online study among U.S. residents sheds more light on the complexity of the topic of AI-driven decision-making in dangerous situations. This study explored the trade-off between saving driver and passengers versus saving pedestrians and other road users – the dilemma illustrated in Figure 1. In principle, participants approved of utilitarian AVs minimizing the number of casualties. Their moral approval increased with the number of lives that could be saved. Participants’ approval of passenger sacrifice was even slightly positive when they had to imagine themselves and another person, particularly a family member, in the AV.
Consumers would like other people to buy AVs relying on such a utilitarian algorithm, but they would themselves prefer to ride in AVs that protect their passengers at all costs. Further, study participants disapproved of enforcing utilitarian regulations for AVs, and would be less willing to buy such AVs. Thus, the moral dilemma triggers a social dilemma that needs to be solved.