AI Products and Human Oversight

Suits against AI chatbot and companion makers are pleaded as design defect and failure to warn and met with a misuse defense, and suits over automated insurance and hiring decisions ask whether a person truly reviewed them. Dr. Sawyer, a human factors scientist who studies trust in and reliance on automation, addresses how human-like design shapes reliance, what a warning does over a long conversation, which uses were foreseeable, and whether human review could work at the pace set for it.

The cases include self-harm and wrongful-death claims against chatbot and companion apps, where a federal court has let product liability claims proceed on the view that a companion chatbot app is a product, and claims that the people reviewing automated insurance and hiring decisions could not meaningfully check the system. Regulators have begun to define the terms: companion chatbots in California’s SB 243, dark patterns in the Federal Trade Commission’s 2022 staff report, Bringing Dark Patterns to Light, and human oversight and automation bias for high-risk systems, such as hiring tools, in Article 14 of the EU AI Act.

He is first author of the Handbook of Human Factors and Ergonomics chapter on human factors in the design of automation, autonomy, and artificial intelligence, and teaches Human-AI Safety Engineering at UCF. His Human Factors Prize-winning research shows why people who watch for rare problems miss them, the central difficulty in reviewing automated decisions, and his driving studies link trust in technology to where people look. He has twice been selected to present in the Briefings at Black Hat USA on how AI systems influence the people who use them. The methods are controlled studies of trust, reliance, and detection, and eye tracking.