To main content

Balancing Automation and Human Oversight: Design Implications for Safety-Critical Systems

Abstract

As advanced autonomous technologies and artificial intelligence (AI) proliferate across safety-critical sectors, they bring both unprecedented opportunities and significant challenges, often described as automation’s double-edged sword. Recent literature highlights the shift from a technocentric to a human-centric focus in designing human- automation interactive systems, aligning with the EU AI regulation's emphasis on Human Oversight. However, as Levels of Automation (LoA) and system complexity increase, maintaining human involvement, control, and the ability to intervene becomes increasingly difficult. Ensuring observability, predictability, and directability of autonomous agents is crucial to achieving transparency in design as a step towards meaningful human oversight. This paper examines the concept of human oversight, its implications for design, and its role in balancing automation’s advancements with the need for human control. Drawing from the MAS (Meaningful Human Control) project, we reviewed twelve articles that explicitly reference oversight, analyzing their contributions to human oversight design principles. Our findings reveal gaps and underscore the need for stronger integration of human oversight to ensure the safety and sustainability of advanced autonomous systems.
Read the publication

Category

Academic chapter

Language

English

Author(s)

  • Mina Saghafian
  • Ole Andreas Alsos
  • Jooyoung Park
  • Stine Thordarson Moltubakk
  • Lene Elisabeth Bertheussen
  • Stig Ole Johnsen

Affiliation

  • SINTEF Digital / Software Engineering, Safety and Security
  • Norwegian University of Science and Technology
  • Institute of Transport Economics

Date

09.06.2025

Year

2025

Publisher

Research Publishing Services

Book

Stavanger ESREL SRA-E 2025 Proceedings : 35th European Safety and Reliability Conference and the 33rd Society for Risk Analysis Europe Conference, 15-19 June 2025, Norway

ISBN

9789819432813

Page(s)

2455 - 2460

View this publication at Norwegian Research Information Repository