To main content

Toadstool: a dataset for training emotional intelligent machines playing Super Mario Bros

Abstract

Games are often defined as engines of experience, and they are heavily relying on emotions, they arouse in players. In this paper, we present a dataset called Toadstool as well as a reproducible methodology to extend on the dataset. The dataset consists of video, sensor, and demographic data collected from ten participants playing Super Mario Bros, an iconic and famous video game. The sensor data is collected through an Empatica E4 wristband, which provides highquality measurements and is graded as a medical device. In addition to the dataset and the methodology for data collection, we present a set of baseline experiments which show that we can use video game frames together with the facial expressions to predict the blood volume pulse of the person playing Super Mario Bros. With the dataset and the collection methodology we aim to contribute to research on emotionally aware machine learning algorithms, focusing on reinforcement learning and multimodal data fusion. We believe that the presented dataset can be interesting for a manifold of researchers to explore exciting new interdisciplinary questions.
Read the publication

Category

Academic chapter

Language

English

Author(s)

  • Henrik Svoren
  • Vajira Lasantha Bandara Thambawita
  • Pål Halvorsen
  • Petter Jakobsen
  • Enrique Garcia-Ceja
  • Farzan Majeed Noori
  • Hugo Lewi Hammer
  • Mathias Lux
  • Michael Riegler
  • Steven Hicks

Affiliation

  • SINTEF Digital / Sustainable Communication Technologies
  • University of Klagenfurt (AAU)
  • University of Bergen
  • University of Oslo
  • Bergen Hospital Trust - Haukeland University Hospital
  • Simula Metropolitan Center for Digital Engineering
  • OsloMet - Oslo Metropolitan University

Year

2020

Publisher

Association for Computing Machinery (ACM)

Book

MMSys '20: Proceedings of the 11th ACM Multimedia Systems Conference

ISBN

9781450368452

Page(s)

309 - 314

View this publication at Norwegian Research Information Repository