Abstract
Human activity recognition (HAR) involves using sensors to collect human data, which is then analyzed to identify their activities. HAR has numerous applications across various fields. One significant area is human-robot interaction (HRI). Recognizing user activities provides robots with valuable insights into the user’s status, enhancing the efficiency of HRI. However, developing HAR models presents privacy challenges due to the collection of user data, especially in robots, which are often equipped with diverse sensors. This paper proposes privacy-preserving HAR methods within the context of HRI.The paper investigates using privacy-preserving sensors, with a particular focus on 3D Lidar, to develop HAR models. It explores the integration of 3D Lidar with other sensors, including user-wearable sensors and robot-based sensors, including force/torque sensors, through multimodal deep learning (DL) approaches. Various DL-based sensor fusion methods, including data-level and feature-level fusion approaches, are thoroughly examined, and their accuracies for HAR are compared.A novel dataset was collected to train the multimodal DL models, capturing various user activities during HRI. This dataset leverages 10 different sensors, including 9 privacy-preserving sensors, along with an RGB camera for reference. The paper considers nine distinct user activities, including physical interactions with a robot and commanding a robot to perform specific tasks. The results indicate that integrating the sensors’ data at the feature-level achieves an 80.73% accuracy in recognizing various user activities during HRI.