An ecosystem for recording and playback of 360 3D (3D60) video and sound was developed in this project. The primary playback platform is VR, where the goal is to provide the user with a realistic 3D audio-visual experience. Contrary to state-of-the-art 360 video systems which only allow for head rotations, 3D60 provides full freedom for head movements (translation and rotation) while preserving the visual 3D effect of the video. In combination with the 3D sound-field, this provides an unparalleled realistic immersive experience – it feels like you are there!
The research in this project was necessary because there do not exist any good solutions for joint recording and realistic display of 3D audio and video in VR goggles which preserves the depth perception without artifacts.
A 3D60 prototype and an initial demo showcasing the potential of the technology was developed in the last reporting period. The 3D60 camera is based on multiple RGB-D cameras which are configured to have minimal overlapping field of views. The automatic post-processing software stitches the RGB-D video from the respective cameras onto a uniform 360 3D representation of the space around the camera. A real-time VR player has been developed for the Oculus Rift head set.