Project Overview

The goal behind this immersive game was to raise awareness for the high stress levels of healthcare workers, which was particularly a hot topic during the pandemic.
In the research for XR, there is a significant focus on the visual aspects of reality and immersiveness. In this project, we wanted focus on the use of 3D spatial sounds to establish a realistic immersive experience.

Please put on your headphones while watching the demonstration below to hear the 3D sounds!

My contributions

In a group of 3, we were all responsable for the programming on node.js - I worked on establishing a Bluetooth connection with the heart rate sensor used in the game as well as creating and integrating 3D spatial sounds to the game using Resonance Audio by Google.

Other team members: Damian Dominik Martinez (TU Berlin), Janine Güldner (TU Berlin)

Concept

The project, as part of the module "Introduction to Physiological Computing", was accompanied by the seminar "Affective Computing" which focused on affect detection using bio-signals such as the heart rate. Our initial inspiration was to develop an AR game, where the player needs to localise the source of sounds. For this, we utilized 3D spatial sounds, which were created using Resonance Audio. The player needs to localize the source of the sounds to find the patients that need treatment. The simulation was meant to be stress-inducing and overwhelming for the player. To evaluate our success with that, the player was wearing a heart rate sensor (Polar H10) during the game.

Behind the scenes: Technicalities

The game was developed in node.js -an open-source, back-end JavaScript runtime environment. We used A-Frame which is an open-source web framework for building virtual reality experiences. The game utilizes 3D spatial sounds, which were created using Resonance Audio, to establish a realistic immersive experience. Once the game starts, the player needs to pair their herat rate sensor. After the pairing, the player is directed to a demo wheere the game instructions are given. Before starting the game, the player is exposed to 10 seconds of white noise to lower their heart rate.

User Testing

The game setting consisted of headphones for the 3D sounds and a smartphone attached to a head mount to run the web application.
An affective analysis was conducted using the heart rate data of the players to confirm that our goal of inducing high stress levels was accomplished.

Outlook

Spatial sound represents an important cue for navigation in space. This project explored the possibility of localization in a tabletop augmented reality environment, based on sound cues. In the research for augmented reality, there is a significant focus on the visual aspect. and there is not enough investigation about the audio side of AR. I believe that the use spatial sounds are crucial to establish an realistic and immersive experience.