Sounding out the HERE virtual reality demo

We’ve already discussed the impressive visuals on display during the HERE virtual reality demo at the Paris Motor Show, but audio plays an equally important part in creating this fully immersive experience, with the sounds of Frankfurt used to further illustrate the power of the new HERE services.

We spoke with Francesco Grani, a professor at Aalborg University Copenhagen, who worked with HERE as a consultant and sound engineer on the demo, supervising the creation of the 3D audio for the VR experience.

Francesco tells us more about his role in the demo:

“I was responsible for ensuring that the 3D audio experience worked well, and helped to design the hardware and software that would make the demo effective both for the person using the VR headset, and the audience in the room.”

Sound and vision

Francesco describes how the sounds were used, and the part they played in the experience:

“There were two levels to creating the audio for the demo. First, we used sounds to represent the services and the use of data, and second was the sounds of the city itself. For the data, we got inspiration from a real OEM data set, and made a piece of software which would audify this data.”

“So when you see the raw data being transformed by the cloud in the demo, the sounds heard are from actual data.”

Francesco explains how realism was key, both in the sounds used for the data, and those used for the city. He adds, “In the demo you’ll hear the real sounds of the train station in Frankfurt, and real noise from all over the city, including traffic and car horns.”

These then had to be transformed into a 3D audio experience, using sound rendering. Different pieces of software were used to place these different sounds into the demo. Then, using an encoding named Ambisonics, the team created an audio experience that can be interacted with by the user.

For example, if the user approaches a highway, the sounds of cars will get louder, or diminish as you walk away.

Playing for the crowd

This is the second time that Francesco has worked with HERE, having been a consultant on a previous project, and he describes the unique challenges he faced with the VR demo’s sound:

“Ensuring that both the person wearing the helmet and the audience in the room could properly experience the audio was very tricky, because the representation of space is different.”

Francesco explains that if the person wearing the helmet moves around, some of the audio cues will become louder or quieter depending on where they are. For those not wearing the headset, this can seem strange and poses a challenge to the developers of the project as it’s harder to create this level of experience for the audience.

He adds, “With visuals it’s a little more straightforward – one person can wear the helmet, while everyone else in the room can look at a screen. Audio is trickier - we didn’t want to make it so that only one person could experience this.”

To tackle this issue, the team relied upon a loud-speaker set up instead of headphones, which is usually used for VR experiences. Then, fixed sounds of the city were played through the speakers as a reference point to the audience, and when the participant moved, changes in the audio were reflected to those in the room.

“We had to split this rendering, using absolute positioning of sounds as well as the more dynamic sounds. It was challenging but very rewarding – everyone in the team had a different way of addressing this issue, and thankfully we were able to work together to get it right.”

Sound of success

When asked what the team hoped to achieve with the audio experience, Francesco says:

“With audio, it’s working well if you don’t notice it. If it’s natural, and if the sounds are the right accompaniments to the situation you’re seeing, then it can often just pass you by. The flip side of that is that when it’s wrong, you notice!”

Francesco explains that when you create sounds for an experience like this, it can’t be intrusive, but it does have an important part to play when immersing the user and telling a story. He adds:

“For instance, if we want the user to turn around and look at how one of the services is collecting data, we don’t want to tell them to ‘turn around’, as it’ll interrupt the experience. But a simple directional sound cue will get the user to turn around and follow the story.

Francesco, who stresses the importance of working closely with his team and, in particular, Tomas Gatial, who helped input the sounds into the coding and test the experience, adds:

“We want to use sound to help tell the audience a story, and for it to feel spontaneous.”

How important do you think the role of sound is in VR? Let us know in the comments below.