Ready Player One (2017)
I wanted to start this blog post off today with something I came across this week when just pottering around playing videogames in my spare time on my PS4 console. I’d been feeling recently that I own a small TV with a crummy speaker system in the back (not connected to my mixing desk) and I’m an audio guy who is not really set up for hearing games in the way the makers have intended. So, given that I did some research and I recently became the owner of some Sony Playstation Wireless Gold 7.1 Surround Sound headphones. As soon as it came through the door, straight away I started up the newest game I had in its current saved game state, put the cushioned head and ear pads on my head and… they absolutely blew me away. Immediately I could not believe the difference they made…. It was like I was swinging through New York City hearing in every zip, swoosh past a building with all the familiarities of the location. From busy hustle of people below going to work, to a police car or dump truck fading in the distance and closer to feeling every hyperrealistic superhero punch in Spiderman PS4. The reaction I extracted even from my little brother too as he took to 7.1 surround sound which threw me, as even at his young age he could understand in his game FIFA 18 the complete transformation the audio could make to the visual that he had never comprehended before in videogames. He said and I quote “The sounds of the football hitting each players foot, the players shouting across the pitch and the crowd chanting all around me it feels amazing I NEED THIS”. I could only answer yes I know, but it gets even better…
3D Audio Technology – The Future For Immersive Gaming?
Sound is more forefront than you would ever imagine in modern entertainment platforms and should be considered of paramount importance to any game under construction. Since the first musing of audible bloopy/bleep sounds, to sound synthesis, to 8 and 16 bit samples; over time audio technology has been forever growing, making its way to surround sound 5.1 capabilities in gaming consoles today. Since 5.1 was officially introduced to cinema in the late 1980s, improvements of the same model waveform/track system have been introduced gradually, but nothing has quite hit home hard enough to warrant significant industry change. Understandably so, with the casual gamer generally not even set up for getting the full 5.1 – 7.1 audio experience in their front room, with their internal television speakers set to default and not a speaker system or headset in sight.
Virtual reality on the other hand is looking to transform this generality into a completely new era of game audio experience, bringing the game world ‘close to the player’ using a HMD (head mounted display) with a new spacial three dimensional headphone audio approach. Applying ‘the ability to track the user’s head orientation and position’ and ‘play a sound as if positioned at a specific point in three dimensional space’, significantly enhancing and immersing the audience in the audio visual experience around them.
For the last two years, as well as producing smaller VR related games, Playstation have been shipping many of their new exclusive AAA game titles such as Uncharted 4, God of War and Horizon Zero Dawn to name a few, to be compatible with their very own three dimensional audio Platinum Wireless Headset. For both work and play purposes I am a proud owner of the Playstation Gold Wireless Headset which caters for 7.1 virtual surround sound, without the option of a 3D Audio mode included with the Playstation Platinum pair. For now I am in the mind of keeping my set for the purpose of but it doesn’t stop me looking up in awe at the sound localisation capabilities involved in the 3D audio spectrum.
The Platinum Wireless Headset (from Playstation), currently valued at RRP £129.99 provides:
Enhanced 7.1 virtual surround sound in all PS4 games.
3D Audio mode that gives precision sound from every direction in compatible games.
Custom audio modes that let you experience games exactly as the developer intended.
Get the competitive edge by pinpointing every move your enemies make and immersing yourself in highly positional, multi-layered soundscapes.
Leading experts and virtual reality pioneers Oculus, argue that the platform of virtual reality is pushing this advancing audio technology forward. Believing strongly, as stated in their development documentation library that in order to create a convincing experience within the realms of virtual reality, sound must ‘play in our sense of being present in an actual, physical space (…) as it will contribute powerfully to the user’s sense of immersion’.
Interested by this concept, I looked into how Oculus and companies alike go into creating this type of technology and specifically how they are going to solve the spatialisation issues within this type of technology. Harnessing the use of a number of monophonic sounds, they seem to look to redirect/transform the signal path so that each individual sound sounds like its coming from a particular space/location. Their research relying on the human psychoacoustic factors, including that of spectral modifications, a sense of timing, level and phase to localise sound help them to achieve this:
Directional Localisation:
Lateral – Sound that is closer to the left, the left ear hears it before the right ear hears it, sounding louder. Ways of which you could achieve this could be using:
ITD (Inter-aural Time Difference) – To primarily localise a sound based on the delay between the sound’s arrival in both ears.
ILD (Inter-aural Level Distance) – To primarily localise a sound based on the difference in the sound’s volume level in both ears.
Front/Back/Elevation – Considered more complex as you cannot rely on time differences, since ITD and/or ILD could be 0 for a sound in front of or behind the listener.
HRTFs (Head Related Transfer Function) – Cornerstone for most modern 3D sound specialisation techniques. ‘HRTF capture is to take an individual, put a couple microphones in their ears (right outside the ear canal), place them in an anechoic chamber (i.e., an anechoic environment), play sounds in the chamber from every direction we care about, and record those sounds from the mics. We can then compare the original sound with the captured sound and compute the HRTF that takes you from one to the other’.
Head Motion – Acts as a localisation assistant as the easily overlooked speed-bump while using HRTFs is that a simple turn of the head changes difficult front/back ambiguity problems into lateral localisation problems. This is solved by a shortening and lengthening/widening of audio parameters so that for example when a sound is behind you and you turn your head over to the left, the sound coming into your right ear (now at the front of your body) as you turn would be set so that it is quieter than your left which is now closer to the sound source.
Distance Localisation:
Loudness – Simply how loud an object is within a game state. Important to note/be familiar with the realistic volume of an object to make them come off in a realistically audible fashion.
Initial Time Delay – Interval between the direct sound and first reflection of the same sound. Note that anechoic related environments (e.g. deserts, plains etc.) make this more challenging due to their non reflective nature.
Ratio of Direct Sound to Reverberation – Used for decades in the music and sound industry, meaning in an intendedly reverberant environment, a long echoing sound tail is audible that very gradually fades.
Motion Parallax – Indicates the distance you are from the speed/movement of a sound source through a certain space.
High Frequency Attenuation – How over long distances we can theorise distance based on how diminished or impaired high frequencies are.
Sources:
Burgess, D.A.: ‘Techniques for low cost spatial audio’. In: Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, pp. 53–59. ACM (1992).
Cheng, C.I., Wakefield, G.H.: ‘Introduction to head-related transfer functions (HRTFs): representations of HRTFs in time, frequency, and space’. In: Audio Engineering Society Convention 107. Audio Engineering Society (1999).
Howard, D.M., Angus, J.: Acoustics and Psychoacoustics. Taylor & Francis, Routledge (2009).
Oculus. 2018. 3D Audio Spatialization. [ONLINE] Available at: https://developer.oculus.com/documentation/audiosdk/latest/concepts/audio-intro-spatialization/#audio-intro-spatialization. [Accessed 25 October 2018].
Playstation. (2016). Crafting 3D Audio for Uncharted 4 | Platinum Wireless Headset. [Online Video]. 14 December 2016. Available from: <https://www.youtube.com/watch?time_continue=1&v=o9wGomzrt44> [Accessed: 10 October 2018].
Staramba, (2018), Wade Watts (Tye Sheridan) entering OASIS from his garage in Ready Player One. Jaap Buitendijk/Warner Bros. Entertainment [ONLINE]. Available at: <https://medium.com/@Staramba/spielbergs-oasis-becomes-reality-with-staramba-cb03adc70943>[Accessed 25 October 2018].