Originally published on Designing Sound, on Aug 25, 2014
As a composer, musicologist and a sound designer who is making a transition to the world of game audio for the last year or so, not only do I have a new level of respect for everyone who works as a game audio professional but I also became aware of various changes I am going through almost daily to adapt my already established skill set and mentality to fit my new chosen profession. These changes affect different aspects of my auditory world to varying degrees, but listening and specifically critical listening ended up being a new kind of challenge for me. As a musician who is used to listening critically to music and its various properties, and as a musicologist who researched film music for years, the inherent interactivity and flow of the gaming experience required a new type of listening capability from me. One that depended on me to not just pay attention to the different aspects of the soundscape, but also to rise to the challenges that were presented to me by the game to succeed as a gamer. It meant orienting my attention to the other aspects of the game; so much so that, I forgot to listen for a while and instead just heard what the soundscape consisted of. So how would it be possible for me to play a game AND critically listen to its audio aspects at the same time?
This article aims to share this and many other questions I asked myself to figure out a way to adapt my critical listening skills to the world of game audio. I do not believe I have clear cut answers regarding how we listen and analyze game audio as we experience the game. I do not even think it is possible to have just one definitive answer, since we all come from different backgrounds and have unique listening abilities. That being said, by sharing my research and opinions on the subject, I hope to raise new questions and start a discussion to learn more about the methods each of us use to critically listen to game audio.
Before moving on to these questions, I think it would be fitting to begin by explaining the role attention plays when we hear and when we choose to listen.
Hearing vs. Listening
Hearing is the second sense we develop at a mere 16 weeks after conception. We start to perceive and react to sounds before we even have fully developed ears . Yet soon after we are born, we slowly start to take for granted how prominent and important to our survival our hearing is. It’s prominence stemming from how direct and hard to obstruct the whole hearing process is; e.g. if you do not want to see something, you close your eyes or look away. But if you do not want to hear something, you better have fully isolated headphones; and even then you are left with hearing your body at work, your heartbeat and your blood flowing through your veins. In short, you can be in the dark whenever you choose to, being in silence takes much more work and is impossible to accomplish completely.
Yet, it’s evident the evolutionary advantage the directness of our hearing provides, because the sooner you are aware of the danger the sooner you can react to it. So you do not have to look in every direction to notice a dangerous object or an entity that is coming towards you. You can pinpoint the location by using your hearing alone. At the same time, this unobstructed flow of sensory information would be hazardous to our mental health without a cognitive system to filter out what is essential to our auditory perception and what is not.
Seth H. Horowitz states in his article about the subject “The difference between the sense of hearing and the skill of listening is attention.” As he explains, we use different types of attention that use different parts of our brains to react to different kinds of auditory stimuli. The two main types of reactionary response we have to auditory stimuli is a “bottom-up” process, which we use to pinpoint the location the sound we hear is coming from, and a “top-down” process that uses our brain in a more computational manner, and lets us focus on what it is that we are hearing and enables us to ignore the irrelevant sights and sounds. So when we talk about listening, we actually talk about paying more, focused and deliberate attention to an auditory sensation so we can process and gather necessary information from it.
This separation between hearing and listening is an important skill that we use all the time in our daily lives to protect ourselves from danger and keep the amount of information we perceive throughout the day manageable. However, what happens when we start to make decisions on what to listen to and what to block out, not just in our own “real life” experiences but on the various forms of entertainment mediums that give us an audio-visual wealth of information to choose from.
First of all, we all experience those forms of entertainment in a different way than we experience our lives, because the audio-visual information we get from those sources are limited to what we are “allowed” to witness. But what if we are also a part of the creation and production process behind those entertainment mediums and therefore knowledgeable about the workings of how our senses are being distorted or manipulated on purpose to create a certain kind of experience. Then how do we choose what to focus our attention to in any kind of manufactured auditory landscape?
That’s when things start to get more complicated.
Critical Listening
It is a known fact that focused and analytical listening makes up a considerable part of the learning process in any profession relating to audio. Listening to the music of the composers that came before us, to the sound design of iconic movies and games, to the mixing and mastering of historical albums, all of which helps us become better at our jobs. So having a “critical ear” and learning to listen analytically is crucial to cultivate ourselves and gather information that we cannot gain in any other way. However, this endeavor as good intentioned as it is, may also become a resource of adversity.
When I first started analyzing the relationship between film and music thoroughly a few years ago, I came upon a startling revelation. All the fun I got out of watching movies and enjoying their music was vanishing slowly. At first I thought it was probably due to my mental approach to it as a duty or a job and not as one of my biggest passions which led me to music research in the first place. Maybe that was a part of it, but after keeping track on my analysis process, I seemed to find the real reason behind my lack of enjoyment. I kept interrupting the movie on purpose; to take notes about the use of music or the editing or the narrative and this behavior created a fragmented experience which resulted in me being ripped apart from that immersive and mesmerizing world created by the movie. This led to an even bigger question concerning the quality and the validity of my research. Because even though I was trying to capture the musical score’s effect on the perception of the audience and I was doing my best to take meticulous notes about the changes in instrumentation in tandem with a cut or a zoom, or how a melody was used in different keys to depict changes in the narrative, I wasn’t really perceiving the work of art in front of me as an audience member who surrenders herself to the reality of that world, but as a musicologist who was outside, looking in. Thus, I wasn’t listening to the whole soundscape as it was intended to be experienced but focusing my attention to certain aspects at certain times by picking and choosing different elements to gather information from.
So now, in addition to the choice between hearing and listening, I was also choosing to employ a different type of listening attitude that sets me apart from someone who is experiencing the same material as a spectator. Which brings us to, what are the options to choose from when it comes to listening critically?
In his 1990 book Audio-Vision, French composer-filmmaker-critic Michel Chion states, “…there are at least three modes of listening, each of which addresses different objects. We shall call them causal listening, semantic listening, and reduced listening.”
We can summarize his further explanation of these three modes as;
Causal Listening; as a means to gather information about what is causing the sound. If the cause is visible sound can still be used to have supplementary information about it, if the source is not visible sound can be our only resource to identify the source.
Semantic Listening, as a tool to understand and interpret messages created by use of a code or a language.
Reduced Listening, which was named by the musique-concréte pioneer Pierre Schaeffer himself, as a mode of listening that solely focuses on the properties of a sound by itself, without any semantic or causal relations it might have attached to it.
If we accept these three main modes of listening as basis and look at their different purposes, it is evident that for critical listening we do not only employ one of these modes to gather information about the auditory aspect we are looking into, but we also might use them in combination with each other to gather more information at the same time.
For example, to analyze the sound design and foley in a scene which only consists of footsteps on different surfaces, we might use two of these modes in combination. Causal listening might clue us into if the footsteps belong to a man or a woman and reduced listening would help us analyze and form an opinion about how these footsteps were created and applied to the scene by analyzing its properties as a sound. Such as if it gets repetitive, if it has a high frequency content or not, if it has any effects applied to it and if these effects support the causal context of the sound, such as walking in a cave or a large hall or on carpet.
So, to listen critically, first we choose to listen and then choose the type of listening mode or modes we need to use to gather information from and switch between them as we see fit. Yet the explanation of the choosing process does not explain how we decide when to listen critically, or if we are even capable of not listening critically at will, once we learned how to do it.
Is ignorance bliss?
I think it would be appropriate to step back for a second and look at the notion of critical listening being a source of adversity at this point. The main question I had after realizing what was causing my somewhat dull film music research experience was, how is it possible for someone who knows exactly how movies are made to still be immersed in it as much as an audience member who are not as informed as they are? Because after learning a lot about how music “works” in movies, I was starting to get a little bit overwhelmed by how I noticed each time a movie used a popular song to crossover from diegetic to non-diegetic score (e.g. if the song was playing in the background at the radio and then suddenly it was heard in a more dominant way, intended for the audience to hear while cutting the scene to a montage or a flashback) or how certain musical textures were used over and over again in scene transitions to keep the audience’s focus away from the cut. In a sense, the more I knew about the process and listened critically, the less I enjoyed the experience as a movie aficionado.
So how is it possible for musicians, composers, sound designers, audio engineers, foley artists etc. to shut off the analytical and informed part of their minds long enough to enjoy all of the entertainment mediums/art-forms available now, without driving themselves crazy obsessing over the simplicity of the chord progressions or the timbre of the sound or how that high frequency boost is ruining that mix, while also listening, watching or playing the same medium?
Of course this doesn’t mean that all forms of entertainment and audio-visual art are forever ruined for whoever is participating in their creation process, since there is definitely a certain satisfaction and enjoyment that comes out of analyzing and finding things out about our favorite movies, games, pieces of music and sounds. But I believe the habitual critical listening still might be problematic in some cases. For example we might miss out on experiencing the medium “as intended” while we are in the constant process of choosing what aspect of the medium to focus our attention to. This is especially true in the case of game audio, where we might just get one shot at experiencing the game due to time constraints. Unlike a two hour long movie or a 40 minutes long album, what if it’s a twenty plus hour long RPG-like AAA title which we were waiting forever to play; do we enjoy it as gamers first or do we pay more attention to audio knowing that we will sacrifice some parts of the gaming experience in the process? Sure it is possible to pay attention to both in some cases, but it is a fact that most of us wear many “audio hats” as game audio professionals, composer, sound designer, audio engineer, implementer, you name it we do it!
So, how do we choose when and what to pay more attention to? And what do we sacrifice in return?
A Dilemma to Live With
French neuroscientist Jean-Philippe Lachaux who specializes on attention, demonstrates in his talk how our brains decide on a regular basis to what to pay attention to. His research shows that we all have what is called a Brain Priority Map and the features of this map is crucial in deciding where we direct our focus next and therefore adjusting our senses to receive stimuli accordingly.
Four Main Features of the Brain Priority Map
1. We all have a priority map on our brains in the parietal lobe
2. Map can/will change to adjust to your current interest
3. Map takes into account what we like and we don’t like
4. We can move our attention away from very salient events
From these four features, in my view number two and four are the most crucial ones for our critical listening processes as game audio professionals.
The second feature of the map suggests that we can change our priorities based on our current interests, which means if we are playing a game and at the same time paying attention to the sound design elements of the same game, we can just choose to focus on the gameplay elements in a very challenging part of the game and then redirect our focus back to the sound design in the next cut-scene or say after we killed the boss we just fought to finish the level. Which suggests it IS possible to play and enjoy the game as a gamer and still pay attention to the parts of the soundscape we want to pay attention to whenever we see fit. That being said, there is still a good chance we missed out on an opportunity to analyze and learn from important sound design elements or the musical progress, the moment we decided to solely focus on the gameplay. But there are a few solutions we can come up with to deal with this problem, one being recording the gameplay to go back and watch/listen to see what we missed.
Fourth feature of the map is crucial for those of us who mostly work in a single capacity as a game audio professional. For example if I am a composer AND only interested in the other elements of the soundscape from a composer’s point of view, it is important for me to be able to tune out the other elements some of the time, such as dialogue and sound design even if they are more prominent in the mix, to focus more on the musical properties of the level. This does not mean composers shouldn’t pay attention to other aspects of the game audio process, on the contrary the more we know about the other facets of the game audio world the better we are for it. Yet a composer might only have time to go through a part of the game once and might want to analyze the musical features and the relationship between music and the narrative in a game at that time. That is when she should be able to move her attention away from the more prominent events as the fourth feature suggests and focus solely on what she aims to listen to critically while still playing the game.
Lachaux also points out there are three main systems in our brains which work on allocating and orienting attention in tandem with the brain priority map.
These systems are;
Habit System (hardwired, orienting attention according to a fixed set of rules, which learns through the repetition of behaviors)
Reward System (orienting attention according to what we like and what we don’t like)
Executive System (allocates attention in a more flexible manner to pursue our present, voluntary goals even if it means battling the other two systems)
Among these systems, while the Habit System is mostly responsible for us finding our way through similarly designed game types and Reward System is there to encourage us to play “one more turn”, it is the Executive System that I like to emphasize from a critical listening viewpoint.
Allocation of attention to pursue our present and voluntary goals is what enables us to take away our focus from the gameplay to direct it to the aspects of the soundscape we are interested in. In my experience, there are times when I realize, I was so captivated by that cut-scene, I forgot to listen to the music or I was too busy trying to jump from that ledge to another I forgot to pay attention to the sound my jump has made. Yet as soon as I notice this change in focus my Executive System kicks in and reminds me that as a game audio professional I should be paying more attention to the music or the combination of the ambience and the dialogue, since it is a voluntary goal for me to learn more about these subjects by paying more attention to it. That is in my view, one of the main reasons it is still possible to critically listen to such an interactive and immersive medium while still being the passionate gamers we are.
To Know More is To Learn More
Until about a year ago, I was only interested in the world of game audio as a gamer, a composer and a researcher who was mostly interested in the aesthetic parallels and contrasts between games and movies. Little did I know about all the work that went into the sound design, implementation and the integration of all elements within a game. Yet in just a year, the amount of knowledge I gathered from the wonderful game audio community has changed the way I paid attention to the soundscape of games.
I first realized this a few months ago when I was finally playing through Remember Me. I noticed that I was not only hearing how the music complimented the narrative as I was used to analyzing from that point of view, but also how it was layered in different tensions that changed during fight sequences adjusting to Nilin’s progress through the fight. As an additional quest, I tried to guess how the sounds were implemented by going in and out of the areas that caused a change in the ambient sounds and the music, which middleware was used or was it a proprietary software etc. In short, I was listening critically in a whole new way, because I knew what to listen to in order to learn more, and learned more as a result of my new found ear for game audio.
Another test I made to see how my critical listening has changed in the last year was to play a game I played a year ago and see if I would find something new to listen to in it. I chose to play The Last of Us after reading Rob Bridgett’s recent blog entry named “Why You Probably Shouldn’t Be Contributing More Than 33% to the Soundtrack,” which makes many great points about the balance between the three main aspects of the game audio soundscape and uses The Last of Us as an example of a well thought-out game audio collaboration. When I played the game a year ago when it first came out, I remember being a bit underwhelmed by how “little” the music was used overall. But on my recent play-through not only was I paying attention to the sound design and the ambiance more than I ever did a year ago, I also noticed how most of the time only two of the three main elements of game audio was prominent throughout the game. It was either sound design with dialog (as it was used in the low tension parts of the gameplay where we wander through the area as we talk with Ellie), sound design with music (mostly during high tension chase or fight sequences) and music with dialog (during the cut-scenes). The orchestration-as Rob Bridgett puts it-between these elements are so refreshing that it enables the player to pay attention to and appreciate each element separately when it’s their time to shine but also makes it possible for them to be a coherent and prominent parts of the whole gaming experience as they were connecting to one another throughout the game seamlessly.
These experiences not only proved to me that critical listening is an adaptive, never-ending process that gets richer by the amount of knowledge we acquire, but also showed how listening to game audio differs from other types of critical listening (e.g. film music analysis) in its interactivity and immersion. Because in the case of game audio, auditory perception is not only used to understand and enjoy the narrative as an emotional and relatable experience, but also as an indicator of imminent danger, success or failure, a change in the time of day, etc. Which means most of the time, in order to succeed in the game, we have to choose between critical listening and focusing our attention on the gameplay elements to get through the challenge we are facing as gamers. Even though this process makes it harder for us to listen to the game audio content whenever we want, we are also never able to fully distance ourselves from the gaming experience as a result of it. We cannot become mere observers who are on the outside, looking in.
Because contrary to film, the game does not go on without you, you have to be engaged to some level to progress through the game whether you are listening to it critically or not. Therefore, even if there are times for necessary but short-term change in focus, the interactivity of the whole experience makes game audio analysis a one of a kind process.
So how do YOU listen to games as a game audio professional?
What is your process to analyze different aspects of game audio on the games you play?
Let’s discuss and learn from one another, so we can cultivate our critical listening with more knowledge and learn to listen better…