The use of real time synthesis and behavioural audio in games is exciting. Less talked about, but just as interesting are the applications for advanced sound design technology in movies. A lot of existing sound design follows the traditional methods of recording, treatment, splicing and mixing of real world sounds for film. An important consideration for film is often that the sounds are very real, organic and rich, which is harder to obtain with pure synthesis. This raises the first point. There are certain situations where unreality is exactly what we are after. Good sound design is an art that employs lots of tricks, using every tool in the box. To take a purist view, either in favour of synthetic methods or to exclusively rely on recording is to limit ones palette and possibilities. So, we should acknowledge that synthesis already plays a very important role in high end sound design, ever since the Baron's used electronic circuits in the soundtrack for Forbidden Planet.
Here's a difficult question. Which is more real, a high resolution photograph or a low quality film clip? It is a question I give to students on sound design courses, often by showing a clip of a famous aviation pioneer Amelia Earhart along with a photograph of her. The photograph is quite posed and iconic, revealing her beauty and character, as if of a 1930's film star. The film clip is grainy and typical of 1930s footage, speeded up and speckled with noise and imperfections. Of course you could try to answer the question in terms of information analysis and work out the data content of each medium. For the sake of this argument the short grainy film clip (of just a few dozen frames) and photograph contain roughly the same data content.
Although each frame of the film clip is a very poor likeness the complete result reveals much more about Amelia. The way she moves, and smiles, her reactions in context, the movement of clothing and the aircraft in the background all speak volumes about her life and character. In just a couple of seconds the film clip captures what the photograph cannot. Behaviour.
And so it is for sounds. A sound recording is a photograph. Like a modern photographer using lighting, makeup, careful positioning and lens filters we sound designers do exactly the same thing during the recording stage. We try to capture the most flattering likeness of the sound, outside its real context. Procedural audio is very like film for sound. Instead of a definitive moment of sound we are interested in the behaviour of a sound object. I like to say that although many people denigrate synthesised sound as less real than recordings, if done properly in some ways it is more real.
For example, let's consider a passing helicopter. The sound it makes at an observer's location is affected by many complex variables to do with its mechanical behaviour. The ratios of engine and gear noises to blade slap, directionality of exhaust, effects of down wash and Doppler are all bound up in the precise movement and relative positions of vehicle and listener. Let's say that we have recorded the perfect fly-by sound for a given scene but the director now wants to shorten the clip and use two edits to change the perspective during the shot. With recorded audio we can do some remedial trickery, maybe cut in another clip of a more distant helicopter (because we didn't get anything suitable during the field session), or use timestretch to change the duration of the by. But these methods introduce imperfections and artifacts and they are time consuming.
As another example, when an object is dropped onto the ground it bounces and rolls in a complex way. We often get away with approximate synchronisation in film because the audio-visual contract, a psychological Gestalt effect, encourages the viewer to ignore small misalignment or inappropriate material behaviour. But what if we could make perfect synchronisation very easily? Procedural audio lets the sound designer place key frame markers and interpolate between movements so that the sound generator can match collision events with sub frame accuracy. Then, if the scene is altered the sound can be rendered again to fit the new behaviour. The designer only has to do the work once, for one object, but can then spawn many copies of the same object each following different behaviours. This is great for creating effects for volumetric extents like swarms.
For film sound design, Pure Data can be a powerful tool. In one patch you can capture not only the audio made of small recorded clips but the logic to control its behaviour from a sophisticated MIDI or OSC control track. You can combine mixing, splicing with granular and resynthesis methods or use components that are pure synthesis. You can use ducking, compression, reverb or any traditional effects within the same object then store it away as code independent of any other settings in the system.
Summary of points for game developers