Nintendo finally delivered some dedicated audio circuitry to offload the huge task from the CPU-GPU and provide richer sounds. Their new solution is an independent Digital Signal Processor or 'DSP' manufactured by Macronix running inside Flipper.
The DSP's job consists of performing different operations over our raw audio data (e.g. volume changes, sample rate conversion, 3D sound effects, filtering, echo, reverb, etc) and then output a 2-channel PCM signal. It doesn't work alone however, the DSP delivers audio with the help of other components.
Its first companion is the Audio Interface (AI), a 16-bit stereo digital-to-analogue converter responsible for sending the final sample through the audio signal that ends on the TV. The AI can only process 32 bytes of audio data every 0.25ms, so if we take into account that each sound sample weights 2 bytes and we need two to create stereo sound, the AI will be able to mix up to eight stereo samples with up to 32 kHz of sampling rate, sound!
Finally, we have the Audio RAM (ARAM) block, which is a large (16 MB) but very slow spare memory that can be used to store raw sound data. There's quite a lot of space, so the GPU can also use it to store additional material (like textures). The CPU doesn't have direct access to this memory so it will resort to DMA to move content around.
For better or worse, the DSP is programmable with the use of microcode (yikes), but fear not, as the official SDK already bundles a general-purpose microcode that almost every game used, except on the console's boot sequence and some Nintendo games (how convenient, as Nintendo didn't document the DSP, so only they know how to program it).
That being said, the process of generating sound works as follows :
- CPU commands DMA to move raw samples to ARAM.
- CPU sends a list of commands that instruct how the DSP should operate these samples. In other words, it uploads the microcode program (only one is officially available for developers).
- DSP fetches samples from ARAM, applies the required operations and mixes them into two channels. Finally, it stores the resulting data on RAM.
- AI fetches processed samples from RAM and outputs them through the audio signal.
Compression and freedom
While we've already reached the sampling age and we are not locked to specific waveforms anymore, the new sound system is still a huge improvement. For starters, the saga of forced music sequencing is gone for good. The system can now stream pre-produced music to the audio endpoint without problems, much like what the Saturn and PS1 accomplished years ago.
Let me show you an example using two games, one released for the Nintendo 64 and its sequel released for the GameCube. Both have different music scores but the context (enemy battle) is the same. Take a look at how both tracks differ in sound quality, taking into account the design of each system (shared vs dedicated).
Video - Paper Mario (2000) for the N64.
Sequenced on the fly by the RSP.
Video - Paper Mario: The Thousand-Year Door (2004) for the GC.
Streamed to the DSP.
As you can hear, the DSP finally gave music composers the flexibility and richness they always asked for.
Additional material
For a more direct side-by-side comparison, I've prepared this interactive widget that shows how composers ended up adapting their arrangements for the GameCube and its predecessor. Here, the same upbeat score is utilised for a Nintendo 64 title and a GameCube one, and the resulting comparison allows me to demonstrate (once again) the technical advantages of the GameCube's DSP.
Nintendo 64:
Click to listen
GameCube:
Click to listen
Interactive player available in the modern edition
Audio samples
Nintendo 64: Kirby 64: The Crystal Shards (2000).
GameCube: Kirby Air Ride (2003).
Now, to visualise what's happening behind each track, here are the two respective spectrograms. Before I start, if you are not familiar with these kinds of charts, I recommend reading my previous NES article, in particular the audio section (where I introduced them).
Spectrogram of the PCM channel in Kirby 64: The Crystal Shards (2000).
Spectrogram of the PCM channel in Kirby Air Ride (2003)
To be fair, mixed tracks are difficult to decompose in a spectrogram, but I believe I can attempt to deduce some patterns from it.
To begin with, almost all the frequency spectrum is being evenly utilised in the GameCube track, which may be attributed to the additional instruments used for accompaniment (which add harmonics and therefore fill more areas on the chart).
Finally, the amplitudes on the GameCube spectrogram look more uniformly distributed. In other words, the volume of each instrument is differently balanced and includes effects like reverb. I'm guessing the latter is what the composer originally intended while producing this score, and these types of controls are possible by the fact that the GameCube supports audio streaming. Thus, composers can use any tool of choice to sequence and mix their music, as opposed to strictly depending on the console (and its limitations) to sequence and mix at runtime.
I wouldn't say that the Nintendo 64 is completely incapable of producing the same result. However, one thing for sure is that, in the world of the Nintendo 64, every single audio function costs extra cycles and/or memory, and this can have an impact on other areas of the game. Hence the need to ration resources. On the other hand, with the GameCube's support for large samples, one can just stream the full produced score altogether.