A dedicated component called Audio Processing Unit (APU) provides this service . Ricoh embedded it inside the CPU chip, presumably to prevent unlicensed cloning of both the CPU and APU.
Functionality
This audio circuitry is commonly referred to as a Programmable Sound Generator (PSG), which vaguely implies that it can only produce a predefined set of waveforms. This is mostly true in this case.
The APU sequences audio data across five channels, each reserved for a specific waveform or signal. The channels contain different properties that alter the waveform's pitch, sound, volume and duration. They are continuously mixed and transmitted through the output audio signal.
The APU's functionality is exposed through memory addresses. The CPU reads the music-related data found in the Program ROM and programs the APU accordingly.
Furthermore, the Famicom model implements extra cartridge pins that send the mixed audio signal to the cartridge, so the latter can mix it with extra channels (requiring additional chips) .
Let's now review the five channels the APU provides :
Pulse
(Video) Pulse 1 channel.
(Video) Pulse 2 channel.
(Video) All audio channels.
The first two channels produce pulse waves . When heard, they exhibit a very distinct beep sound that is mainly used for melody or sound effects. By varying the pulse width (also known as the duty cycle), the respective sequencer can generate three types of pulse waves. The circuits are also connected to a sweep unit (allowing to bend the pitch) and an envelope generator to lower the volume over time (commonly referred to as decay).
Most games use one pulse channel for melody and the other for accompaniment. You'll often find that when a game needs to play a sound effect, the accompaniment channel is temporarily switched to play the effect before returning to its original role. This prevents interrupting the melody during gameplay.
I believe it's fair to say that pulse waves are one of the emblems of this generation of consoles. I assume their adoption was primarily driven by cost-effectiveness: the (limited) CPU can only process a finite amount of data at a time, and pulse waves are ideal because they require few parameters to play simple melodies (which, in turn, frees up CPU cycles for other operations).
Triangle
(Video) Triangle channel.
(Video) All audio channels.
One of the specialities of the APU, when compared to the competition, is its ability to produce triangle waves. These are often used as a bassline for melodies. Additionally, by dramatically modifying its pitch, it can also be used for percussion.
The APU has one channel reserved for this type of wave. Behind the scenes, a dedicated sequencer takes 32 cycles to generate a triangle signal ; this limitation causes the resulting triangle waveform to resemble a step ladder.
On the other hand, the respective circuitry does not provide volume control. In any case, some games discovered alternative methods by fiddling with the mixer's volume control.
Noise
(Video) Noise channel.
(Video) All audio channels.
The concept of 'Noise' refers to a series of waveforms that lack any discernible pattern or order. In turn, our ears interpret it as white static. That said, the APU allocates one channel to play different kinds of noise.
Behind the scenes, the noise generator relies on an envelope generator (similar to the Pulse channel) that is randomly muted by an OR gate . The condition for muting is determined by the value of a 15-bit shift register connected to a feedback loop. All in all, this makes the circuitry output a signal with pseudo-unpredictable patterns, resulting in noise.
For control, four bits adjust the period of the envelope generator, while one bit modifies the 'Mode' of the shift register. This layout provides 32 noise presets. Half (16) of these presets generate clean static, and the other half produce robotic static.
Generally speaking, games utilise the noise channel for percussion or ambient effects.
Sample
(Video) Sample channel.
(Video) All audio channels.
Samples are recorded pieces of music that can be replayed. As you can see, samples are not confined to a single waveform, but they consume a lot more storage space.
The APU has one channel dedicated to samples. Here, samples are limited to 7-bit resolution (encoded with values from 0 to 127) and a ~15.74 kHz sampling rate . To program this channel, games can either stream 7-bit values (which steals significant cycles and storage) or use delta modulation to encode only the variation between consecutive samples.
The delta modulation system in the APU only accepts 1-bit values, meaning games can only indicate whether the sample increments or decrements by 1 each time the counter kicks in. Thus, at the cost of fidelity, delta modulation can save games from having to stream continuous values to the APU.
Since programming this channel takes longer space and CPU cycles, games typically store small pieces (like drum sounds) that can be replayed repeatedly. Be that as it may, throughout the NES' lifespan, numerous developers have come up with clever uses for this channel.
Secrets and limitations
While the APU could not match the audio quality of vinyl, cassette or CD, programmers did find ways to extend its capabilities, largely thanks to the modular architecture of the NES.
Extra Channels
(Video) Castlevania III (USA/Europe, 1989).
(Video) Akumajō Densetsu (Japan, 1989).
Remember that the Famicom provided exclusive cartridge pins available for sound expansion? Well, games like Castlevania 3 took advantage of this feature by incorporating an additional chip called Konami VRC6, which introduced two extra pulse waves and a sawtooth wave to the mix.
The two examples illustrate the difference between the Japanese and the American versions of the game. The Japanese variant, running on the Famicom, benefits from the enhanced audio capabilities, while the American version, designed for the NES, is constrained by the five predefined channels.
Tremolo
Video - Oscilloscope view of Final Fantasy III (1990).
Rather than increasing cartridge costs, some games prioritised creativity over technology to simulate additional channels.
For instance, Final Fantasy III came up with the idea of using tremolo effects to create the illusion of extra channels.
A more refined observation
Now that you've had a glimpse of what the APU is capable of, let me show you a more precise method for observing how its sound behaves. This will not only complement what you already know about the APU but also provide a more objective examination.
First things first, let's start with a brief introduction to sound theory.
Thanks to the principles of Fourier Analysis, we can decompose every single sound we hear into a sum of sine waves with different frequencies and amplitudes . The lowest-frequency sine wave is called the fundamental, while the others are referred to as overtones. If you add the fundamental wave and its overtones, you get the original sound back. Having said that, with sounds that have a recognisable pitch, you'll find most (if not all) overtones have frequencies that are multiples of the fundamental frequency. Thus, these overtones are called harmonics .
Harmonics will be a recurring topic in this section, as waveforms such as pulses, triangles, and sawtooths adhere to a formula that determines the harmonics they must contain. Otherwise, these waveforms might deviate from their 'perfect' shape.
Introduction to spectrograms
As sine waves are now the key ingredient that can make up any sound, we can now analyse the sounds that we hear by their sine waves. Now, for any kind of data analysis, there's nothing more convenient than plotting a graph to organise vast amounts of information. Well, in the case of sound analysis, we've got Spectrograms. These encode all the information from an audio sample in a single plot. The X-axis denotes time (in seconds), the Y-axis denotes the frequencies (in Hz) of sine waves produced during that time, and the Z-axis (intensity of each dot) denotes the power/loudness (in decibels) of each frequency.
Video - Example of spectrogram visualising one Pulse channel.
As you can observe from this example, each horizontal line (a.k.a. sequences of dots) corresponds to a sine wave (the lowest is the fundamental, while the rest are the harmonics) and their intensity indicates the amplitude. With this in mind, we can extract the following information:
- Over time, horizontal lines tend to drastically displace. That's the pitch of the Pulse channel changing.
- The start of each note is loud (the dots are vibrant) followed by a quick quell. That's the APU's envelope control in action.
- Moreover, at the end of the decay period, a vertical line shows up. It's quite brief so it's not easily audible but, in any case, that's noise (a short popping sound) and I suspect it's a deficiency of the APU.
- Holding a note for more than 0.25 seconds makes vibrato emerge (continuous fluctuations in the Y-axis). I'm not sure if that's intentional (using the sweep function) or an adverse effect of the APU.
Notice how most of these observations are not easily derived by just listening to an audio sample; this is the reason for writing this section.
Plotting the APU
To study the NES' APU, I've compiled five spectrograms, each corresponding to one of the APU's channels, using the previous examples. Alongside them, you'll find my attempt at unravelling what the data is exhibiting.
Before we start, I must confess that, to gather the data without inaccuracies (such as extra noise), some compromises were made. The recordings were obtained using a Windows program called 'towave', which uses band-limited synthesis to solve a fundamental problem with the emulation of PSG-based audio chips. Essentially, pulses, triangles and sawtooths consist of infinite harmonics. However, this doesn't mix well with modern sound cards which are limited to 44.1 kHz samples. Thus, a technique called 'band-limited synthesis' is employed to select the right harmonics within the sound card's limitations. All in all, this technique provides a feasible balance between performance, accuracy, and aliasing prevention. However, the data may not be 100% identical to its analogue counterpart (which, by contrast, would also introduce other issues, like noise from the recording equipment), but I believe it is to an acceptable degree and, most importantly, does the job for this section of the article.
That being said, let's get on with the analysis.
Pulse
Spectrogram of the Pulse 1 channel.
The theory says that a pulse tone only contains odd harmonics. In other words, the fundamental is combined with its third harmonic, fifth and so forth. Moreover, each harmonic decreases its amplitude the further it is from the fundamental. The amplitude formula is amplitude = 1 ÷ harmonic number .
Hence, notice how the intensity of each harmonic on the spectrogram dims the higher it is on the Y-axis. However, the APU's pulse waves also seem to exhibit the aforementioned vibrato effect which intensifies at each harmonic number. Moreover, areas of the spectrogram that should be empty of any sound contain hushed overtones (possibly the result of noise and other imperfections).
Triangle
Spectrogram of the Triangle channel.
A triangle wave is also made of odd harmonics, but their amplitude decreases more rapidly (following the formula amplitude = 1 ÷ harmonic number² ).
However, this is not what is shown here. The step-ladder-shaped triangle that the APU produces introduces additional harmonics and increased amplitudes.
Noise
Spectrogram of the Noise channel.
Naturally, noise doesn't abide by the rules of harmonics and may randomly fill the entire frequency spectrum, hence the absence of a clearly recognisable pitch.
Although, by observing the timeline, you can distinguish between the various noise presets provided by the APU, each exhibiting a unique set of overtones.
Sample
Spectrogram of the Sample channel.
Unlike the previous channels, the sample channel only plays back whatever low-resolution recording the developer feeds to the APU. Considering the example played a drum kit, I can't see any identifiable traits on the spectrogram (apart from similarities to white noise).
Sawtooth
Spectrogram of the VRC6's Sawtooth channel.
As a bonus, let's also check out the Sawtooth channel from the VRC6 expansion. To begin with, a perfect Sawtooth wave is made of all the harmonics and each exhibits decreasing amplitudes (where amplitude = 1 ÷ harmonic number ).
This is quite a requirement for digital equipment and is naturally unaffordable for a game cartridge (which may not even need such perfection!). So, similarly to the APU's triangle waves, the VRC6 sequences Sawtooth waves in 7 cycles (and thus produces similar step-ladder effects).
Consequently, the respective spectrogram appears very cluttered, as the VRC6's approximation techniques fill the wave with extra harmonics in various places.
Conclusion
Well, it seems that the NES' synthetic waveforms are nowhere near shaped as the theory dictates. Does this mean the APU is flawed? Not at all! The design of the APU ultimately gave this console its unique and identifiable sounds - these characteristics, whether intentional or not, made the spectrograms display unusual patterns.
As a side note, while perfect geometry may be visually appealing, our ears are surprisingly less receptive to waveforms with excellent edges! (you may start hearing popping noises).
Looking ahead, sound analysis using spectrograms will prove useful in other articles, whether for simple analysis or for comparing different systems. However, it is important to note that these graphs are not the mother lode tool by all means, especially with sound samples that have been mixed with too many channels or instruments, which can complicate their decomposition. Nevertheless, they provide a solid foundation for any kind of objective study.