What you see on screen is produced by a large chip designed by Silicon Graphics called Reality Co-Processor (RCP), which runs at 62.5 MHz . This package contains a lot of circuitry, so don't worry if you find it difficult to follow - the graphics subsystem has a very complex architecture!
This design is based on the philosophy that the GPU should not be a 'mere' rasteriser like the competition. Instead, it should also be capable of accelerating geometry calculations (offloading work from the CPU), and for that, more circuitry is needed.
Architecture
The Reality Co-Processor is divided into three main modules, two of which are dedicated to graphics processing:
Reality Signal Processor
Architecture of the Reality Signal Processor (RSP).
Also known as RSP, the Reality Signal Processor is a CPU package composed of :
- The Scalar Unit: Another cut-down derivative of the MIPS R4000. This time, it only implements a subset of the MIPS III ISA, and therefore lacks many general-purpose functions, including interrupts, the 64-bit extension, multiplication, and division.
- The Vector Unit: A co-processor that performs vector operations with 32 128-bit registers. Each register is sliced into eight parts to operate eight 16-bit vectors at once (similar to SIMD instructions in conventional CPUs). As you can see, this component does the heavy lifting for the Scalar Unit (while also working concurrently).
- The System Control: Another co-processor that provides DMA functionality and controls its neighbouring module, the RDP (explained in the next section).
To operate the RSP, the CPU stores a series of commands called Display list in RAM, along with the data to be manipulated. The RSP then reads the list and performs the required operations on it. Available functions include geometry transformations (such as perspective projection), clipping, and lighting.
This may seem straightforward, but how does it perform these operations? Well, here's the interesting part: unlike its competitors (the PlayStation and Sega Saturn), the geometry engine is not hard-wired. Instead, the RSP contains some memory (4 KB for instructions and 4 KB for data) to store microcode : A small program, with no more than 1000 instructions, that implements the graphics pipeline. In other words, it tells the Scalar Unit how to process the graphics data. The microcode is fed by the main CPU at runtime.
Nintendo provided multiple microcodes to choose from and, similar to the SNES' background modes, each one balances system resources differently .
The resulting data is transmitted either via a dedicated bus called XBUS or through main RAM. Over the console's lifetime, this choice fluctuated due to memory constraints: the XBUS path was faster but required additional buffers in the RSP's internal memory to hold (and transfer) the new data. To give you an idea, early microcode programs like Fast3D offered both options. However, the later and faster F3DEX focused entirely on main RAM. Finally, its successor, F3DEX2, reintroduced XBUS support after sorting out memory usage.
Reality Display Processor
Architecture of the Reality Display Processor (RDP).
After the RSP finishes processing the data, it will start sending rasterisation commands to the next module - the Reality Display Processor (RDP) - to draw the frame.
The RDP is another processor (this time with fixed functionality) that includes multiple engines to rasterise vectors, map textures onto polygons, mix colours, and compose the new frame.
It can process either triangles or rectangles as primitives, the latter being useful for drawing sprites. The RDP's rasterisation pipeline contains the following blocks :
- A Rasteriser: Converts primitives (made of vertices) into pixels.
- A Texture Unit: Processes textures using 4 KB of dedicated memory (called 'TMEM'), allowing up to eight tiles to be used for texturing. It can perform the following operations on them:
- Bilinear filtering: Maps the selected 2D texture onto the 3D shape and smooths it to avoid pixelated areas (caused by oversampling).
- A 'complete' filter would require four points to carry out the interpolation; however, this console uses only three (triangular interpolation), resulting in some anomalies. Thus, certain textures have to be 'adapted' beforehand.
- Mip-Mapping: Automatically selects a scaled-down version of the texture depending on its level of detail. This avoids computing large textures that are seen far from the camera and prevents aliasing (the result of undersampling).
- If enabled, the RDP maps textures using trilinear filtering instead. This new algorithm also interpolates between mipmaps to soften abrupt changes between the levels of detail.
- Perspective correction: The chosen algorithm for mapping textures onto triangles. Unlike other inverse-mapping algorithms, this one accounts for the depth value of each primitive, achieving more faithful results.
- Bilinear filtering: Maps the selected 2D texture onto the 3D shape and smooths it to avoid pixelated areas (caused by oversampling).
- A Colour Combiner: Mixes and interpolates multiple layers of colours. As the next-generation color math unit, it performs addition, multiplication, and subtraction of colour values.
- A Blender: Mixes pixels with the current frame buffer to apply translucency, anti-aliasing, fog, and dithering. It also performs z-buffering, which I explain more later on.
- A Memory Interface: Used by the previous blocks to read from and write to the current frame buffer in RAM and/or populate TMEM.
The RDP provides four operating modes, each of which combines these blocks differently to optimise specific tasks.
Since this module constantly updates the frame buffer, it interacts with main RAM in a unique way. Remember the unusual 9-bit? The ninth bit is used as metadata for frame buffer-related calculations (i.e. z-buffering and antialiasing) and is only understood by the Memory Interface.
Remaining steps
The resulting frame must be sent to the Video Encoder to be displayed on-screen. This is taken care of by DMA and the Video Interface component.
The Video Interface (VI) acts as a bridge between the game's framebuffer and the television signal. It basically converts the rendered frame into a format the TV will understand, which tends to vary by region. For instance, the Video Interface can broadcast frames of 640x480 or 320x240 pixels to NTSC tellies; whereas PAL systems get 640x576 or 320x288 frames . To help with the translation, the VI provides anti-aliasing (in conjunction with the RDP), scaling, and many colour correction modes.
Other capabilities include supporting up to 32 bits of colour depth (16.8 million colours), but it's worth mentioning that using the maximum settings can be resource-hungry, so programmers often opt for lower specifications to free up enough resources for other services.
Quick demo
Let's put all the previous explanations into perspective, for that, I will use Nintendo's Super Mario 64 as a case study to show, in a nutshell, how a basic frame is composed. Just bear in mind that, in practice, however, games may employ additional buffers to compose richer frames.
Vertex Processing
Primitive view of our scene. In order to save polygons, some characters are modelled using sprites (quads)
Initially, materials such as 3D models are located in the cartridge ROM. However, to keep a steady bandwidth, we need to copy them to RAM first. In some cases, the data is pre-compressed within the cartridge, requiring the CPU to decompress it before use.
Once that's done, it's time to build a scene using our models. The CPU could carry out the entire graphics pipeline by itself, but that takes ages, so many tasks are offloaded to the RCP. The CPU will instead send orders to the RCP; this is done by carrying out these steps:
- Compose the Display Lists that contain the operations to be executed by the RSP, and store them in RAM. The structure of Display Lists is dictated by the choice of microcode .
- Point the RSP to the location of the Display Lists.
- Upload the chosen microcode to the RSP, kickstarting the Scalar Unit.
Afterwards, the RSP begins working on the first batch of tasks. The result is then passed to the RDP in the form of rasterisation commands.
Pixel processing
Rendered frame (Tada!).
So far, we've managed to process our data and apply some effects, but we still need to:
- Rasterise vectors, apply textures and other effects.
- Display the frame buffer.
As you might expect, these tasks are performed by the RDP. Additionally, to make this work, textures must be transferred into the 4 KB of Texture Memory using DMA.
Unlike the previous processor, the RDP's pipeline is fixed, but we can select the optimal mode of operation based on workload, performance and specific tasks needed. For instance, commanding the RDP to render a single texture is faster than combining two.
Once the RDP finishes processing the data, it writes the final bitmap to the frame buffer area in RAM. Afterwards, the CPU must transfer the new frame to the Video Interface (VI), preferably using DMA. The VI, in turn, sends it to the Video Encoder for display .
Designs
Here are some examples of classic 2D characters from the Super Nintendo that were redesigned for the 3D era. Notice the texture detail when compared with models from other consoles of the same generation.
Interactive model available in the modern edition
The Legend of Zelda: Ocarina of Time (1998).
704 triangles.
Interactive model available in the modern edition
Kirby 64: The Crystal Shards (2000).
516 triangles.
Modern visible surface determination
If you've read about the preceding consoles, you'll have come across the never-ending problem of visibility of surfaces, and by now may think polygon sorting is the only way out of this. Well, for the first time in this series, the graphics chip features a hardware-based approach called Z-buffering. In a nutshell, the RDP allocates an extra buffer (called Z-buffer) in memory. This has the same dimensions as a frame buffer, but instead of storing RGB values, each entry contains the depth (Z-value) of the nearest pixel relative to the camera.
After the RDP rasterises the vectors, the z-value of the new pixels is compared against the corresponding value in the z-buffer. If the new pixel has a smaller z-value, it means the new pixel is positioned in front of the previous one, so it is applied to the frame buffer and the z-buffer is also updated. Otherwise, the pixel is discarded.
Overall, this is a hugely welcome addition: programmers no longer need to worry about implementing software-based polygon sorting methods, which drain significant CPU resources. However, the z-buffer does not prevent feeding unnecessary geometry (whether discarded or overdrawn, both wasting resources). For this, game engines may choose to include an occlusion culling algorithm to discard unseen geometry as early as possible.
Secrets and limitations
SGI clearly invested a lot of technology into this system. Nevertheless, the Nintendo 64 was a console intended for the home market and, as such, needed to keep its cost down. Some tough compromises resulted in difficult challenges for programmers:
Pipeline Stalls
The deprecation of load delay slots in MIPS II, in favour of automatic pipeline stalls, had the benefit of alleviating the need for filler instructions. However, this breakaway from MIPS' core philosophy also exposed new bottlenecks. Due to the large number of components and operations in the graphics pipeline, the RCP became highly susceptible to excessive stalls: an undesirable situation in which sub-components remain idle for considerable periods, as the required data is delayed at the back of the pipeline.
This invariably results in performance degradation, and it is the programmer's job to avoid. Although, to help mitigate this, MIPS CPUs had long provided pipeline bypassing, a mechanism that enables similar instructions to execute at a faster rate by bypassing some execution stages that can be skipped .
For example, if the CPU has to compute sequential ADD instructions that are dependent on each other, there's no need to write the result back to a register and then read it again after each operation. Instead, the CPU can forward the values within the datapath and perform the write-back only after the last ADD has been completed.
Texture memory
The RDP relies on 4 KB of Texture Memory (TMEM) as a single source for loading textures. Unfortunately, 4 KB proved to be insufficient for high-resolution textures. Furthermore, when mipmapping is activated, the available memory is reduced to half.
As a result, some games used solid colours with Gouraud shading (Super Mario 64 being a notable example), while others relied on pre-computed textures (particularly, where multiple layers had to be combined).
The universal video out
Nintendo carried on using the 'universal' Multi Out port from its predecessor. Bad news is that it no longer carries the RGB signal! It looks to me like another cost-saving measure, given that RGB wasn't widely adopted in the previous console.
The good news is that, on early revisions of the Nintendo 64, the three RGB lines can still be restored by soldering some cables and fitting an inexpensive signal amplifier. This is possible because the video Digital-to-Analogue Converter (DAC) still transmits an RGB signal to the video encoder . However, later motherboard revisions combined both chips, so the only remaining option is to bypass the video DAC and encoder altogether with custom circuitry that exposes RGB signals.