Nintendo held on to the cartridge medium for storage instead of adopting optical discs. As a consequence, games enjoyed higher bandwidths (an average of 5 MB/sec) while also being more expensive to manufacture. The largest cartridge found in the market had a capacity of 64 MB.
Inside cartridges, manufacturers often included extra memory (in the form of Electrically Erasable Programmable ROM (EEPROM), flash, or Static RAM (SRAM) backed by battery) to hold save data. However, this became less essential, as certain accessories (like the Controller Pak) provided alternative storage.
Cartridges interface with the RCP using a dedicated 16-bit bus known as the Parallel Bus (PBUS) or 'Parallel Interface' (PI) .
Source Development Kit
In general, development was mainly done in C and assembly, the latter was often required to achieve better performance.
While we've seen this system supports 64-bit operations, the new instructions were rarely used since, in practice, 32-bit instructions happened to execute faster (given that the R4300i/VR4300 came with a 32-bit data bus).
The libraries in the official SDK feature several layers of abstractions to command the RCP. For example, C structs like the Graphics Binary Interface (GBI) were designed to facilitate the assembly of Display lists . The same applies to audio functions; its struct was called Audio Binary Interface (ABI).
In terms of microcode development, Nintendo provided a set of prewritten microcode programs to choose from. However, if developers wanted to customise them, that would indeed be a challenging task: the Scalar Unit instruction set wasn't initially documented. Later on, however, Nintendo and SGI changed their stance and released some documentation and tools for microcode programming .

An SGI Indy I came across at The Centre for Computing History (Cambridge, UK) when I visited in August 2024. By comparison, this computer houses a MIPS R4400 CPU, an improved successor of the R4000 (all in all, miles ahead of the VR4300).
Hardware used for development included workstations supplied by SGI , such as Indy machines equipped with an extra daughterboard called U64. This board contained the hardware and I/O of the retail console. Development tools were supplied for Windows-based computers as well .
Furthermore, there were third-party tools such as custom cartridges housing a long ribbon cable that connected to the workstation. These cartridges fitted in a retail Nintendo 64 but incorporated internal circuitry to redirect 'read' requests from the console to the workstation's RAM. The deployment and debugging process involved transferring a copy of the game into RAM so that, upon powering the console, it would start reading from there.
The alternative medium
Interestingly, the PBUS branches out to a secondary connector located on the underside of the Nintendo 64's motherboard. This was intended for the Nintendo 64 Disk Drive (64DD), a peripheral that worked as an 'extra floor' beneath the console, housing a proprietary magnetic disk reader. The disks offered up to 64 MB of storage capacity.
Although the 64DD was only released in Japan, it opened the door to an alternative (and cheaper) medium for distributing games.

The Nintendo 64 Disk Drive .
Released on 01/12/1999 in Japan.

The 64DD attached to the console .
The magnetic medium used by the 64DD is slower than cartridges, with transfer speeds of up to 1 MB/sec - but still faster than 4x CD-ROM readers. The disks are double-sided and operate using Constant Angular Velocity (CAV), like the later miniDVD. The smallest readable area is called a block and corresponds to half of a concentric circle on the disk surface.
Notably, the reader doesn't include buffer memory, so data is streamed directly into RDRAM for execution. To accommodate the increased memory demands, Nintendo bundled the RAM Expansion Pak with the 64DD. In doing so, it also standardised the extended RAM space, ensuring that all 64DD games could reliably take advantage of the additional memory.
Furthermore, portions of the disk are rewritable to enable save storage. The amount of writable space varies depending on the disk type (Nintendo offered seven types).
On the software side, game data is structured with a filesystem called 'Multi File System' (MFS) provided by Nintendo with their SDK . Games could access disk data either through the filesystem or via direct block-level access.
The Disk Drive also houses an internal ROM, known as 'DDROM', which stores code executed by the N64 during boot. This code bootstraps the disk and shows the splash animation, effectively adding a new IPL stage on top of the traditional boot process. The ROM also stores fonts (Latin and Kanji) and some sounds.