Generally speaking, there are two operating systems residing in the Wii. One is executed on Broadway (main CPU) and the other one on Starlet (I/O CPU). Both reside inside those 512 MB of NAND memory and can be updated.
Starlet's OS
Starlet is already an interesting piece of hardware, but its software is even more intriguing. You see, not only does this OS have complete access to every single corner of the console, but it's also the first thing that runs when the power button is pressed.
Starlet runs a system unofficially referred to as Input/Output Operating System or 'IOS' (please, do not confuse this with Apple's iOS) . IOS is a fully-featured operating system composed of:
- A Microkernel: Controls the ARM9 CPU, executes processes and talks with other hardware using drivers.
- Drivers: Enables the communication with hardware outside the CPU (I/O).
- Processes: Performs a task, such as network management or implementing a file system.
- Cryptographic core: Accelerates encryption-related operations (AES and SHA-1 only).
With this in mind, the main job of IOS is to offload the workload of the main CPU by abstracting I/O and security. For that reason, programmers don't have to worry about those matters. In order to accomplish this, Starlet reserves between 12 and 16 MB of GDDR3 RAM for its tasks, the rest is used by Broadway and the GPU.
Broadway and Starlet communicate with each other using an Inter-Process Communication or 'IPC' protocol: In a nutshell, both CPUs share two registers each. One CPU can write on the other's registers (the written data may represent a command or a value) and from there, the receiver CPU can perform a function in response.
The update system of IOS is a bit tricky: Updated IOS versions are not installed on top of old ones, but in another slot instead (the reserved area in NAND for IOS is divided into 'slots'). This is purely for compatibility reasons, since it allows older Wii software to keep using the same IOS version it was developed for.
Nintendo often released IOS updates to improve hardware support (which was necessary when a new accessory was shipped). There's only one exception when IOS updates actually replace older ones: When a specific version was discovered to have an exploitable vulnerability. This was only for security reasons.
When a GameCube game is inserted, a different thing happens: Starlet boots a MIOS instead. This IOS variant just orders Starlet to emulate the original IPL.
Broadway's OS
This one is commonly known as the System Menu and effectively runs on the main PowerPC CPU (Broadway).

System menu with lots of channels installed.

Settings menu used to change settings.

The message board stores letters grouped by date.
Compared to IOS, I wouldn't consider this a 'fully fledged' OS, but more like a 'program' that allows the user to perform the following operations:
- Start the Wii/GameCube game: Only if there is a valid one inserted.
- Change console settings: Including time, date, video mode or sensor bar location, among others.
- Run apps: One of the novelties of this console is the ability to install small Wii games (called 'WiiWare'), retro games ('Virtual Console' games) or just convenient applications (such as an internet browser). Nintendo called these channels, but they are also referred to as titles by the OS.
- Users can download/buy channels through a pre-installed channel called Wii Shop Channel.
- Virtual Console titles embed an emulator to run the game itself. Curiously enough, the emulator is not shared across the system or even between games of the same platform. This allows to optimise the emulator for specific games.
- Send/Receive messages: Wiis have a unique ID (burned in their SEEPROM chip) which can be shared to exchange messages between other Wiis. Messages can be seen on the Message Board.
- Nintendo and Wii games also used this medium to provide a newsletter as well.
Just like IOS, Nintendo released multiple updates to this system too. Some fixed security holes, others added more features. A notable new feature was the ability to store channels on the SD card.
Any program running on Broadway (including the System Menu) relies on a specific IOS version to work. When a game or a channel is booted, Starlet reboots itself using the declared version of IOS needed.
Update medium
Nintendo refers to them as System updates. They contain the two OSes in the same package and use ordinal numbers for versioning. The last version known is 4.3E, released in June 2010.
System update packages can be fetched from Nintendo's Servers or game discs. Users can manually check for updates using the System Menu. Updates are forced if a game requires a specific version of IOS that is not installed (and the disc happens to contain the required packages).
Boot sequence
So far we have discussed two very different operating systems that reside in this console and run concurrently. This seems fairly simple, although both have to be carefully coordinated during the start of the console to work properly afterwards.
That being said, the boot process of this console is as follows :
- User taps the ON button on the console.
- Boot0 stage: Starlet runs a hardwired program found in its embedded Mask ROM (1.5 KB).
- In a nutshell, Starlet decrypts and checks the integrity of the first 96 KB of NAND, Starlet then calculates its hash and compares it against a saved hash found on Starlet's embedded OTP memory. If both hashes do not match, then the console is induced in an infinite loop.
- Boot1 stage: Starlet runs a small program found in that aforementioned 96 KB of NAND.
- This program will just instruct Starlet to initialise (clear) the 64 MB of GDDR3 RAM, then decrypt and verify the rest of NAND.
- Boot2 stage: Starlet loads the initial IOS (needed for the System Menu) and then kickstarts Broadway.
- Broadway starts the System Menu. The user is now in control.