PlayStation 3 Architecture

Now that home consoles have become powerful multimedia hubs, a more convoluted operating system will be needed to provide users with more services and games with a thicker layer of abstraction. All of these, while keeping security and performance up to the task.

Consequently, terms like shell or BIOS are no longer used to describe this area, not because they don't exist anymore but because they describe a small fraction of the new system. The generic term is now 'operating system', this comprises many areas (boot loader, kernel, user interface) analysed separately. As always, I recommend checking out the PSP's OS first, since its modular design is a recurrent ingredient in the PS3.

Cell's privileged security

Before we dive into details, I need to mention the different modes of operation in Cell. I originally planned to describe this in the 'CPU' section, but since that got incredibly dense, I'll introduce it here where you'll see its practical use right away. Furthermore, its modes also affect the design of any operating system running within Cell - not just the one Sony developed for this console.

Having said that, to safeguard against unauthorised access to sensitive data and/or resources, Cell implements a set of privilege levels inherited from the PowerPC specification. In other words, Cell executes programs in two modes:

• Privileged mode: Cell grants access to every corner of its hardware (registers, memory addresses, opcodes, etc) . For security reasons, this mode should only be used by the operating system's core (i.e. the Kernel).
  • Moreover, Cell was also prepared to run multiple operating systems concurrently and to achieve that in the hardware level, 'privilege mode' can be further divided into Privilege 1 and Privilege 2. 'Privilege 2' is meant to be used by a Kernel, while 'Privilege 1' is used by a Hypervisor, the latter arbitrates resources between different Kernels running at the same time.
  • The 'hypervisor' functionality also turned into an area of research in IBM's headquarters .
• User Mode: As the name indicates, Cell only grants a limited set of resources and it's directed to traditional applications running on top of the operating system. If for any reason, a program requests access to a protected location, then execution jumps to the Kernel or Hypervisor to request whether access should be granted or not.

Additionally, SPEs contain a modus operandi called isolated mode and it shields the execution process within the SPU so no external unit (the PPE or other SPEs) can access it until the SPU finishes. This can be activated after uploading a program to any SPE and ensures that the processor is not tampered with while the sensible code (i.e. an encryption routine) is being executed.

Sony's operating system, which I'm going to describe in the following paragraphs, uses all the modes described to handle its security.

Overview

As I said before, the OS is quite complex. So, to be able to follow this section without too much trouble, we can divide the type of files we'll find in the operating system of this console into different layers:

• Loaders: to make a long story short, programs/binaries in this console are systematically encrypted. So, 'Loaders' are programs that execute 'real' programs. To put it in another way, Loaders grab binaries, decrypt them, check their authenticity and finally send them to the respective processor (the PPE or either SPE) for execution. If that doesn't sound complicated, Loaders are chained together to protect the software even further. Finally, Loaders are found across many media.
  • Some Loaders are updated by Sony (through software updates) while others can't be changed. This is independent of whether they are installed in re-writable storage, as some loaders are encrypted using console-specific keys, so they can't be altered after the console leaves the factory (at least through traditional means).
• System files: these comprise low-level binaries (executed through Loaders), metadata for organising the hardware, utilities and other assets (i.e. fonts, imagery). Just like Loaders, there are console-specific system files that cannot be replaced or autogenerated.
  • Some binaries borrowed code from the Free BSD and NetBSD projects .
• User Content: These include configuration files (i.e internet settings), data used by games (i.e. game installation files and saves) and data generated automatically by the console (i.e. hard drive information).
  • Unlike the other layers, destruction of this data does not lead to catastrophic outcomes.

OS' security hierarchy

Generally speaking, the PS3's OS is designed with the same modular approach as the PSP. To recall the previous article, the OS is made of multiple modules. These may serve the user (like a game or app) or reside in memory indefinitely to serve other modules (in the form of system calls and/or drivers). Some modules have more privilege access than others (kernel module vs. user module).

Diagram showing how the components of the PlayStation operating system fit in Cell's privilege levels.
References to 'OtherOS' are explained further in the next sections.

The operating system, throughout its lifecycle, will call upon many modules with different privileges. Sony constructed its OS so that modules run under Cell's three privilege levels:

• Level 1: here is where a Hypervisor programmed by Sony resides. Also referred to as lv1, this program is the door to every single bit of this console and chained to exceptions triggered by the MMU. That being said, the hypervisor only accepts requests from programs authorised by Sony (residing in the next privilege level). While the Hypervisor resides in memory, it also provides low-level system calls and FAT16 filesystem support.
• Level 2: naturally reserved for the Kernel, a privileged program also named lv2 or 'Supervisor'. The kernel abstracts the hypervisor so level 3 programs don't interact with it directly. The Kernel provides multi-threading functions for both PPU and SPU. Ultimately, the Kernel bootstraps userland modules.
• Level 3: the rest of the programs (called userland or userspace), including games and the visual shell, run at this level. These plebs are under the will of the Kernel to communicate with the console's hardware and cannot spawn any new process or program unilaterally.

Storage medium

With all being said, where is all this data stored? From a general user's perspective, there are only two visible media: Blu-ray discs for games and a Hard Disk for saves. Well, there are a few more, so we will now examine every one of them!

Cell BootROM

It turns out, within Cell there's a small ROM hidden somewhere that manufacturers may store a 'protected' boot-loader. IBM provides this space to spare any company (not just Sony) from having to manually implement tailored obfuscation methods to protect their boot code, as off-the-shelf components are not always prepared for bespoken needs.

Since this piece is already physically protected with obfuscation, it doesn't have to be encrypted. Thus, it's ideal for a first-stage boot-loader (which can't be encrypted) and the PlayStation 3 stores its early boot stage there.

NAND/NOR Flash Memory

Remember about those 256 MB of NAND flash I briefly mentioned before? Well, here is where most of the operating system resides. That is, until Sony released the CECHH model in late 2007, replacing the 256 MB NAND for a puny 16 MB NOR. As a consequence, some files had to be moved elsewhere. For the sake of simplicity, let's first see what these chips store :

• Console-specific Loaders: specifically, two loaders called bootldr and metldr. These files are encrypted with a key engraved during manufacturing, so they can't be replaced!
  • Be that as it may, there are hidden functions in Sony's hypervisor that allow to update these, although, for some reason, they have never been used .
• CoreOS: the first half of the operating system. It mainly consists of more loaders that will continue the boot process and eventually bootstrap the second half (GameOS). CoreOS also supplies the Recovery Menu, an alternative shell that contains maintenance utilities that users can use to (attempt to) repair their console.
• Unique IDs: similar to the PSP's IDStorage, these are used by the console to control secured hardware like the Blu-ray drive; or by Sony to authenticate the console with their online servers (i.e. the IDPS key).
• Security assets: some programs depend on them to perform security operations. For instance, Blu-ray movies with DRM check a block called Virtual Table Rights Management (VTRM). Sony also stores revocation tools and records to blacklist security certificates that have been compromised in the past.

Due to their larger size, models with NAND Flash also store the remainder of the OS (called GameOS or devflash). This includes:

• The Visual Shell (VSH): a continuation of the PSP's signature interface, it's also bundled with tons of modules (plugins) and assets.
• Emulators: aforementioned programs that enable a PS3 to run PS1, PS2 or PSP games. The specific PS2 emulator loaded depends on the console revision (whether it has full PS2 hardware, partial hardware or uses no PS2 hardware whatsoever).
• Runtime libraries: programs developed with Sony's SDK are dynamically linked to a set of libraries stored here.
• Blu-ray Player: programs that handle interaction with the Blu-ray drive and movie decoding.
• System assets: such as fonts and certificates that binaries depend on to work.

As if it wasn't enough, NAND consoles also house other data like the xRegistry (collection of network settings, PlayStation Network accounts and a list of Bluetooth-paired devices), more revocation records and a Loader for OtherOS (a really interesting piece which we'll discuss more in the next paragraphs).

Hard Drive

The debuting 2.5-inch Hard drive, ranging between 20 GB and 500 GB (as more revisions were shipped), provides persistent data storage for:

• User content: including game saves, trophies (more information in the 'Games' section) and other user-related data.
• Game assets: games can copy files from the disc to the hard disk to improve loading times. These are treated as 'game data' by the operating system.
• Cache: a separate 2 GB partition is available to games for temporary storage (in case main RAM is not enough).

NOR systems, however, also store GameOS in the HDD. As a consequence, whenever the user swaps out the hard drive, the console asks for an update file to re-install GameOS into the disk. Be that as it may, neither NOR nor NAND systems will boot without the hard disk.

Certain user data can be backed up using a USB stick and then moved to another console if necessary, though this process re-formats the newer console before copying the old data.

eMMC

In 2012, Sony unveiled a redesigned revision of the console called 'SuperSlim' (codenamed CECH-4xxx). These were available in three flavours: One with a 250 GB Hard drive, another with a 500 GB Hard drive, or a third option with only a 12 GB eMMC internal flash. The first two options follow the file system layout implemented in NOR models, while the third one stores everything in the eMMC (including user data) and follows the NAND layout to store system files.

There's a catch in the eMMC model, however. In the place of the NOR chip, Sony fitted a Panasonic 'MN66840' chip which, according to the PS3 Dev Wiki , it seems to redirect the NOR bus to the eMMC. I assume this is just a trick to save costs, as it recycles the same Southbridge from the other variants.

Curiously enough, if the user decides to fit a Hard Drive into the eMMC model, the console moves all user data away from eMMC to the new Hard drive. As a consequence, the user can fully utilise the hard drive, though the empty space in eMMC is now left to waste.

Boot process

All right, using all the previous knowledge, you are going to learn now how the system boots up - and let me tell you, it's pretty complicated. The reason is simple: Sony doesn't want you fiddling with their hardware or software, so they built many layers of obfuscation and encryption to prevent you from breaking in and side-loading your own code (and hopefully give up and keep buying games/movies/whatever) but, as history will tell you, the opposite happened.

In the following section, I'm going to describe what this console does once you push the power button. Note that this process only drastically changed once (after hackers cracked it). So, for simplicity purposes, we'll start with the 'original' boot process (implemented before system version 3.60) :

1. A separate chip in the motherboard (called Syscon) powers on and executes instructions from its internal ROM. It then sends a 'Configuration Ring' to Cell via SPI (a serial connection), this initialises Cell and deactivates the eighth SPU. Finally, it latches the power line and gives life to Cell.
2. Cell's PPU reset vector points to its hidden ROM, which stores the routines to locate and decrypt bootldr from Flash. The decrypted piece is then loaded by the first SPU in isolation mode.
3. The now-isolated SPU, having loaded bootldr, initialises part of the hardware (XDR memory and I/O interfaces) and decrypts a binary named lv0 and instructs the PPU to run it.
4. The PPU, now executing lv0, decrypts metldr (a console-specific loader) and sends it to the third SPU, again in isolation mode.
5. The SPU2, now executing metldr, executes five more loaders sequentially:
   1. lv1ldr decrypts and loads lv1, which contains the Hypervisor that takes over the first privilege level. Moreover, lv1 sets up the hard drive, Blu-ray drive and RSX.
   2. lv2ldr decrypts and loads lv2, which contains the kernel and runs on top of the hypervisor. It also finishes initialising RSX, the PS2 emulation, Bluetooth, USB controller and the Multi-card reader.
   3. appldr decrypts and loads vsh (the Visual Shell) and other dependencies. vsh will later enable the user to load a game.
   4. isoldr decrypts and loads modules that will run in the third SPU in isolation module. These modules are critical for security and perform many cryptographic functions throughout the console's lifecycle. Consequently, the third SPU is reserved for security functions and games can't use it (leaving only six SPEs for games).

The PPU, having loaded vsh, grants the user control through a graphical user interface, which manifests itself with an iconic orchestral splash sound followed by the XMB menu.

Revisioned boot process

In March 2011, a hacker known by the name 'GeoHot' broke the security of metldr, thereby compromising the authenticity of subsequent loaders. Thus, Sony retaliated by issuing security updates for both their hardware and software. These fixes are further discussed in the 'Anti-piracy' section of this article.

Visual Shell

Are you getting tired from all this theory? Let me switch to something everyone can actually see: The Visual Shell.

XrossMediaBar (XMB), a new user interface that gained international recognition two years before, has been slightly adapted so it can be interacted from the sofa (the so-called '10-foot user interface') and expanded to take advantage of 'full HD' resolution (1920x1080 pixels).

XMB in the PSP (2004).
Rendered at 480×272 pixels.

XMB in the PS3 (2006).
Rendered at 1920x1080 pixels.

While PSP users will find many familiarities, Sony added a new set of apps that use the potential of Cell, RSX and the Blu-ray drive. Many of them related to multimedia (i.e. video player and image slideshow), television (such as on-demand TV apps, like BBC's iPlayer), social profile (online avatars) and online purchases (PlayStation Now and PlayStation Store, to name a few).

Additionally, since this is a home console that might be shared by multiple members, XMB supports multiple users, where each one may use a different PlayStation Network account and store separate user data (purchased games and saves).

Just like in the PSP, highlighting a game may style the background to get your attention!.

The XMB provides an immense amount of settings, especially helpful when you need to setup your shiny new 1080p telly with 3.1 surround audio.

Various multimedia options.

XMB can install games, updates and expansions (DLCs) using a native package installer.

Finally, the inclusion of a hard drive is a relief for the veterans that in the past were obliged to buy expensive proprietary storage (Memory Stick Pro Duo) whenever they ran out of space.

Lend me your PS3

Impressively enough, not every app bundled with this console had self-interest goals. With the advent of distributed computing and the capabilities of Cell for data science projects, Stanford University joined hands with Sony to enable PlayStation 3 owners to contribute to medical research. The result was Folding@home (pronounced 'folding at home').

Folding@home was an application installed in every PlayStation 3 that, once opened by the user, connected to a central server and ran protein simulations. Moreover, the app was also allowed to run in the background during off-peak times.

Folding@home displaying the work accomplished since the user started the app .

Throughout its lifetime, the joined computing power of 15 million PS3 users worldwide assisted Folding@home with their research towards curing Alzheimer's disease . In the end, Folding@home and Sony retired the app in 2012 and the former lives on on other platforms.

This is my personal opinion, but I enjoy reading about projects that make global contributions using the capabilities of distributed computing, as opposed to the never-ending sensational articles whining about cryptocurrency mining. I guess we shouldn't forget that, with every new powerful technology, there will always be selfless applications developed for it.

A multi-OS proposal

When IBM described Cell from the software level, they mentioned that Cell is capable of running multiple OS at the same time, due to Cell's many execution cores . Thus, Sony took this idea forward and added an option in XMB to install a secondary operating system . This feature was called OtherOS and, in a nutshell, provides a partition manager (XMB just guides the user to resize GameOS' partition and allocate new space for the second OS) and a button to boot from the second OS (thanks to OtherOS' boot files already setup in Flash). So, the user just need to fill the new partition with an OS. Consequently, many Linux distributions (i.e. Ubuntu and Fedora) added the PS3 as another possible target to install it on. You may consider this a spiritual successor to Linux for PS2.

Red Ribbon GNU/Linux is a distribution exclusive for the PS3/Cell and compiled using the PPC64 target .

Thanks to OtherOS, experienced users had the opportunity to develop homebrew applications running on Cell without licensing restrictions, this was particularly interesting for research/scientific purposes , as this console carried a more affordable price tag than a mainframe. For multimedia purposes, the Blu-ray drive and Multi-card reader were also accessible from OtherOS.

On the other side, while OtherOS' privileges may surpass GameOS' (at the kernel level), they don't overtake the hypervisor, which still resides in memory. So, any hardware access from OtherOS still depends on the will of Sony's hypervisor, and it so happens that the latter blocks access to RSX's command buffers (preventing the use of the shader units, among other components used for accelerating graphics operations). Consequently, resulting Linux distributions resort to software rendering (all graphics are drawn by Cell) and then stream the frame-buffer to the RSX for display. While it's disappointing that OtherOS can't make use of the full capabilities of this console, this was probably done to reduce attack surfaces. Ironically, OtherOS' use of Cell is similar to how IBM/Toshiba/Sony may have originally envisioned the PS3!

Sharing the same fate of Folding@home, OtherOS was eventually removed in a subsequent update, but due to different causes (mainly related to security). Shortly afterwards, OtherOS was unofficially restored thanks to software exploits and reverse engineering efforts. At the moment, OtherOS is available if the user installs a custom firmware. I explain this further in the 'Anti-piracy and homebrew' section.

At the time of this writing, developer René Rebe is currently implementing proper xf86 drivers that take advantage of the acceleration provided by RSX and its 256 MB of memory . His work is combined with other developments that removed the restrictions imposed by the hypervisor (initially thanks to the discovery of software exploits and later with the use of a custom firmware, the latter is explained further in the 'Anti-piracy and homebrew' section). Mr. Rebe publishes his progress on his Youtube channel and relies on voluntary donations to continue his work .

Updatability

For the final part of this long section, let's talk about the updatability capabilities of GameOS.

In a nutshell, just like the PSP, Sony distributed PS3UPDAT.PUP files that package all the new OS binaries. Due to the console's security system, only those files which aren't secured with unique console keys and are stored in re-writable storage (Flash, Hard drive, eMMC) are updatable, the rest must stay as it is.

PUP files were distributed through Sony's official website, the XMB update assistant or found in the contents of a game disc (all games embed a PUP file, reflecting the SDK version developed for). Since models with NAND Flash contain only 256 MB of space and store the whole OS there, Sony never released update files with sizes higher than 256 MB.