Have you ever wondered how the graphics in your favorite video games worked? This is the start of a series on game graphics, and what better place to start than how exactly the original Mario Bros. got those glorious pixely pixels onto the screen. Buckle in, because we’re “racing the beam” with systems like the NES, Commodore 64, and many other classics from the 1980s.

And to understand the 1980’s, it’s important to understand how the televisions of the time worked. Cathode Ray Tube (CRT) televisions work by precisely bombarding a phosphor layer with electrons, which excites the phosphor, which then releases visible light. The beam scans from left to right then top to bottom, giving each pixel a small fraction of a second of time. All of this effectively means that pixel data needs be sent at the same time as when the pixels are being lit up, which is why this type of graphics is often dubbed “racing the beam”.

Another thing to understand about the 1980s is that 64 kB was a lot of memory. Back then, there was almost never enough memory to store an entire frame of video, nor were the CPUs of the time anywhere near fast enough to fill such a framebuffer. Because of this, these computers grouped 8 x 8 pixels into “tiles” or “characters” and used those repeatedly when they needed them, greatly reducing the amount of memory required. Such a tile is usually represented by a single byte in the “tile map” memory, which is in turn used as an index into the “tile data”, which is what actually stores the images of the tiles.

Coloring By Numbers

Colors were implemented in many different ways, but there are two solutions that are far more common than the rest: tile colors and tile palettes.

The simpler approach of the two, which I call “Tile Colors”, gives you two colors to work with. The tile stores the “picture” and then colors it in with a per-tile foreground color, against a larger background color. The tile color was stored in a similar way to the tile map and was a simple index to a color value. For example, the Commodore 64 uses one background color shared across all tiles and another foreground color per tile.

Using just tile colors and a background color might sound very restrictive (and it is) but that didn’t prevent Commodore 64 games from looking great. This screenshot demonstrates the technical limitations of tile colors very well, showing clearly a black background with a single color per tile in the foreground, but also tiles of different colors.

The more advanced approach is to use multiple bits per pixel in the tile data and to assign a palette to a tile using the color data. As is the theme here, the color palettes were in yet another piece of memory, though this time it’s usually in the actual video chip instead of RAM.

This screenshot shows a combination of multiple layers for tiles and palettes for said tiles, allowing for parallax scrolling of a background and more colorful tiles respectively.

Bitmap Modes

Some games do decide to use more memory in return for a higher resolution. In such “bitmap modes”, there is no “tile data” to look up because instead of using tiles, the image data is used directly. Bitmap modes trade memory for detail, but still use tile-sized chunks for colors. This is often eight times as much memory, because tiles are eight by eight, which is often too much if you also want to play a game. But for those few games that do use bitmap mode during gameplay there’s another drawback: it takes orders of magnitude more time to update than tiles.

This art shows what a Commodore 64 is capable of in multi-color bitmap mode. In this mode, the pixels are stretched horizontally, and can have one of four colors: background, alternate 1 and 2 and foreground. This art combines multi-color bitmap mode and manual dithering to give the illusion of higher color resolution.

Sprites

Tiles are great for static background images, but can be quite limiting when displaying moving characters, and bitmap mode still wouldn’t fix clashing colors. This is where sprites come in: sprites are usually made out of one or more tiles, or tiny bitmap images, that can be placed arbitrarily on the screen. Instead of replacing all the pixels like tiles do, sprites are overlays on top of the background image.

This screenshot from Super Mario World shows about 16 sprites, which is half of how many the SNES can show on one scanline. Mario, the shell, the points, and the turtles are all rendered as sprites.

On the SNES, for instance, sprites are tiny bitmap images that use palettes the same way tiles do, so they’re even more flexible. Rendering these to the screen was taken care of by the graphics hardware, so the CPU overhead in using a sprite was minimal.

Scanline Interrupts

Scanline interrupts are a technique that’s more reliant on the CPU than on the graphics chip. They use the fact that the video chip must necessarily know what scanline it is drawing to synchronize the CPU to it. This allows effects that render each scanline normally, but change the parameters between said scanlines.

This allows games to show multiple different graphics modes, more sprites than there can be on one scanline, or even make the illusion of 3D! In combination with affine transformations, Super Mario Kart created the illusion of depth. The street is made up of the same alternating patterns, but depending on where they’re drawn on the screen, they are drawn wider or thinner, and shifted off to the left or right, and voila, the illusion is complete.

We’ll talk more about the matrix math in game graphics later on, but note that this is really just multiplying to change the width, and adding an offset to make the road curve.

Conclusion

While racing the beam is a very different approach to graphics than we have now, it’s definitely possible to make games look good and even do some very nice looking art. Basic tiles are good at displaying text but are inferior in detail to bitmap mode. On the other hand, bitmap mode consumes much more memory than tiles do, and is too detailed to be effectively updated by 1980s CPUs alone. Sprites can be used to have moving elements in otherwise “static” scenes and are present in almost all implementations of racing the beam. Finally, scanline interrupts allow programmers to get much more out of the hardware by strategically changing the settings at the perfect time. Putting it all together lets an animated plumber drive a go-kart.

Next, we’ll step forward in time to when memory wasn’t so scarce, and talk about basic 2D graphics with a framebuffer and how shape primitives are drawn. See you then!