m9b · part 4 of 6

Picture and sound over HDMI

A 720 × 480 picture at 16 bits per pixel is 675 KB, thirteen times the chip’s block RAM. So the picture lives in PSRAM and is fetched a line at a time, just before it’s shown. The sound rides along in the same cable.

The signal

The mode is 720 × 480 at 59.94 Hz, a standard TV mode with a 27 MHz pixel clock. Every frame is really 858 × 525 clocks: the visible picture, plus the horizontal and vertical blanking left over from CRTs. A line lasts 31.8 µs, and the signal never stops: whatever the CPU is doing, the next line has to be there.

HDMI sends each pixel on three channels, red, green and blue, each its own differential pair, plus a fourth pair for the clock. Each 8-bit colour becomes a 10-bit TMDS symbol, which keeps the line DC-balanced and cuts down on transitions. The encoder is based on Sameer Puri’s open-source HDMI core, one pixel per clock with a pipelined TMDS encoder. Its output goes to the chip’s hard OSER10 serializers, which shift ten bits out per pixel on both edges of the 135 MHz bit clock: 270 Mbit/s per channel.

The GW1NR-9 has no true LVDS outputs on these pins. Each differential pair is two ordinary 3.3 V pins driven in opposition (Gowin’s emulated LVDS buffer), which is fine at 270 Mbit/s.

Why not 720p

The first design ran 1280 × 720, at 742.5 Mbit/s per channel. It worked as long as nothing else happened. As soon as the CPU got busy, on of my momitors lost sync, and another which is way more lenient on signal quality and encoding showed noise and blinked. I ruled things out one at a time: drive strength, PSRAM traffic, sudden load steps, the PLL’s reference clock, moving the CPU to another clock. None of them was it, and the severity changed from build to build with nothing but placement changing. Even the one build that looked fine only looked fine because its firmware was idle.

At 270 Mbit/s the link is solid on both monitors. It’s fewer pixels, but the games I wanted to get running initially only need 320 × 200 of them.

The video engine

Between the PSRAM and the encoder sits the video engine. Its central idea is a line table: one 32-bit entry per screen line, in a small video RAM, saying where that line’s pixels are in PSRAM and how to show them.

Each visible line is one of three modes: 16 bits per pixel RGB565 (1,440 bytes a line), 8 bits per pixel through a 256-entry palette (720 bytes), or blank, which costs no memory bandwidth at all. An 8-bit line can also be doubled: each byte becomes two pixels inside a window, black outside it. The scroll field starts the line part-way into its first block.

Nothing says the lines have to come from one frame buffer, in order. Point two entries at the same data, and a line repeats; point them anywhere, and the screen splits, or parts of it scroll separately. The CPU can rewrite an entry at any time, and the change shows two lines later; the vertical-blank interrupt, at the start of line 480, is the natural moment to rewrite the whole table for the next frame. The hardware cost is one 32-bit read per line, and it replaces a stack of features a video chip would otherwise need: page flipping, scaling, split screens, scrolling.

Fetching a line

At the start of each line’s horizontal blanking, the engine reads the entry for the line after next and asks the PSRAM controller to DMA its pixels into one of two line buffers in the transfer RAM, an 8 KB dual-clock block RAM written at the controller’s 72 MHz and read at the pixel clock. While one buffer is on screen, the other fills. A 16-bit line is six 256-byte blocks (seven if the scroll offset makes it straddle one more), an 8-bit line three. The controller fits those bursts between the CPU’s cache misses (part 3), and the line is ready long before it’s needed.

320 × 200 at 4:3

All five games draw a 320 × 200 picture, a byte per pixel, the way a DOS PC’s VGA did. On a 4:3 monitor that mode had pixels 1.2 times taller than wide, and the games’ art was drawn for it. The line table recreates that for free:

The games triple-buffer the same way: each draws into one of three buffers, and the vertical-blank interrupt points the table at the finished one, so the picture never tears. The palette lives next to the table, so palette effects, such as Doom’s red damage flash, are free too.

The text mode that went away

The engine once had a fourth mode: 90-column text, with a font in video RAM. Drawing text in software into an 8-bit line is fast enough for a boot screen, and it can use a proportional font. Removing the text mode freed about 250 cells and a block RAM, at a point where both were scarce.

Sound in the blanking

The board has no audio jack, and adding a DAC would break the only-this-chip rule. But HDMI carries sound, and every display with speakers plays it.

HDMI is DVI with extras. In the blanking periods, where DVI sends nothing, HDMI can send packets, called data islands, in a different encoding (TERC4: 4 bits per 10-bit symbol), fenced by guard bands so the receiver can tell them from pixels. Three packet types matter for sound:

• Audio samples, here 48 kHz, 16-bit stereo. At 48 kHz there are about one and a half stereo samples per line, with plenty of blanking to carry them.
• Audio clock regeneration, which tells the receiver how the sample rate relates to the pixel clock: for 27 MHz and 48 kHz, the standard pair N = 6144, CTS = 27000.
• The audio infoframe, which describes the format.

Sameer Puri’s core does all of this as well; the packet side is about 215 of its 630 cells.

Feeding it

The samples come from PSRAM, through the transfer RAM, like the pixels. There are two 1 KB halves, each 256 stereo samples, 5.33 ms of sound. The engine plays half A, then B, then A again, and each time it moves on it raises an interrupt: the half it just finished is free. The CPU writes the next 256 samples into PSRAM, cleans them out of its data cache so the PSRAM really has them, and writes that half’s pointer register, which DMAs them across. The delay from a game to the speaker is 5 to 11 ms.

The audio interrupt is also where the games mix their sound, and it taught the project’s sharpest lesson about this memory system. Doom’s first mixer produced all 48,000 output samples per second for every channel and cost 8 to 9% of the CPU. A version that summed into a 2 KB buffer first cost 12% of the whole game, not because of the arithmetic, but because that buffer, touched from an interrupt, kept evicting the game’s own data from the cache. Mixing at 12 kHz, close to the effects’ own 11 kHz, and holding each result for four output samples brought it down to between 1 and 4%. The later ports mix at 12 or 24 kHz the same way. Music is a different story, for part 6.

The first sound the board makes, every time it starts, is a two-note chime, computed 256 samples at a time.

The bill