m9b · part 2 of 6

The CPU: VexiiRiscv, cut to fit

The CPU takes more than half of the chip: about 4,300 of its 8,640 logic cells and 18 of its 26 block RAMs. This part is about how it got there, and about what each of those cells and blocks buys.

RISC-V: The obvious choice

There’s quite a few open cores available out there, but for a project like this I wanted something that would work mostly out of the box: a well established toolchain and flexible but efficient verilog implementation were a must.

Considering the limited number of LUTs on this FPGA, RISC-V was the only serious contender here, since it provides a great developer experience for C or C++ program as well as a very extensible instruction set. The base set, RV32I, is about forty instructions; the M extension adds multiply and divide, which games use a lot, and then many others on top of that.

Any RISC-V core needs 32 registers of 32 bits. That’s 1,024 bits, a sixth of the chip’s 6,480 flip-flops if built from them, so in every core that fits here the register file is two block RAMs. Beyond that, the decisive factor is memory. Programs and data live in PSRAM, which takes dozens of CPU cycles to start a transfer. A core without caches would wait that long for every instruction.

From FemtoRV to VexiiRiscv

The first version of the design ran FemtoRV32, from Bruno Levy’s learn-fpga, which is small and easy to read. The PetitBateau variant, with its floating-point unit, didn’t make 27 MHz: its slowest path compared two registers, decided a branch and computed the next address in one cycle, and on this chip that took longer than 37 ns. The Electron variant (RV32IM, no floating point) did make it, and it ran the first PSRAM tests. But FemtoRV has no caches.

Next came the default VexRiscv from LiteX: a five-stage pipeline with 4 KB instruction and data caches. This was quite a bit faster. Soon after came its successor, Charles Papon’s VexiiRiscv, for one reason above all: its data cache is write-back. VexRiscv’s writes every store through to memory; VexiiRiscv’s keeps stores in the cache until the line is evicted, so the PSRAM only ever sees whole 64-byte lines. That suits a memory that is slow to start a transfer and fast once it’s going.

VexiiRiscv isn’t a file you download, it’s a generator: you describe the core you want on the command line, and it writes out Verilog. Every option trades logic cells, block RAM and clock speed against performance, and the rest of this part is about those trades. This is the command line for the core in m9b, generated from a fork with a few options of its own:

--xlen 32 --physical-width 24 --with-rvm --with-rvZcbm --with-rvZba
--lsu-software-prefetch --with-rvZicboz --div-radix 4
--with-btb --btb-hash-width 11 --btb-target-width 21 --with-ras
--relaxed-branch --relaxed-shift --regfile-async --allow-bypass-from 0
--fetch-l1 --fetch-wishbone --fetch-l1-sets 128 --fetch-l1-ways 1
--fetch-l1-mem-data-width-min 64
--lsu-l1 --lsu-l1-wishbone --lsu-wishbone --lsu-l1-sets 128 --lsu-l1-ways 2
--lsu-l1-mem-data-width-min 64 --lsu-l1-word-writes --lsu-l1-wishbone-unbuffered
--reset-vector 0
--region base=0,size=20000,main=1,exe=1          # user flash: cached
--region base=800000,size=800000,main=1,exe=1    # PSRAM: cached
--region base=400000,size=400000,main=0,exe=0    # I/O: uncached

The caches

Both caches use 64-byte lines and talk to memory over 64-bit buses, a whole line per request. Long lines suit the PSRAM, whose fixed setup cost per transfer is shared across more bytes.

• Fetch L1: 8 KB, direct-mapped (128 sets of one line). 4 block RAMs for the data, 1 for the tags. Code mostly runs in loops, and going from 4 to 8 KB was worth about 2% on Doom.
• Data L1: 16 KB, 2-way set-associative, write-back (128 sets of two lines). 8 block RAMs for the data, 2 for the tags.

The data L1 was direct-mapped for most of the project: 256 sets of one line, every address with exactly one place to go. Its weakness is conflicts. Doom’s column drawer, for example, walks down a column of the screen, touching 200 different lines; whatever else is hot (the textures, the colour maps) lands on some of the same sets, and they evict each other. Moving a buffer by a few hundred bytes changed the timedemo by 2%. A second way removes most of that: on the Doom timedemo it was worth 14%.

This took a long time to land because of the block RAMs’ byte enables.

The byte-enable bug

A cache that takes byte stores needs byte enables on its memory: a write mask that says which bytes of a word to change. Gowin’s block RAMs have them in practice, although they disappeared from the documentation a long time ago. And now I also know why: they don’t work reliably.

The first symptom was data corrupted by byte stores while the memory tests, which wrote whole words, kept passing. The simulation model was fine, timing was fine, the netlist was fine. A test design that exercises all 26 block RAMs eventually found the rule: when a port writes in two consecutive cycles, a byte lane that was enabled in the first write and disabled in the second gets written anyway, in 10 to 36% of cases, on every block, at 27, 36 and 72 MHz. An idle cycle or a read in between avoids it, and full-word writes never fail.

The first fix was to avoid byte enables altogether: one block RAM per byte lane, eight 2 KB lanes, which makes exactly 16 KB of direct-mapped cache. A 2-way cache of that shape would have needed 16 blocks.

The fix that made 2 ways possible is an option added in the fork, --lsu-l1-word-writes: the data L1 never writes partial words. A byte or half-word store reads its whole 32-bit word (through the pipeline’s bypass, so stores still in flight are included), merges its bytes in, and writes the word back. Each way’s bank then needs a write mask per 32-bit word, not per byte, and Gowin builds it from plain write enables: two blocks per way and word half, 8 blocks for 16 KB, the same as before. The boot firmware still checks byte, half-word and word stores in every lane, every time the board starts.

What each option is worth

Every change was measured on the board, mostly with the Doom timedemo (a fixed demo, drawn one frame per game tic, so runs are comparable to the frame: they checksum every frame), and kept only if it paid for its cells.

A few of these taught something beyond their number.

The radix-4 divider looks like a luxury on a chip this full, but System Shock’s texture mapper divides once per span, about 150,000 times in five seconds of play. Halving the divide time made the whole game 8% faster.

Prefetching helps loops that know where they are going: a read loop that prefetches the next line nearly triples its speed. Doom’s column drawer, which could in principle prefetch, gained at best 0.5%: most of its misses are the ones nobody can predict.

Smaller and faster: 24-bit addresses

The memory map needs 16 MB: flash at 0, I/O at 4 MB, PSRAM at 8 MB. A 32-bit CPU without an MMU still carries 32-bit addresses everywhere: in every copy of the program counter down the pipeline, in the branch and PC adders, in the load/store unit’s address stages, and in the cache tags.

VexiiRiscv’s --physical-width sets that width. At 24 bits, the tags shrink from 19 to 11 bits, every PC copy loses a byte, and anything at or above 16 MB traps. It saved 258 logic cells and 190 registers, and the CPU’s maximum clock went from 36.05 to 39.96 MHz: the critical path, from the branch predictor’s read to the PC increment, runs through the PC adder. Software only had to move to the new memory map.

A similar trick shrank the branch target buffer: with an 11-bit hash and 21 target bits, an entry is 36 bits, one block RAM instead of two.

Timing at 88%

The CPU runs at 36 MHz, half the PSRAM controller’s clock. It has to stay synchronous with the controller (part 1), so the next step up would be the full 72 MHz, which is out of reach. Every change has to keep the CPU above 36 MHz on a chip that is 88% full, and where things land matters more and more as it fills.

• --relaxed-branch was needed to reach 27 MHz at all, and --relaxed-shift (the shifter’s result is forwarded a stage later) to reach 36.
• --relaxed-btb would shorten the branch predictor’s path, but didn’t route: it made a congestion hot spot.
• The place-and-route options decide between passing and failing: one routing option gave 35.7 MHz, another 38.75 for the same design. Every build ends with a look at the timing report.

The last build reached 37.8 MHz, with 4% margin.

The bill

Plus about 4,300 logic cells and two of the chip’s hard multiplier blocks. The Doom timedemo now runs at about 22 frames per second; the options above account for a good part of that, and the memory system in the next part for much of the rest.