# DOOM on a Superscalar RISC-V Core This blog is the complete record of a project that runs DOOM on a superscalar RISC-V core. The core and its peripheral bridge are plain synthesizable Verilog targeting an FPGA board with memory, a display, a keyboard and a serial console, and at that clock rate the game runs in real time. This project works with the same RTL and the same guest binary in a Verilator prototype of that board, where each board component is replaced by a testbench stand-in presenting the same interface, so every capture in this document is cycle accurate behaviour of the design itself. All capture runs are deterministic given the ELF, the WAD and the key schedule, so every GIF in this document is reproducible by re-running the listed simulation. ## The game DOOM is id Software's 1993 first person shooter. Its source was released in 1997 and has been ported to essentially every platform that can draw pixels. This project uses the doomgeneric port of the classic Linux Doom 1.10 lineage source tree: the game is unmodified except for a thin platform file that supplies six hooks (frame display, key read, tick counter, sleep, init, window title). The content is FreeDOOM Phase 1, a free, BSD licensed replacement IWAD, level E1M1. Understanding the captures later in this blog needs five facts about the engine. One, the game runs at a fixed 35 Hz tick called a tic. Every tic the main loop samples the keyboard into a ticcmd structure: a forward move, a side move, an angle turn and a button word. Nothing moves between tics. Two, world simulation walks linked lists of thinkers: player movement and weapon state in p_user and p_pspr, enemies in p_enemy, moving geometry such as doors and platforms in p_spec. A door is a linedef with a special type; type 1 is a manual door that opens when the player uses it, stays open a few seconds, then closes, implemented as a thinker that raises the door sector's ceiling a little per tic. Three, the renderer is BSP based. It walks the map's binary space partition tree front to back, clips wall segments against a per column visibility table, and draws walls, flats and sprites as vertical spans into a 320x200 byte per pixel framebuffer. A fixed 256 entry palette (the PLAYPAL lump) maps bytes to colours. That framebuffer and that palette are exactly what the testbench captures and what the GIFs are assembled from. Four, weapons are animated sprite pairs called psprites, one for the weapon and one for the muzzle flash, driven by a state machine with lower, raise, ready and fire states. Weapon selection writes pendingweapon and the state machine performs the visible lower and raise. Five, and this one cost real debugging time, the use action is not sector based in this code base. P_UseLines casts a 64 unit trace from the player along the facing direction (USERANGE in p_local.h) and offers the first special line it hits to P_UseSpecialLine via PTR_UseTraverse in p_map.c. To open a door you must be within 64 units and facing it. ## The core The core under test is the most optimised variant of a small superscalar RV32I design family; this blog calls it the superscalar core. Everything in this section was read from its RTL (about 2,300 lines) during this project, and every size and policy below is a parameter or a wire in that code, not a marketing claim. ### Pipeline and issue The core is an in-order, two-way superscalar, five stage pipeline: IF, ID, EX, MEM, WB. Each cycle it fetches an aligned pair of instructions, instr0 at PC and instr1 at PC+4, and can issue both into slot 0 and slot 1. Slot 1 is squashed to a replay when the pair cannot co-execute, so the core degrades gracefully to single issue. The register file has 32 entries of 32 bits with two write ports and four read ports, enough for two instructions reading two sources each and two instructions retiring per cycle. ### ISA and ALU The ISA is RV32IM in hardware. The ALU implements the full M extension combinationally: MUL via explicit 64-bit product wires, and MULH, MULHSU, MULHU via separately named signed and unsigned 64-bit products, plus DIV, DIVU, REM, REMU with the spec corner cases written out (division by zero returns all ones for DIV and the dividend for REM; INT_MIN divided by -1 returns INT_MIN for DIV and 0 for REM) instead of relying on simulator defined behaviour. The RTL comments record two real bugs found here during development: an inline 32 bit product shifted right by 32 that silently returned 0 for MULHU and MULHSU, and a Verilator width inference problem that made the fix miscompile until the products were named as explicit wires. FENCE is treated as a no-op and ECALL produces the halt and trap path. The shape of the fix, straight from alu.v: ```verilog // Named, explicitly widened products. The inline form // ({1'b0, a} * {1'b0, b}) >> 32 // miscompiled on this toolchain; naming the wires removes // every width inference from the expression. wire signed [63:0] a_ss = {{32{a[31]}}, a}; wire signed [63:0] b_su = {32'b0, b}; wire signed [63:0] prod_su = a_ss * b_su; wire [63:0] prod_uu = {32'b0, a} * {32'b0, b}; wire div_by_zero = (b == `ZERO_WORD); wire div_ovf = (a == {1'b1, {`DATA_WIDTH-1{1'b0}}}) && (b == {`DATA_WIDTH{1'b1}}); ``` ### Branch prediction Prediction runs on the slot 0 fetch stream. It is a McFarling style hybrid predictor with three 256 entry tables of 2-bit saturating counters: a local PHT indexed by pc[9:2] xor pc[15:8], a global PHT indexed by the same value xor an 8 bit global history register, and a chooser PHT that selects between them per branch. Taken branches get their target from a 256 entry, direct mapped BTB with a full 32 bit tag and a 32 bit target per entry; on a BTB miss the target falls back to PC plus the decoded immediate. JAL is decoded in IF and predicted unconditionally with its computed target. JALR is predicted through the BTB, except returns (jalr with rs1 of x1 or x5 and zero immediate) which are predicted through a 16 entry return address stack with a saturating count. Control redirects that survive to EX flush ID and EX, a two cycle penalty; slot 0 branches and JALRs are additionally resolved early, in ID, so an ID stage redirect also squashes slot 1 of the same pair. ### Hazards, forwarding and the memory pair The hazard unit implements what the design family calls hzopt: a load-use stall is only taken when the consumer needs its operand in the ID stage, which in this core means a slot 0 branch or JALR reading a value that a load in EX has not yet produced; every other consumer is covered by forwarding. The forwarding network supplies each slot from slot 0 and slot 1 EX/MEM results, from MEM/WB, and, for slot 1 only, from slot 0's same cycle EX result, which is what makes dependent pairs like add-then-use issue back to back. Both slots can perform memory operations, so the core exposes two complete load/store ports. The inter-slot memory rule (luopt) squashes slot 1 only on a real conflict: a slot 0 store followed in the pair by a slot 1 load of the same width whose low address bits alias (word compares imm[1:0], half compares imm[1], byte always aliases), a slot 0 load whose rd is read by slot 1, or a slot 0 control transfer. Everything else dual-issues. ### CSRs A 16 entry CSR file implements the machine mode registers mstatus, mie, mtvec, mscratch, mepc, mcause, mtval and mip plus the ID registers, with CSRRW, CSRRS and CSRRC semantics. mtvec resets to 0x100. ### Memories and the MMIO map The instruction and data memories are plain synchronous arrays with no caches; every access is a one cycle array read, which is what makes the core's CPI numbers a property of the core rather than of a cache. Sizes are parameters: the stock testbenches build with 4,096 instruction words and 2,048 data words, and the DOOM build overrides to 524,288 instruction words (2 MiB of code) and 16,777,216 data words. The data memory's index slice is clog2(W) bits wide, so the reachable window is 2 times W bytes, 64 MiB with the DOOM override; addresses above that wrap silently, a property that produced the single most expensive bug of the project and gets its own section below. The DOOM build adds a peripheral bridge, mem_top_dg.v, on the same two memory ports. Its map, identical on both ports: - read 0x004 and 0x008: cycle counter, low and high 32 bits - read 0x00C and 0x010: retired instruction counter, low and high - read 0x014: keyboard state, key byte plus press/release and valid bits - read 0x018: frame counter - read 0x01C: framebuffer base register - read 0x020: exit request status - write 0x000: exit request with exit code - write 0x014: keyboard acknowledge - write 0x01C: framebuffer base - write 0x024: UART byte - write 0x028: frame commit, which dumps the framebuffer on the testbench ### Measured performance The core retires two instructions per cycle whenever the fetch stream cooperates, and the predictors keep it fed on real code. Measured in this simulation, with instruction counts taken from the core's own retired instruction counter: | workload | cycles | instructions | CPI | IPC | |---|---:|---:|---:|---:| | CoreMark, O2, 1 iteration | 235,625 | 322,978 | 0.7295 | 1.3707 | | CoreMark, O2, 10 iterations | 2,267,464 | 3,102,275 | 0.7309 | 1.3682 | | CoreMark, O3, 100 iterations | 21,484,387 | 29,546,308 | 0.7271 | 1.3753 | | binary_search | 20 | 23 | 0.8696 | 1.1500 | | fibonacci | 426 | 464 | 0.9181 | 1.0892 | | hello | 239 | 216 | 1.1065 | 0.9038 | | DOOM E1M1 gameplay window | 362,341,398 | 498,514,179 | 0.7268 | 1.3758 | The DOOM row is the one that matters for this blog: a 500 million instruction boot of a real game runs at CPI 0.727, the same number CoreMark gets, which says the hybrid predictor, the BTB, the RAS and the dual issue memory paths are holding up on branch heavy, pointer chasing code, not just on a benchmark. The core also passes the project's 31/31 instruction regression, and its CoreMark cycle counts were the same after every integration change, confirming the bridge work never perturbed the core. ## The board, and what stands in for it here The design's natural home is an FPGA board. Published RISC-V DOOM ports of this class run five stage cores at 100 to 125 MHz on mid-range parts, and at the cycle cost per tic measured here, 100 MHz delivers about 60 tics per second of silicon time against the 35 the game needs. This project does not place and route the design; it prototypes the board in Verilator, and every component the board would carry gets a testbench stand-in that presents the same interface to the core, so the guest cannot tell which world it is in. The mapping: - crystal oscillator: `forever #5 clk = ~clk;` in tb_doom.v - reset button: an initial block holding rst for 20 time units - program flash or BRAM: the inst_mem.v array, $readmemh from hex - data DDR or BRAM: the data_mem.v array, 64 MiB window in the DOOM build - keyboard controller, PS/2 or USB on a board: the schedule driven injector latching presses and releases into the KEY register - VGA or DVI display controller scanning the framebuffer: the framebuffer array in the bridge plus the commit handler writing one PGM per frame - UART to a serial console: a 1 MiB accumulation buffer flushed at exit - microsecond timer: the cycle counter registers ```verilog // tb_doom.v: the oscillator and reset the board would provide clk = 0; forever #5 clk = ~clk; initial begin rst = 1; #20; rst = 0; end ``` ```verilog // mem_top_dg.v: the keyboard stand-in. The guest sees a real key // register: byte plus press/release and valid bits, cleared by ack. always @(posedge clk) begin if (inject_valid) begin key_reg <= inject_key[7:0]; key_rel <= inject_key[8]; key_valid <= 1'b1; end end ``` ```verilog // mem_top_dg.v: the display stand-in. One commit store, one dump. 12'h028: begin $sformat(fname, "frame_%0d.pgm", frame_cnt); fd = $fopen(fname, "wb"); $fwrite(fd, "P5\n320 200\n255\n"); for (i = 0; i < FB_WORDS; i = i + 1) $fwrite(fd, "%c%c%c%c", fb[i][7:0], fb[i][15:8], fb[i][23:16], fb[i][31:24]); $fclose(fd); end ``` ### Inputs and interrupts Which interrupts does the game rely on? None. The CSR file implements mie, mip, mtvec and the rest of the machine mode trap registers, but the port never enables them and the testbench never asserts an IRQ line. Everything DOOM needs is polled: the keyboard through DG_GetKey reading the KEY register, time through DG_GetTicksMs reading the cycle counter, and the display is push, one commit store per frame. The only trap path event in the system is ECALL, which the port uses as a clean exit at the end of a run. In practice the guest relies on exactly three inputs, the key register, the cycle counter and reset, and on a board those three come from the keyboard controller, the timer and the reset button instead of the testbench. ```c /* dg_platform.c: the whole input path. Poll, decode, acknowledge. */ int DG_GetKey(int *pressed, unsigned char *key) { uint32_t k = MM_KEY; if (!(k & 0x80000000u)) return 0; /* nothing new */ *pressed = (k & 0x40000000u) ? 0 : 1; /* bit30 = release */ *key = (unsigned char)(k & 0xFFu); MM_KEY = 0u; /* ack clears key_valid */ return 1; } uint32_t DG_GetTicksMs(void) { return MM_CYC_LO / 1000u; } ``` The polling model is also why the singletics flag matters: with the clock derived from the cycle counter, one doomgeneric tick advances exactly one tic, which keeps the frame stream, the tic stream and the key schedule in a fixed relationship whether the counter is fed by a board oscillator or by the prototype. ## Adding a screen, a keyboard and a clock The stock core boots an ELF from hex files and prints cycle counts. It has no display, no input and no wall clock. DOOM needs all three. This part is the order in which they were added, and each step ends at a capture. ### Step 1: the framebuffer The port defines CMAP256 so the game renders one byte per pixel, and DOOMGENERIC_RESX/RESY of 320 and 200 so the game's native Mode 13h layout is used directly. DG_DrawFrame copies the 64,000 byte screen into the framebuffer window and then stores to the frame commit register, 0x028. The bridge watches that store on either issue port and dumps the framebuffer to a P5 PGM file stamped with the exact cycle and retired instruction counts. An offline tool pairs the PGM bytes with PLAYPAL from the WAD and produces PNG stills and GIFs. The whole guest side of the display path is two lines: ```c /* dg_platform.c: DG_Init points DG_ScreenBuffer at FB_ADDR so the * renderer draws straight into the bridge's framebuffer page. */ void DG_DrawFrame(void) { MM_DUMP = 1u; /* one MMIO write, one frame_.pgm on the host */ } ``` On a board the same commit store would instead kick a display controller that scans the framebuffer page out over VGA or DVI; the guest code is identical either way. The first still, the E1M1 3D view with status bar, was the proof that the render path worked end to end. ### Step 2: the WAD in memory, and the 64 MiB bug The IWAD is linked into the data memory as a blob, with the heap placed below it. The first boot died with W_GetNumForName: PNAMES not found even though the bytes were present. The cause was the data memory index slice, visible in data_mem.v: ```verilog // data_mem.v: the index slice is clog2(W) bits wide, so the // reachable window is exactly 2*W bytes and nothing more. wire [$clog2(`DATA_MEM_WORDS)-1:0] idx0 = offset0[$clog2(`DATA_MEM_WORDS)+1:2]; ``` With a 40 MB heap the WAD's lump directory landed at 0x04490B34, past the 64 MiB window, and silently wrapped to an alias that read zeroes. The fix was a heap size in the linker script that keeps the WAD end inside the window, plus a build time check in doom/build.sh that fails the build if _wad_end ever crosses 0x04000000 again. ### Step 3: time DOOM asks the platform for milliseconds. The port answers from the cycle counter MMIO (0x004/0x008), dividing cycles by the measured cycles per millisecond, and runs in singletics mode so exactly one tic elapses per rendered frame. That keeps the frame stream, the tic stream and the key schedule in a fixed relationship: one captured frame is one tic. ### Step 4: the keyboard, and the dual port bug The testbench injects keys from a schedule file of cycle and keycode pairs (+keyfile), pressing and releasing through a 9 bit inject path where bit 8 means release; the guest acknowledges each key by writing 0x014. The first version of the bridge decoded every MMIO write except the UART on port 0 only. The core dual-issues stores, so an acknowledge landing in slot 1 was silently dropped, key_valid never cleared, and the guest re-read the same key on every poll. Movement half worked by luck of pairing; fire never registered at all. The fix decodes both ports in the MMIO task, servicing the older instruction first so paired UART stores also keep their order: ```verilog // mem_top_dg.v, after the fix: an MMIO store may arrive on either // issue port, because the core dual-issues stores. always @(posedge clk) begin if (dg_enable && mem_we0 && mmio0) mmio_write(mem_addr0[11:0], mem_wdata0); if (dg_enable && mem_we1 && mmio1) mmio_write(mem_addr1[11:0], mem_wdata1); end ``` On a board the same rule applies: a peripheral bridge must accept stores from both issue ports or the second one vanishes. Every action GIF after this point depends on that fix. ### Step 5: seeing what the game sees Route debugging later needed the player's own state, so the port prints a per-tic trace over the UART: leveltime, ready weapon, ammo, x, y, angle and use button. On a board this is ordinary printf to the serial console; here the UART is accumulated in a 1 MiB buffer in the bridge and flushed at simulation exit. The hook, in doom/port/d_main.c: ```c printf("[tk%05d] rw=%d ammo=%d x=%d y=%d ang=%d u=%d\n", leveltime, (int)pl->readyweapon, (int)pl->ammo[am_clip], pl->mo ? (int)(pl->mo->x >> FRACBITS) : 0, pl->mo ? (int)(pl->mo->y >> FRACBITS) : 0, pl->mo ? (int)(pl->mo->angle >> 24) : 0, pl->usedown); ``` This trace is what turned the door work from guesswork into measurement. ## One action at a time With the platform in place, each game action was exercised, captured and checked frame by frame. The captures below come from two long runs, a movement run and a combat run, plus a door run; the slicing tool cuts each action out of its run using the cycle stamps. ### 01 Raising the pistol ![Pistol raise](gifs/01_pistol_raise.gif) The level opens by lowering then raising the starting pistol through the psprite state machine. No input is needed, so this clip doubled as the frame path smoke test. The bug it exposed was in the recording plan: the first capture schedule started walking at tic zero, so the raise overlapped forward movement and could not be shown alone. The fix was an idle opening, a couple of seconds with no keys pressed, at the start of the combat run, with the raise cut from there. The first slicer version also crashed indexing an empty frame list when computing this clip's bounds; deriving the bounds from the following movement segment fixed it. ### 02 Walking forward ![Walk forward](gifs/02_walk_forward.gif) The up arrow held for about a second of game time. This is the clip that made the dual port acknowledge bug visible: before the fix the same schedule produced a garbled key stream and the player moved by accident. After the fix, pressed means pressed and released means released, and the corridor scrolls as it should. ### 03 Running forward ![Run forward](gifs/03_run_forward.gif) Running is walking with shift held, so two keys must be down together. The schedule generator originally serialised every hold, which turned run into walk, pause, walk. It was reworked so several hold pairs inside one action string start together, while a serialiser keeps the actual MMIO injections 20,000 cycles apart because the bridge latches one pending key. The clearly faster motion in the clip is the run segment following the walk segment in the same capture. ### 04 Walking backward ![Walk backward](gifs/04_walk_backward.gif) The down arrow, completing the movement set and confirming the backward ticcmd path in p_user. Cut from the same run as 02 and 03, a few seconds later in the cycle log. ### 05 and 06 Turning ![Turn right](gifs/05_turn_right.gif) ![Turn left](gifs/06_turn_left.gif) Turning exposed the calibration bug. The first schedules assumed 35,714 cycles per tic, the value you get from 35 ms at 1.25 MHz. The measured core spends about 1.62 million cycles per tic, so every press landed tens of tics off and the turns overlapped the walk segments. Regenerating schedules from the measured cycle cost fixed the alignment, and the captures show the expected roughly three degrees per tic of rotation, the number later used to aim the player at a door. ### 07 and 08 Strafing ![Strafe right](gifs/07_strafe_right.gif) ![Strafe left](gifs/08_strafe_left.gif) Side move without angle turn: the view translates while the facing stays fixed, which is also evidence that the sidemove field of the ticcmd is wired correctly, a separate code path from forward movement. By this point the input bridge, the calibration and the frame capture were all behaving, so these clips came out on the first cut. ### 09 Firing the pistol ![Fire pistol](gifs/09_fire_pistol.gif) Fire was the action the dual port bug killed outright: before the fix no press registered and ammo never changed. After the fix the clip shows the flash sprite, the recoil pose and the ammo counter stepping from 50 to 49 at the expected tic, matching the trace. Before and after, the same schedule: no shot, then a shot. ### 10, 11 and 12 Switching weapons and punching ![Switch to fists](gifs/10_switch_to_fists.gif) ![Punch](gifs/11_punch.gif) ![Switch back to pistol](gifs/12_switch_to_pistol.gif) The number keys set pendingweapon; the current weapon lowers, the fists raise, the ready state returns. The first clip shows the pistol disappearing, the second the fist lunge, the third the pistol returning. The fix here was again in the slicer: cutting at the number key's release trimmed the clips to a fraction of a second, while the lower and raise animations run about thirty tics; cutting from one press to the next produced complete animations. ### 13 Opening a door ![Open door](gifs/13_open_door.gif) The longest fight in the project. The goal: walk up to a closed door, press use, watch it slide open. The spawn room has no door; a 360 degree scout of stills showed only windows, crates and an open doorway. Parsing the WAD lumps (THINGS, LINEDEFS, SIDEDEFS, VERTEXES) produced a top down map and a list of type 1 doors, and picked one north of the eastern corridor the player reaches by walking straight out of the spawn area. Route planning then went through wrong models of use. The first, borrowed from older Doom documentation, was that use fires on any special line of the player's sector; several runs ended with the player standing in the door recess pressing use at a door that never moved, because in this code base use is the 64 unit hitscan described in the game section, and the player had been facing away from the door. The second problem was reaching the recess at all: the dirt area in front of it holds live zombie troopers whose hits add damage thrust, so small schedule changes moved the player a lot. The per-tic trace described in the bring-up notes turned that into measurement. The final schedule walks east, takes the diagonal lane, fires a burst at the trooper blocking the lane, pushes into the shallow recess, turns to face the door line, and taps use. With the player ten units from the door and facing it, the trace shows usedown set and the next frames show the door rising and the room beyond, including one of its inhabitants. About half a dozen full runs ended at walls, at the wrong sector, or facing the wrong way; each failure corrected the model the route was built on. ## Board time and prototype time Nothing in the guest or in the RTL is simulation specific. Fitted to an FPGA board, the same ELF drives the real peripherals, and at 100 MHz the measured cycle cost per tic works out to about 60 tics per second of silicon time, above the 35 the game needs; prior public RISC-V DOOM boards run this class of core at 100 to 125 MHz, so the design sits comfortably inside what a mid-range part provides. The simulation in this project is a prototype of that board, and a prototype is slower than the thing it prototypes. One tic costs about 1.62 million cycles on this core, so one second of game time is about 57 million cycles. The Verilator build sustained about 1.7 million cycles per second of wall time on this host during the capture runs, so one second of gameplay costs about thirty seconds of wall time: the prototype runs the game at roughly a thirtieth of board time, about one tic per second. Live play against the prototype is therefore slow motion, and the practical deliverable is offline recordings driven by precomputed key schedules, which is exactly what the rest of this blog shows. The gap between prototype time and board time is a property of the verifier, not of the design; the core is not the bottleneck anywhere in this project. ## Longer takes ![First gameplay recording](../doom/e1m1_gameplay.gif) The first continuous recording made for this project: 200 tics, about six seconds of game time, walking, turning and firing out of the spawn area. ![Extended single take](../doom/e1m1_extended_take.gif) The action clips were sliced out of runs scripted to exercise one move at a time. The capture above is a single uninterrupted run instead: 934 rendered frames, about 26 seconds of game time at 35 fps, produced by exactly the schedule the door section describes. It walks out of the spawn room, runs the eastern corridor, turns into the dirt area, drops the trooper blocking the lane, pushes into the door recess, faces the door, opens it and walks through into the imp on the other side, so the earlier clips appear in context: weapon raise, walk, run, turns, gunfire and the door sequence all flowing out of one boot and one input file. The take is also the cleanest picture of how input works here. There is no live keyboard; the whole run is driven by a text schedule of cycle and keycode pairs that the testbench injects through the keyboard MMIO, at cycle numbers the schedule compiler computes from the measured cycle cost of each route segment. The guest renders a frame, the testbench dumps it to a PGM stamped with the cycle count, and the offline tool places every frame at its true time, so the playback rate is measured, not assumed. The schedule is a plain file you can read and diff, and the simulation is deterministic given the ELF, the WAD and the schedule, so re-running the capture reproduces the clip. It stops where it stops because of wall time: 26 seconds of game time is about twelve minutes of simulation. A clear run of E1M1 remains its own project and is not recorded. ## Appendix: What was changed - rtl/mem/mem_top_dg.v: the dual port peripheral bridge: MMIO decode on both issue ports with in-order servicing, 1 MiB UART accumulation buffer, 9 bit key injection, framebuffer dump on commit. - tb/tb_doom.v: keyfile schedule parser, frame capture, console flush, exit and timeout handling. - doom/port/d_main.c and companions: the doomgeneric platform hooks, WAD blob handling, per-tic trace print. - doom/doom.ld and doom/build.sh: heap sizing against the 64 MiB data memory window, with a build time check. - doom/gen_keys.py: the schedule compiler (hold, tap, gap, absolute reposition, overlapping holds). - doom/frames2video.py: PGM plus palette to PNG and GIF assembly with cycle accurate delays and range slicing. The core's RTL was not modified for this project. Every fix lived in the integration layer, and the core's own regression results and CoreMark cycle counts were unchanged after each integration change.