CoreMark Performance Across Microarchitectural Variants

Build & Measurement Flow

All results are generated using a consistent toolchain and simulation flow.

Compilation

riscv32-unknown-elf-gcc -O3 -ffreestanding -fno-builtin -march=rv32im -mabi=ilp32 -nostdlib -Ttext 0x0 \
  -I scripts/coremark -DITERATIONS=100 -DVALIDATION_RUN=1 -DTOTAL_DATA_SIZE=2000 \
  -o coremark_i100_o3.elf \
  scripts/coremark/crt0.S \
  scripts/coremark/core_main.c \
  scripts/coremark/core_list_join.c \
  scripts/coremark/core_matrix.c \
  scripts/coremark/core_state.c \
  scripts/coremark/core_util.c \
  scripts/coremark/core_portme.c \
  -lgcc

Binary Conversion

python3 scripts/elf2hex.py coremark_i100_o3.elf hex/inst_mem.hex hex/data_mem.hex

Simulation

make verilator-prog
./obj_dir/Vtb_program
cat tb_program_results.txt
cp tb_program_results.txt tb_program_results_i100_o3.txt

Result Parsing & Metrics

import re

def parse(path):
    txt = open(path).read()
    cycles = int(re.search(r"Total cycles:\s*(\d+)", txt).group(1))
    inst = int(re.search(r"Retired instructions:\s*(\d+)", txt).group(1))
    return cycles, inst

runs = [
    ("O2", 1, "tb_program_results_i1_o2.txt"),
    ("O2", 10, "tb_program_results_i10_o2.txt"),
    ("O2", 100, "tb_program_results_i100_o2.txt"),
    ("O3", 1, "tb_program_results_i1_o3.txt"),
    ("O3", 10, "tb_program_results_i10_o3.txt"),
    ("O3", 100, "tb_program_results_i100_o3.txt"),
    ("Ofast", 1, "tb_program_results_i1_ofast.txt"),
    ("Ofast", 10, "tb_program_results_i10_ofast.txt"),
    ("Ofast", 100, "tb_program_results_i100_ofast.txt"),
]

for opt, iters, path in runs:
    cycles, inst = parse(path)
    cpi = cycles / inst
    ipc = inst / cycles
    cm_mhz = (iters * 1_000_000) / cycles
    print(f"{opt} ITER={iters}: cycles={cycles}, inst={inst}, CPI={cpi:.9f}, IPC={ipc:.9f}, CoreMark/MHz={cm_mhz:.9f}")

1. RV32I Single-Cycle (rv32i-sc)

Results

Explanation

• CPI ≈ 1.0 across all runs
• IPC ≈ 1.0 (ideal)
• No hazards, no overlap → every instruction completes in one cycle
• Performance limited entirely by long clock period

2. RV32I Multi-Cycle (rv32i-mc)

Results

Explanation

• CPI ≈ 4.0
• IPC ≈ 0.25
• Each instruction broken into sequential steps (IF/ID/EX/MEM/WB)
• No overlap → very low throughput

3. RV32I Pipelined (rv32i-pipe)

Results

Explanation

• CPI ≈ 1.31

• IPC ≈ 0.76

• Performance loss from:

  • Branch penalties
  • Load-use hazards

• Still significantly better than multi-cycle

4. RV32I Superscalar

Results

Explanation

• CPI ≈ 1.04
• IPC ≈ 0.96
• Dual-issue capability
• Limited by dependency and control hazards

All Variants Implemented

Base Architectures

• rv32i-sc — single-cycle
• rv32i-mc — multi-cycle
• rv32i-pipe — baseline pipeline
• rv32i-superscalar — dual-issue
• rv32i-ooo — out-of-order
• rv32i-super-ooo — optimized OOO

Pipeline Evolution Variants

Branch Handling

• rv32i-pipe-br — ID-stage branch resolution
• rv32i-pipe-sbp — static predictor
• rv32i-pipe-gbp — global predictor
• rv32i-pipe-gbp-tuned — tuned global predictor
• rv32i-pipe-hybp — hybrid predictor

Direction + Target

• rv32i-pipe-br-2dbp — 2-bit predictor
• rv32i-pipe-br-2dbp-btb — + BTB

Hazard Optimisations

• rv32i-pipe-br-2dbp-btb-hzopt
• rv32i-pipe-br-2dbp-btb-hzopt-luopt
• rv32i-pipe-br-2dbp-btb-hzopt-luopt-fulu

Advanced Prediction

• rv32i-pipe-br-hybp-btb-hzopt-luopt-fulu
• rv32i-pipe-br-hybp-btb-ras-hzopt-luopt-fulu

Experimental

• rv32i-pipe-1dbp
• rv32i-pipe-2dbp

RV32I Pipelined Optimization Journey

Scope and benchmark method

• This log tracks the single issue pipeline path from rv32i-pipe to the latest optimized variants.
• CoreMark was run at -O2, -O3, and -Ofast, with ITERATIONS=1/10/100.
• For step by step comparison, -O3, ITERATIONS=100 is the main reference point.
• Metrics used:
  • CoreMark/MHz = (ITERATIONS * 1,000,000) / cycles
  • CPI = cycles / retired_instructions
  • IPC = retired_instructions / cycles
• CRC checks were consistent in all successful runs:
  • ITER=1: 0x0000e3c1
  • ITER=10: 0x0000c64e
  • ITER=100: 0x0000844d

Starting point: plain pipelined core

Variant: rv32i-pipe

• O2 i100: cycles 41,275,362, CPI 1.335811572, CoreMark/MHz 2.422752828
• O3 i100: cycles 38,877,313, CPI 1.315809553, CoreMark/MHz 2.572194225
• Ofast i100: cycles 38,877,308, CPI 1.315809473, CoreMark/MHz 2.572194556

This is the baseline. The initial CPI point you mentioned is correct, around 1.33 at O2 and 1.3158 at O3.

Step 1: early branch resolution

Variant: rv32i-pipe-br

What changed

• Branch decision moved earlier to ID stage.
• Added ID stage operand forwarding for branch compare so branch operands are ready sooner.
• Added redirect from ID when branch decision differs from sequential path.

Why it helped

• Reduces control hazard penalty because branch wait time is shorter.

Result (O3 i100)

• Cycles: 38,877,313 -> 35,812,817
• CPI: 1.315809553 -> 1.212091142
• CoreMark/MHz: 2.572194225 -> 2.792296400
• Improvement vs previous: +8.557% (cycle reduction based)

Branch prediction type sweep (before BTB/hzopt/luopt/fulu)

This sweep compares direction predictors on the simpler pipeline lines.

Best of this sweep was local 2-bit dynamic, so the main optimization line continued with that style.

Step 2: branch resolve + 2-bit dynamic predictor

Variant: rv32i-pipe-br-2dbp

What changed

• Kept ID stage branch resolution.
• Added dynamic 2-bit branch history table in IF.

Result (O3 i100)

• Cycles: 35,812,817 -> 33,466,269
• CPI: 1.212091142 -> 1.132671809
• CoreMark/MHz: 2.792296400 -> 2.988083315
• Improvement vs previous: +6.547% over rv32i-pipe-br

Step 3: BTB

Variant: rv32i-pipe-br-2dbp-btb

What changed

• Added BTB for target prediction in IF.
• Direction and target prediction were both available earlier.

Why it helped

• Correctly predicted taken branches no longer wait for late target compute.

Result (O3 i100)

• Cycles: 33,466,269 -> 32,347,569
• CPI: 1.132671809 -> 1.094809209
• CoreMark/MHz: 2.988083315 -> 3.091422419
• Improvement vs previous: +3.458%

Step 4: hzopt (false load use cleanup)

Variant: rv32i-pipe-br-2dbp-btb-hzopt

What changed

• Hazard logic was made source aware, using decoded use_rs1 and use_rs2.
• Removed false positive load use stalls when source register is not actually consumed by the instruction.

Result (O3 i100)

• Cycles: 32,347,569 -> 32,346,765
• CPI: 1.094809209 -> 1.094781998
• CoreMark/MHz: 3.091422419 -> 3.091499258
• Improvement vs previous: +0.002%

Step 5: luopt (first true load use reduction)

Variant: rv32i-pipe-br-2dbp-btb-hzopt-luopt

What changed

• Relaxed one true load use case for store data (load -> store rs2) when safe.
• Added matching EX path load data forwarding so correctness is preserved.

Result (O3 i100)

• Cycles: 32,346,765 -> 32,342,681
• CPI: 1.094781998 -> 1.094643774
• CoreMark/MHz: 3.091499258 -> 3.091889630
• Improvement vs previous: +0.013%

Step 6: fulu (generalized load use reduction)

Variant: rv32i-pipe-br-2dbp-btb-hzopt-luopt-fulu

What changed

• Generalized MEM to EX load forwarding for both EX operands.
• Stalls kept only for ID stage consumers that truly need value in ID (branch and jalr related reads).
• Removed extra load use stalls for EX stage consumers now covered by forwarding.

Result (O3 i100)

• Cycles: 32,342,681 -> 31,709,661
• CPI: 1.094643774 -> 1.073219100
• CoreMark/MHz: 3.091889630 -> 3.153613027
• Improvement vs previous: +1.996%

Later revisit: predictor path that started from global and was tuned

After fulu, predictor alternatives were revisited on a separate line.

The tuned path improved over its own earlier versions and then moved to a hybrid predictor.

Using that predictor in the full optimized line

Variant: rv32i-pipe-br-hybp-btb-hzopt-luopt-fulu

What changed

• Replaced the local 2-bit direction predictor in the full optimized line with tournament style prediction.
• Kept BTB, hzopt, luopt, and fulu behavior.

Result (O3 i100)

• Cycles: 31,709,661 -> 31,566,209
• CPI: 1.073219100 -> 1.068363941
• CoreMark/MHz: 3.153613027 -> 3.167944557
• Improvement vs previous: +0.455%

Final control-flow add on this line

Variant: rv32i-pipe-br-hybp-btb-ras-hzopt-luopt-fulu

What changed

• Added JAL fast path prediction in IF.
• Added RAS for return style jalr prediction.

Result (O3 i100)

• Cycles: 31,566,209 -> 31,540,922
• CPI: 1.068363941 -> 1.067508098
• CoreMark/MHz: 3.167944557 -> 3.170484363
• Improvement vs previous: +0.080%

Consolidated O3 i100 timeline

Full i100 values for main milestones

Superscalar comparison (fair same-ELF numbers)

From rv32i-superscalar/COREMARK_RESULTS.md fair run:

• Superscalar O3 i100: cycles 30,832,113, CPI 1.043518297, CoreMark/MHz 3.243371611
• Best pipelined line so far (rv32i-pipe-br-hybp-btb-ras-hzopt-luopt-fulu):
  • cycles 31,540,922, CPI 1.067508098, CoreMark/MHz 3.170484363

Remaining gap to superscalar on O3 i100:

• CoreMark/MHz gap: 3.243371611 - 3.170484363 = 0.072887248
• Relative gap: about 2.299%

Final status summary

• The main pipeline line improved from 2.572194225 to 3.170484363 CoreMark/MHz at O3 i100.
• That is about 23.260% better than the original pipelined baseline.
• The largest single gains came from:
  • early branch resolution,
  • better branch prediction direction,
  • BTB target prediction,
  • generalized load use forwarding (fulu).
• Later tuning with tournament style direction selection and RAS gave additional smaller gains.

Reference

Full optimization breakdown and detailed analysis: ../../blogs/optm-riscv-core/

Summary

• Single-cycle: Perfect CPI, limited by clock
• Multi-cycle: High CPI (~4), low throughput
• Pipeline: Balanced (~1.3 CPI)
• Superscalar: Best (~1.04 CPI), but underutilized

The full optimized pipeline approaches superscalar performance with significantly lower complexity.