Emulate Ox64 BL808 in the Web Browser: Experiments with TinyEMU RISC-V Emulator and Apache NuttX RTOS

📝 21 Jan 2024

Ox64 BL808 Emulator with TinyEMU RISC-V Emulator and Apache NuttX RTOS

(Live Demo of Ox64 BL808 Emulator)

(Watch the Demo on YouTube)

In olden times we had Computer Games (plus Operating Systems) on 5.25-inch Floppy Disks. And we’d boot the Floppy Disks (clackety-clack) on Apple II Computers with 64 KB RAM.

Today (40 years later) we boot microSD Cards (clickety-click) on Ox64 BL808 RISC-V Single-Board Computers with 64 MB RAM. (Pic below)

What if we could turn it into a Virtual Ox64 SBC that boots in our Web Browser? (Pic above) Exactly like an Emulated Apple II!

In this article we…

Why NuttX?

Apache NuttX RTOS is a tiny operating system for 64-bit RISC-V Machines. (Also Arm, x64, ESP32, …)

Which makes it easier to understand everything that happens as NuttX boots on our Ox64 Emulator.

Pine64 Ox64 64-bit RISC-V SBC (Bouffalo Lab BL808)

§1 Install TinyEMU Emulator

What’s this TinyEMU?

TinyEMU is a barebones 64-bit RISC-V Emulator.

It doesn’t have all the features of QEMU Emulator. But TinyEMU runs in a Web Browser and it’s much simpler for modding!

We begin by installing (our modded) TinyEMU for the Command Line

## Download TinyEMU modded for Ox64
git clone https://github.com/lupyuen/ox64-tinyemu
cd ox64-tinyemu

## For Ubuntu:
sudo apt install libcurl4-openssl-dev libssl-dev zlib1g-dev libsdl2-dev
make

## For macOS:
brew install openssl sdl2
make \
  CFLAGS="-I$(brew --prefix)/opt/openssl/include -I$(brew --prefix)/opt/sdl2/include" \
  LDFLAGS="-L$(brew --prefix)/opt/openssl/lib -L$(brew --prefix)/opt/sdl2/lib" \
  CONFIG_MACOS=y

(See the Build Script)

What about TinyEMU for the Web Browser?

No Worries! Everything that runs in Command Line TinyEMU… Will also run in Web Browser TinyEMU.

We tweak TinyEMU for Ox64…

BL808 Memory Map (Page 41)

BL808 Memory Map (Page 41)

§2 Change RISC-V Addresses in TinyEMU

TinyEMU needs to emulate our Ox64 BL808 SBC. What shall we tweak?

TinyEMU is hardcoded to run at Fixed RISC-V Addresses. (Yep it’s really barebones)

We tweak the RISC-V Addresses in TinyEMU, so that they match the Bouffalo Lab BL808 SoC (pic above): riscv_machine.c

// RISC-V Addresses for TinyEMU
// (modded for Ox64 BL808)
#define LOW_RAM_SIZE    0x00010000ul  // 64 KB of Boot Code at Address 0x0
#define RAM_BASE_ADDR   0x50200000ul  // Our Kernel boots here
#define PLIC_BASE_ADDR  0xe0000000ul  // Platform-Level Interrupt Controller (PLIC)
#define PLIC_SIZE       0x00400000ul  // Address Range of PLIC
#define CLINT_BASE_ADDR 0x02000000ul  // CLINT is Unused
#define CLINT_SIZE      0x000c0000ul  // CLINT is Unused
...
#define PLIC_HART_BASE  0x201000  // Hart 0 S-Mode Priority Threshold in PLIC
#define PLIC_HART_SIZE  0x1000    // Address Range of Hart 0 PLIC

(How we got the RISC-V Addresses)

What’s this Boot Code?

TinyEMU needs a tiny chunk of RISC-V Machine Code that will jump to our Kernel Image (and pass the Device Tree): riscv_machine.c

// At TinyEMU Startup: Init the
// Emulated RAM...
static void copy_bios(...) {
  ...
  // Init the TinyEMU Boot Code at
  // Address 0x1000 (ram_ptr is 0x0)
  uint32_t *q = (uint32_t *)(ram_ptr + 0x1000);

  // Load into Register T0 the RAM_BASE_ADDR (0x5020_0000)
  // Load into Register A1 the Binary Device Tree
  q[0] = 0x297 + RAM_BASE_ADDR - 0x1000;    // auipc t0, jump_addr
  q[1] = 0x597;                             // auipc a1, dtb
  q[2] = 0x58593 + ((fdt_addr - 4) << 20);  // addi  a1, a1, dtb

  // Load into Register A0 the Hart ID (RISC-V CPU ID: 0)
  // Jump to Register T0: Our Kernel at RAM_BASE_ADDR (0x5020_0000)
  q[3] = 0xf1402573;  // csrr a0, mhartid
  q[4] = 0x00028067;  // jalr zero, t0, jump_addr

And that’s our barebones Ox64 Emulator, all ready to run…

(Remember to enable Exception Logging)

TinyEMU Emulator at the Command Line

§3 Run TinyEMU Emulator

We modded TinyEMU to emulate Ox64. What happens when we run it?

We see signs of life… NuttX Kernel is actually booting in our Ox64 Emulator!

## Download the TinyEMU Config
## and NuttX Kernel Image
$ wget https://raw.githubusercontent.com/lupyuen/nuttx-tinyemu/main/docs/ox64/root-riscv64.cfg
$ wget https://github.com/lupyuen/nuttx-ox64/releases/download/nuttx-ox64-2024-01-20/Image

## Boot TinyEMU with NuttX Kernel
$ temu root-riscv64.cfg | more

## NuttX Kernel writes to CSR Registers
csr_write: csr=0x104 val=0x0
csr_write: csr=0x105 val=0x50200090
csr_write: csr=0x100 val=0x200000000
csr_write: csr=0x140 val=0x50400cd0
csr_write: csr=0x180 val=0x0
csr_write: csr=0x105 val=0x50200090
csr_write: csr=0x100 val=0x200002000
csr_write: csr=0x003 val=0x0
csr_write: csr=0x100 val=0x8000000200006000

## NuttX Kernel does invalid
## reads and writes
target_read_slow:
  invalid physical address
  0x30002084
target_write_slow: 
  invalid physical address 
  0x30002088

(See the Complete Log)

What’s root-riscv64.cfg?

It’s the TinyEMU Config that will boot NuttX Kernel in our Ox64 Emulator: root-riscv64.cfg

/* VM configuration file */
{
  version: 1,
  machine: "riscv64",
  memory_size: 256,
  bios: "Image",
}

Image is the NuttX Kernel Image that comes from a typical NuttX Build for Ox64.

What are the CSR Writes?

## NuttX Kernel writes to CSR Registers
csr_write: csr=0x104 val=0x0
csr_write: csr=0x105 val=0x50200090
csr_write: csr=0x100 val=0x200000000

CSR refers to Control and Status Registers. They’re the System Registers in our RISC-V SoC (BL808)…

These are all Supervisor-Mode CSR Registers. (We’ll find out why)

Why is it writing to CSR Registers?

This comes from our NuttX Boot Code (in RISC-V Assembly): bl808_head.S

/* Disable all interrupts
  (i.e. timer, external)
  in SIE CSR */
csrw  sie, zero

/* Set the Interrupt Vector Table
   in STVEC CSR */
la    t0, __trap_vec
csrw  stvec, t0

Now we talk about the funny reads and writes…

BL808 UART Registers (Page 427)

BL808 UART Registers (Page 427)

§4 UART Registers for BL808 SoC

What are 0x3000_2084 and 0x3000_2088? Why are they Invalid Addresses?

## NuttX Kernel does invalid
## reads and writes
target_read_slow:
  invalid physical address
  0x30002084

target_write_slow: 
  invalid physical address 
  0x30002088

(See the Complete Log)

We dig around the BL808 Reference Manual (pic above) and we discover these UART Registers

NuttX Kernel is trying to print something to the UART Console! (Pic below)

// NuttX sends a character to
// the UART Port...
void bl808_send(struct uart_dev_s *dev, int ch) {
  ...
  // Wait for Transmit FIFO to be empty.
  // FIFO_CONFIG_1 is 0x3000_2084
  // TX_CNT_MASK is 0x3F
  while ((getreg32(BL808_UART_FIFO_CONFIG_1(uart_idx)) &
    UART_FIFO_CONFIG_1_TX_CNT_MASK) == 0) {}

  // Write character to Transmit FIFO.
  // FIFO_WDATA is 0x3000_2088
  putreg32(ch, BL808_UART_FIFO_WDATA(uart_idx));

(0x3000_2000 is the UART3 Base Address, Page 41)

(More about BL808 UART)

But why are they Invalid Addresses?

We haven’t defined in TinyEMU the addresses for Memory-Mapped Input / Output. (Like for UART Registers)

That’s why TinyEMU won’t read and write our UART Registers. Let’s fix this…

Emulating the UART Registers with TinyEMU

§5 Intercept the UART Registers

NuttX tries to print something and fails…

How to fix the UART Registers in our Ox64 Emulator?

Inside TinyEMU, we intercept all “read 0x3000_2084” and “write 0x3000_2088”. And we pretend to be a UART Port (pic above)…

§5.1 Emulate the UART Status

Earlier we said…

0x3000_2084 is uart_fifo_config_1 (Page 427)

We read this UART Register to check if UART Transmit is ready (for more output)

In TinyEMU: We intercept “read 0x3000_2084” and return the value 32: riscv_cpu.c

// TinyEMU reads a Memory Address...
int target_read_slow(RISCVCPUState *s, mem_uint_t *pval, target_ulong addr, int size_log2) {
...
  // If the Memory Address is
  // not mapped...
  pr = get_phys_mem_range(s->mem_map, paddr);
  if (!pr) {

    // Ignore the Upper Bits of the Memory Address.
    // Because of T-Head MMU Flags, our Kernel might read from 0x4000000030002084 
    // https://lupyuen.codeberg.page/articles/plic3#t-head-errata
    switch(paddr & 0xfffffffffffful) {  

      // If we're reading uart_fifo_config_1:
      // Tell Emulator that UART
      // Transmit is always ready
      case 0x30002084:
        ret = 32; break;  // UART Transmit Buffer Size defaults to 32

      // If Unknown Address:
      // Print "target_read_slow: invalid physical address"
      default:
        ...

Why 32?

Our NuttX UART Driver checks the lower bits of 0x3000_2084: bl808_serial.c

// NuttX sends a character to
// the UART Port...
void bl808_send(struct uart_dev_s *dev, int ch) {
  ...
  // Wait for Transmit FIFO to be empty.
  // FIFO_CONFIG_1 is 0x3000_2084
  // TX_CNT_MASK is 0x3F
  while ((getreg32(BL808_UART_FIFO_CONFIG_1(uart_idx)) &
    UART_FIFO_CONFIG_1_TX_CNT_MASK) == 0) {}

  // Omitted: Write character to Transmit FIFO.

And the UART Transmit Buffer Size (Page 427) defaults to 32. Thus we always return 32.

Emulating the UART Output Register with TinyEMU

§5.2 Emulate the UART Output

Earlier we saw…

0x3000_2088 is uart_fifo_wdata (Page 428)

We write to this UART Register to print a character to UART Output

In TinyEMU: We intercept all “write 0x3000_2088” by printing the character (pic above): riscv_cpu.c

// TinyEMU writes to a Memory Address...
int target_write_slow(RISCVCPUState *s, target_ulong addr, mem_uint_t val, int size_log2) {
...
  // If the Memory Address is
  // not mapped...
  pr = get_phys_mem_range(s->mem_map, paddr);
  if (!pr) {

    // Ignore the Upper Bits of the Memory Address.
    // Because of T-Head MMU Flags, our Kernel might write to 0x4000000030002088
    // https://lupyuen.codeberg.page/articles/plic3#t-head-errata
    switch(paddr & 0xfffffffffffful) { 

      // If we're writing to uart_fifo_wdata:
      // Print the character (val)
      case 0x30002088:
        char buf[1];
        buf[0] = val;
        print_console(NULL, buf, 1);
        break;

      // If Unknown Address:
      // Print "target_write_slow: invalid physical address"
      default:
        ...

NuttX is ready to talk, we test our Emulated UART Port…

(print_console is defined here)

(riscv_machine_init inits the console)

TinyEMU Emulator emulates UART Output

§6 Emulator Prints To Console

We modded our Ox64 Emulator to handle UART Output. Does it work?

Yep, we see NuttX booting on our Ox64 Emulator yay! (Pic above)

## Boot TinyEMU with NuttX Kernel
$ temu root-riscv64.cfg | more

## NuttX Kernel inits the Kernel Memory
nx_start: Entry
mm_initialize: Heap: name=Kmem, start=0x50407c00 size=2065408
mm_addregion:  [Kmem] Region 1: base=0x50407ea8 size=2064720

## NuttX Kernel starts the UART Driver
## (What are the Invalid Addresses?)
uart_register: Registering /dev/console
target_read_slow:  invalid physical address 0x0000000030002024
target_write_slow: invalid physical address 0x0000000030002024

## NuttX Kernel starts the NuttX Shell
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init

## NuttX Kernel creates the Heap Memory
## for NuttX Shell
mm_initialize: Heap: name=(null), start=0x80200000 size=528384
mm_addregion: [(null)] Region 1: base=0x802002a8 size=527696

(See the Complete Log)

Followed by this RISC-V Exception

## NuttX Shell crashes with a
## RISC-V Exception, MCAUSE is 8
raise_exception2: cause=8, tval=0x0
pc =00000000800019c6 ra =0000000080000086 sp =0000000080202bc0 gp =0000000000000000
tp =0000000000000000 t0 =0000000000000000 t1 =0000000000000000 t2 =0000000000000000
s0 =0000000000000001 s1 =0000000080202010 a0 =000000000000000d a1 =0000000000000000
a2 =0000000080202bc8 a3 =0000000080202010 a4 =0000000080000030 a5 =0000000000000000
a6 =0000000000000101 a7 =0000000000000000 s2 =0000000000000000 s3 =0000000000000000
s4 =0000000000000000 s5 =0000000000000000 s6 =0000000000000000 s7 =0000000000000000
s8 =0000000000000000 s9 =0000000000000000 s10=0000000000000000 s11=0000000000000000
t3 =0000000000000000 t4 =0000000000000000 t5 =0000000000000000 t6 =0000000000000000
priv=U mstatus=0000000a0006806
mideleg=0000000000000000 mie=0000000000000000 mip=0000000000000080

## Illegal Instruction at 0x0
raise_exception2: cause=2, tval=0x0

(See the Complete Log)

Why? We investigate the alligator in the vest…

System Call Fails in NuttX Kernel

§7 RISC-V Exception in Emulator

What’s this RISC-V Exception?

## NuttX Shell crashes with a 
## RISC-V Exception, MCAUSE is 8
raise_exception2:
  cause=8, tval=0x0
  pc=800019c6

(See the Complete Log)

We track down the offending Code Address: 0x8000_19C6

Someplace in NuttX Kernel?

This address comes from the Virtual Memory of a NuttX App (not the NuttX Kernel): nsh/defconfig

## Virtual Memory of NuttX Apps:
## Code, Data and Heap
CONFIG_ARCH_TEXT_VBASE=0x80000000
CONFIG_ARCH_DATA_VBASE=0x80100000
CONFIG_ARCH_HEAP_VBASE=0x80200000

What NuttX App are we running?

The only NuttX App we’re running at Startup is the NuttX Shell.

Thus we look up the RISC-V Disassembly for the NuttX Shell: init.S

// NuttX Shell makes a System Call
// to fetch a Scheduler Parameter
nuttx/syscall/proxies/PROXY_sched_getparam.c:8
  int sched_getparam(pid_t parm1, FAR struct sched_param * parm2) {
  ...
  // ECALL fails with a 
  // RISC-V Exception
  nuttx/include/arch/syscall.h:229
    19c6: 00000073  ecall

(See the Source Code)

What’s this ECALL?

At 0x19C6 we see the RISC-V ECALL Instruction that will jump from our NuttX App (RISC-V User Mode) to NuttX Kernel (RISC-V Supervisor Mode).

Hence our NuttX Shell is making a System Call to NuttX Kernel. (Pic above)

Why did it fail? We’ll come back to this, first we surf the web…

(We quit if MCAUSE is 2, otherwise we loop forever)

Ox64 BL808 Emulator with TinyEMU RISC-V Emulator and Apache NuttX RTOS

Live Demo of Ox64 BL808 Emulator

§8 Emulator in the Web Browser

Will our Ox64 Emulator run in the Web Browser?

Let’s find out! We compile TinyEMU to WebAssembly with Emscripten

## Download the Web Server files
cd $HOME
git clone https://github.com/lupyuen/nuttx-tinyemu

## Compile TinyEMU into WebAssembly with Emscripten
## For macOS: brew install emscripten
sudo apt install emscripten
cd $HOME/ox64-tinyemu
make -f Makefile.js

## TODO: `--memory-init-file` is no longer supported

## Copy the generated JavaScript and
## WebAssembly to our Web Server
cp js/riscvemu64-wasm.js \
   js/riscvemu64-wasm.wasm \
   $HOME/nuttx-tinyemu/docs/ox64/

## Start the Web Server
cargo install simple-http-server
simple-http-server $HOME/nuttx-tinyemu/docs

(See the Build Script)

(See the Web Server Files)

We point our Web Browser to…

http://0.0.0.0:8000/ox64/index.html

And our Ox64 Emulator appears in the Web Browser! (Pic above)

(Live Demo of Ox64 Emulator)

(Watch the Demo on YouTube)

(How we got the WebAssembly Files)

What about Console Input?

Console Input requires UART Interrupts. (Pic below)

We’ll explain the details in the next article…

One more thing to tweak…

(We’ll emulate BL808 GPIO to Blink a Virtual LED)

(Maybe wrap TinyEMU with Zig for Memory Safety and Simpler WebAssembly?)

UART Interrupts for Ox64 BL808 SBC

§9 Machine Mode vs Supervisor Mode

Back to our earlier question: Why did our System Call fail?

Our NuttX App (NuttX Shell) tried to make a System Call (ECALL) to NuttX Kernel. And it failed: init.S

// NuttX Shell makes a System Call
// to fetch a Scheduler Parameter
nuttx/syscall/proxies/PROXY_sched_getparam.c:8
  int sched_getparam(pid_t parm1, FAR struct sched_param * parm2) {
  ...
  // ECALL fails with a
  // RISC-V Exception
  nuttx/include/arch/syscall.h:229
    19c6: 00000073  ecall

(See the Source Code)

What’s ECALL again?

The RISC-V ECALL Instruction normally jumps…

System Calls are absolutely essential. That’s how our apps will execute system functions, like printing to the Console Output.

Why did ECALL fail?

That’s because our NuttX Kernel is actually running in RISC-V Machine Mode, not Supervisor Mode!

Machine Mode is the most powerful mode in a RISC-V System, more powerful than Supervisor Mode and User Mode. However NuttX expects to boot in Supervisor Mode. (Pic below)

(Which explains the Supervisor-Mode CSR Registers we saw earlier)

NuttX Kernel won’t work in Machine Mode

Huh! How did that happen?

TinyEMU always starts in Machine Mode. Everything we saw today: That’s all running in (super-powerful) Machine Mode.

Which sounds super simplistic: A Real Ox64 SBC will run in Machine, Supervisor AND User Modes (pic below)…

  1. Ox64 boots the OpenSBI Supervisor Binary Interface in Machine Mode (Think BIOS for RISC-V Machines)

  2. OpenSBI starts the U-Boot Bootloader in Supervisor Mode

  3. U-Boot starts the NuttX Kernel, also in Supervisor Mode

  4. And NuttX Kernel starts the NuttX Apps in User Mode

Ox64 SBC will run in Machine, Supervisor AND User Modes

So we’ll boot NuttX Kernel in Supervisor Mode?

Yep we shall tweak TinyEMU to start NuttX in Supervisor Mode. (Instead of Machine Mode)

We’ll explain the details in the next article…

TinyEMU will boot NuttX in Supervisor Mode

Any other gotchas?

There’s a tiny quirk: NuttX Kernel will make an ECALL too…

NuttX Kernel makes a System Call to OpenSBI to start the System Timer. (Pic above)

Will we run OpenSBI on TinyEMU?

That’s not necessary. We’ll emulate the System Timer in TinyEMU.

We’ll explain the details in the next article…

JavaScript Console of Ox64 BL808 Emulator

§10 What’s Next

Today we created a barebones Ox64 BL808 Emulator that runs in the Web Browser…

Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/tinyemu2.md

BL808 Memory Map (Page 41)

BL808 Memory Map (Page 41)

§11 Appendix: RISC-V Addresses for Ox64

Earlier we tweaked the RISC-V Addresses in TinyEMU, so that they match the Bouffalo Lab BL808 SoC (pic above): riscv_machine.c

// RISC-V Addresses for TinyEMU
// (modded for Ox64 BL808)
#define LOW_RAM_SIZE    0x00010000ul  // 64 KB of Boot Code at Address 0x0
#define RAM_BASE_ADDR   0x50200000ul  // Our Kernel boots here
#define PLIC_BASE_ADDR  0xe0000000ul  // Platform-Level Interrupt Controller (PLIC)
#define PLIC_SIZE       0x00400000ul  // Address Range of PLIC
#define CLINT_BASE_ADDR 0x02000000ul  // CLINT is Unused
#define CLINT_SIZE      0x000c0000ul  // CLINT is Unused
...
#define PLIC_HART_BASE  0x201000  // Hart 0 S-Mode Priority Threshold in PLIC
#define PLIC_HART_SIZE  0x1000    // Address Range of Hart 0 PLIC

This is how we derived the above RISC-V Addresses…

// 64 KB of Boot Code at Address 0x0
#define LOW_RAM_SIZE    0x00010000ul

Low RAM: This setting is specfic to TinyEMU, we left it unchanged. The Low RAM contains…

// Our Kernel boots here
#define RAM_BASE_ADDR   0x50200000ul

RAM Base: NuttX (also Linux) boots at the above RAM Address (because of U-Boot Bootloader), as explained here…

// Platform-Level Interrupt Controller (PLIC)
// and Address Range of PLIC
#define PLIC_BASE_ADDR  0xe0000000ul  
#define PLIC_SIZE       0x00400000ul

Platform-Level Interrupt Controller (PLIC): We documented the PLIC Addresses here…

// Hart 0 S-Mode Priority Threshold in PLIC
// and Address Range of Hart 0 PLIC
#define PLIC_HART_BASE  0x201000
#define PLIC_HART_SIZE  0x1000

PLIC Hart: We specify Hart 0, Supervisor Mode as explained here…

// Core-Local Interrupt Controller (CLINT) is Unused
#define CLINT_BASE_ADDR 0x02000000ul
#define CLINT_SIZE      0x000c0000ul

Core-Local Interrupt Controller (CLINT) is unused. We left the setting unchanged.