📝 21 Jan 2024
(Live Demo of Ox64 BL808 Emulator)
In olden times we had Computer Games (plus Operating Systems) on 5.25-inch Floppy Disks. And we’d boot the Floppy Disks (clackety-clack) on Apple II Computers with 64 KB RAM.
Today (40 years later) we boot microSD Cards (clickety-click) on Ox64 BL808 RISC-V Single-Board Computers with 64 MB RAM. (Pic below)
What if we could turn it into a Virtual Ox64 SBC that boots in our Web Browser? (Pic above) Exactly like an Emulated Apple II!
In this article we…
Take Apache NuttX RTOS precompiled for Ox64
(Without any modifications)
Boot it on the TinyEMU RISC-V Emulator
Create our own Emulator for Ox64 SBC
(With minor tweaks to TinyEMU)
And run everything in our Web Browser
(Thanks to WebAssembly)
Why NuttX?
Apache NuttX RTOS is a tiny operating system for 64-bit RISC-V Machines. (Also Arm, x64, ESP32, …)
Which makes it easier to understand everything that happens as NuttX boots on our Ox64 Emulator.
What’s this TinyEMU?
TinyEMU is a barebones 64-bit RISC-V Emulator.
It doesn’t have all the features of QEMU Emulator. But TinyEMU runs in a Web Browser and it’s much simpler for modding!
We begin by installing (our modded) TinyEMU for the Command Line…
## Download TinyEMU modded for Ox64
git clone https://github.com/lupyuen/ox64-tinyemu
cd ox64-tinyemu
## For Ubuntu:
sudo apt install libcurl4-openssl-dev libssl-dev zlib1g-dev libsdl2-dev
make
## For macOS:
brew install openssl sdl2
make \
CFLAGS="-I$(brew --prefix)/opt/openssl/include -I$(brew --prefix)/opt/sdl2/include" \
LDFLAGS="-L$(brew --prefix)/opt/openssl/lib -L$(brew --prefix)/opt/sdl2/lib" \
CONFIG_MACOS=y
What about TinyEMU for the Web Browser?
No Worries! Everything that runs in Command Line TinyEMU… Will also run in Web Browser TinyEMU.
We tweak TinyEMU for Ox64…
TinyEMU needs to emulate our Ox64 BL808 SBC. What shall we tweak?
TinyEMU is hardcoded to run at Fixed RISC-V Addresses. (Yep it’s really barebones)
We tweak the RISC-V Addresses in TinyEMU, so that they match the Bouffalo Lab BL808 SoC (pic above): riscv_machine.c
// RISC-V Addresses for TinyEMU
// (modded for Ox64 BL808)
#define LOW_RAM_SIZE 0x00010000ul // 64 KB of Boot Code at Address 0x0
#define RAM_BASE_ADDR 0x50200000ul // Our Kernel boots here
#define PLIC_BASE_ADDR 0xe0000000ul // Platform-Level Interrupt Controller (PLIC)
#define PLIC_SIZE 0x00400000ul // Address Range of PLIC
#define CLINT_BASE_ADDR 0x02000000ul // CLINT is Unused
#define CLINT_SIZE 0x000c0000ul // CLINT is Unused
...
#define PLIC_HART_BASE 0x201000 // Hart 0 S-Mode Priority Threshold in PLIC
#define PLIC_HART_SIZE 0x1000 // Address Range of Hart 0 PLIC
(How we got the RISC-V Addresses)
What’s this Boot Code?
TinyEMU needs a tiny chunk of RISC-V Machine Code that will jump to our Kernel Image (and pass the Device Tree): riscv_machine.c
// At TinyEMU Startup: Init the
// Emulated RAM...
static void copy_bios(...) {
...
// Init the TinyEMU Boot Code at
// Address 0x1000 (ram_ptr is 0x0)
uint32_t *q = (uint32_t *)(ram_ptr + 0x1000);
// Load into Register T0 the RAM_BASE_ADDR (0x5020_0000)
// Load into Register A1 the Binary Device Tree
q[0] = 0x297 + RAM_BASE_ADDR - 0x1000; // auipc t0, jump_addr
q[1] = 0x597; // auipc a1, dtb
q[2] = 0x58593 + ((fdt_addr - 4) << 20); // addi a1, a1, dtb
// Load into Register A0 the Hart ID (RISC-V CPU ID: 0)
// Jump to Register T0: Our Kernel at RAM_BASE_ADDR (0x5020_0000)
q[3] = 0xf1402573; // csrr a0, mhartid
q[4] = 0x00028067; // jalr zero, t0, jump_addr
And that’s our barebones Ox64 Emulator, all ready to run…
(Remember to enable Exception Logging)
We modded TinyEMU to emulate Ox64. What happens when we run it?
We see signs of life… NuttX Kernel is actually booting in our Ox64 Emulator!
## Download the TinyEMU Config
## and NuttX Kernel Image
$ wget https://raw.githubusercontent.com/lupyuen/nuttx-tinyemu/main/docs/ox64/root-riscv64.cfg
$ wget https://github.com/lupyuen/nuttx-ox64/releases/download/nuttx-ox64-2024-01-20/Image
## Boot TinyEMU with NuttX Kernel
$ temu root-riscv64.cfg | more
## NuttX Kernel writes to CSR Registers
csr_write: csr=0x104 val=0x0
csr_write: csr=0x105 val=0x50200090
csr_write: csr=0x100 val=0x200000000
csr_write: csr=0x140 val=0x50400cd0
csr_write: csr=0x180 val=0x0
csr_write: csr=0x105 val=0x50200090
csr_write: csr=0x100 val=0x200002000
csr_write: csr=0x003 val=0x0
csr_write: csr=0x100 val=0x8000000200006000
## NuttX Kernel does invalid
## reads and writes
target_read_slow:
invalid physical address
0x30002084
target_write_slow:
invalid physical address
0x30002088
What’s root-riscv64.cfg?
It’s the TinyEMU Config that will boot NuttX Kernel in our Ox64 Emulator: root-riscv64.cfg
/* VM configuration file */
{
version: 1,
machine: "riscv64",
memory_size: 256,
bios: "Image",
}
Image is the NuttX Kernel Image that comes from a typical NuttX Build for Ox64.
What are the CSR Writes?
## NuttX Kernel writes to CSR Registers
csr_write: csr=0x104 val=0x0
csr_write: csr=0x105 val=0x50200090
csr_write: csr=0x100 val=0x200000000
CSR refers to Control and Status Registers. They’re the System Registers in our RISC-V SoC (BL808)…
CSR 0x104: Supervisor-Mode Interrupt Enable
(Enable or Disable Interrupts)
CSR 0x105: Supervisor-Mode Trap Vector Base Address
(Set the Interrupt Vector Table)
CSR 0x100: Supervisor-Mode Status
(Set the Status)
These are all Supervisor-Mode CSR Registers. (We’ll find out why)
Why is it writing to CSR Registers?
This comes from our NuttX Boot Code (in RISC-V Assembly): bl808_head.S
/* Disable all interrupts
(i.e. timer, external)
in SIE CSR */
csrw sie, zero
/* Set the Interrupt Vector Table
in STVEC CSR */
la t0, __trap_vec
csrw stvec, t0
Now we talk about the funny reads and writes…
What are 0x3000_2084 and 0x3000_2088? Why are they Invalid Addresses?
## NuttX Kernel does invalid
## reads and writes
target_read_slow:
invalid physical address
0x30002084
target_write_slow:
invalid physical address
0x30002088
We dig around the BL808 Reference Manual (pic above) and we discover these UART Registers…
0x3000_2088 is uart_fifo_wdata (Page 428)
We write to this UART Register to print a character to UART Output.
0x3000_2084 is uart_fifo_config_1 (Page 427)
We read this UART Register to check if UART Transmit is ready (for more output).
Which explains why we always see “read 0x3000_2084” before “write 0x3000_2088”…
NuttX Kernel is trying to print something to the UART Console! (Pic below)
// NuttX sends a character to
// the UART Port...
void bl808_send(struct uart_dev_s *dev, int ch) {
...
// Wait for Transmit FIFO to be empty.
// FIFO_CONFIG_1 is 0x3000_2084
// TX_CNT_MASK is 0x3F
while ((getreg32(BL808_UART_FIFO_CONFIG_1(uart_idx)) &
UART_FIFO_CONFIG_1_TX_CNT_MASK) == 0) {}
// Write character to Transmit FIFO.
// FIFO_WDATA is 0x3000_2088
putreg32(ch, BL808_UART_FIFO_WDATA(uart_idx));
(0x3000_2000 is the UART3 Base Address, Page 41)
But why are they Invalid Addresses?
We haven’t defined in TinyEMU the addresses for Memory-Mapped Input / Output. (Like for UART Registers)
That’s why TinyEMU won’t read and write our UART Registers. Let’s fix this…
NuttX tries to print something and fails…
How to fix the UART Registers in our Ox64 Emulator?
Inside TinyEMU, we intercept all “read 0x3000_2084” and “write 0x3000_2088”. And we pretend to be a UART Port (pic above)…
Earlier we said…
0x3000_2084is uart_fifo_config_1 (Page 427)
We read this UART Register to check if UART Transmit is ready (for more output)
In TinyEMU: We intercept “read 0x3000_2084” and return the value 32: riscv_cpu.c
// TinyEMU reads a Memory Address...
int target_read_slow(RISCVCPUState *s, mem_uint_t *pval, target_ulong addr, int size_log2) {
...
// If the Memory Address is
// not mapped...
pr = get_phys_mem_range(s->mem_map, paddr);
if (!pr) {
// Ignore the Upper Bits of the Memory Address.
// Because of T-Head MMU Flags, our Kernel might read from 0x4000000030002084
// https://lupyuen.codeberg.page/articles/plic3#t-head-errata
switch(paddr & 0xfffffffffffful) {
// If we're reading uart_fifo_config_1:
// Tell Emulator that UART
// Transmit is always ready
case 0x30002084:
ret = 32; break; // UART Transmit Buffer Size defaults to 32
// If Unknown Address:
// Print "target_read_slow: invalid physical address"
default:
...
Why 32?
Our NuttX UART Driver checks the lower bits of 0x3000_2084: bl808_serial.c
// NuttX sends a character to
// the UART Port...
void bl808_send(struct uart_dev_s *dev, int ch) {
...
// Wait for Transmit FIFO to be empty.
// FIFO_CONFIG_1 is 0x3000_2084
// TX_CNT_MASK is 0x3F
while ((getreg32(BL808_UART_FIFO_CONFIG_1(uart_idx)) &
UART_FIFO_CONFIG_1_TX_CNT_MASK) == 0) {}
// Omitted: Write character to Transmit FIFO.
And the UART Transmit Buffer Size (Page 427) defaults to 32. Thus we always return 32.
Earlier we saw…
0x3000_2088is uart_fifo_wdata (Page 428)
We write to this UART Register to print a character to UART Output
In TinyEMU: We intercept all “write 0x3000_2088” by printing the character (pic above): riscv_cpu.c
// TinyEMU writes to a Memory Address...
int target_write_slow(RISCVCPUState *s, target_ulong addr, mem_uint_t val, int size_log2) {
...
// If the Memory Address is
// not mapped...
pr = get_phys_mem_range(s->mem_map, paddr);
if (!pr) {
// Ignore the Upper Bits of the Memory Address.
// Because of T-Head MMU Flags, our Kernel might write to 0x4000000030002088
// https://lupyuen.codeberg.page/articles/plic3#t-head-errata
switch(paddr & 0xfffffffffffful) {
// If we're writing to uart_fifo_wdata:
// Print the character (val)
case 0x30002088:
char buf[1];
buf[0] = val;
print_console(NULL, buf, 1);
break;
// If Unknown Address:
// Print "target_write_slow: invalid physical address"
default:
...
NuttX is ready to talk, we test our Emulated UART Port…
(print_console is defined here)
(riscv_machine_init inits the console)
We modded our Ox64 Emulator to handle UART Output. Does it work?
Yep, we see NuttX booting on our Ox64 Emulator yay! (Pic above)
## Boot TinyEMU with NuttX Kernel
$ temu root-riscv64.cfg | more
## NuttX Kernel inits the Kernel Memory
nx_start: Entry
mm_initialize: Heap: name=Kmem, start=0x50407c00 size=2065408
mm_addregion: [Kmem] Region 1: base=0x50407ea8 size=2064720
## NuttX Kernel starts the UART Driver
## (What are the Invalid Addresses?)
uart_register: Registering /dev/console
target_read_slow: invalid physical address 0x0000000030002024
target_write_slow: invalid physical address 0x0000000030002024
## NuttX Kernel starts the NuttX Shell
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
## NuttX Kernel creates the Heap Memory
## for NuttX Shell
mm_initialize: Heap: name=(null), start=0x80200000 size=528384
mm_addregion: [(null)] Region 1: base=0x802002a8 size=527696
Followed by this RISC-V Exception…
## NuttX Shell crashes with a
## RISC-V Exception, MCAUSE is 8
raise_exception2: cause=8, tval=0x0
pc =00000000800019c6 ra =0000000080000086 sp =0000000080202bc0 gp =0000000000000000
tp =0000000000000000 t0 =0000000000000000 t1 =0000000000000000 t2 =0000000000000000
s0 =0000000000000001 s1 =0000000080202010 a0 =000000000000000d a1 =0000000000000000
a2 =0000000080202bc8 a3 =0000000080202010 a4 =0000000080000030 a5 =0000000000000000
a6 =0000000000000101 a7 =0000000000000000 s2 =0000000000000000 s3 =0000000000000000
s4 =0000000000000000 s5 =0000000000000000 s6 =0000000000000000 s7 =0000000000000000
s8 =0000000000000000 s9 =0000000000000000 s10=0000000000000000 s11=0000000000000000
t3 =0000000000000000 t4 =0000000000000000 t5 =0000000000000000 t6 =0000000000000000
priv=U mstatus=0000000a0006806
mideleg=0000000000000000 mie=0000000000000000 mip=0000000000000080
## Illegal Instruction at 0x0
raise_exception2: cause=2, tval=0x0
Why? We investigate the alligator in the vest…
What’s this RISC-V Exception?
## NuttX Shell crashes with a
## RISC-V Exception, MCAUSE is 8
raise_exception2:
cause=8, tval=0x0
pc=800019c6
We track down the offending Code Address: 0x8000_19C6
Someplace in NuttX Kernel?
This address comes from the Virtual Memory of a NuttX App (not the NuttX Kernel): nsh/defconfig
## Virtual Memory of NuttX Apps:
## Code, Data and Heap
CONFIG_ARCH_TEXT_VBASE=0x80000000
CONFIG_ARCH_DATA_VBASE=0x80100000
CONFIG_ARCH_HEAP_VBASE=0x80200000
What NuttX App are we running?
The only NuttX App we’re running at Startup is the NuttX Shell.
Thus we look up the RISC-V Disassembly for the NuttX Shell: init.S
// NuttX Shell makes a System Call
// to fetch a Scheduler Parameter
nuttx/syscall/proxies/PROXY_sched_getparam.c:8
int sched_getparam(pid_t parm1, FAR struct sched_param * parm2) {
...
// ECALL fails with a
// RISC-V Exception
nuttx/include/arch/syscall.h:229
19c6: 00000073 ecall
What’s this ECALL?
At 0x19C6 we see the RISC-V ECALL Instruction that will jump from our NuttX App (RISC-V User Mode) to NuttX Kernel (RISC-V Supervisor Mode).
Hence our NuttX Shell is making a System Call to NuttX Kernel. (Pic above)
Why did it fail? We’ll come back to this, first we surf the web…
(We quit if MCAUSE is 2, otherwise we loop forever)
Live Demo of Ox64 BL808 Emulator
Will our Ox64 Emulator run in the Web Browser?
Let’s find out! We compile TinyEMU to WebAssembly with Emscripten…
## Download the Web Server files
cd $HOME
git clone https://github.com/lupyuen/nuttx-tinyemu
## Compile TinyEMU into WebAssembly with Emscripten
## For macOS: brew install emscripten
sudo apt install emscripten
cd $HOME/ox64-tinyemu
make -f Makefile.js
## TODO: `--memory-init-file` is no longer supported
## Should we remove it?
## Copy the generated JavaScript and
## WebAssembly to our Web Server
cp js/riscvemu64-wasm.js \
js/riscvemu64-wasm.wasm \
$HOME/nuttx-tinyemu/docs/ox64/
## Start the Web Server
cargo install simple-http-server
simple-http-server $HOME/nuttx-tinyemu/docs
We point our Web Browser to…
http://0.0.0.0:8000/ox64/index.html
And our Ox64 Emulator appears in the Web Browser! (Pic above)
(How we got the WebAssembly Files)
What about Console Input?
Console Input requires UART Interrupts. (Pic below)
We’ll explain the details in the next article…
One more thing to tweak…
(We’ll emulate BL808 GPIO to Blink a Virtual LED)
(Maybe wrap TinyEMU with Zig for Memory Safety and Simpler WebAssembly?)
Back to our earlier question: Why did our System Call fail?
Our NuttX App (NuttX Shell) tried to make a System Call (ECALL) to NuttX Kernel. And it failed: init.S
// NuttX Shell makes a System Call
// to fetch a Scheduler Parameter
nuttx/syscall/proxies/PROXY_sched_getparam.c:8
int sched_getparam(pid_t parm1, FAR struct sched_param * parm2) {
...
// ECALL fails with a
// RISC-V Exception
nuttx/include/arch/syscall.h:229
19c6: 00000073 ecall
What’s ECALL again?
The RISC-V ECALL Instruction normally jumps…
From our NuttX App
(In RISC-V User Mode)
To the NuttX Kernel
(In RISC-V Supervisor Mode)
In order to make a System Call
(Which failed)
System Calls are absolutely essential. That’s how our apps will execute system functions, like printing to the Console Output.
Why did ECALL fail?
That’s because our NuttX Kernel is actually running in RISC-V Machine Mode, not Supervisor Mode!
Machine Mode is the most powerful mode in a RISC-V System, more powerful than Supervisor Mode and User Mode. However NuttX expects to boot in Supervisor Mode. (Pic below)
(Which explains the Supervisor-Mode CSR Registers we saw earlier)
Huh! How did that happen?
TinyEMU always starts in Machine Mode. Everything we saw today: That’s all running in (super-powerful) Machine Mode.
Which sounds super simplistic: A Real Ox64 SBC will run in Machine, Supervisor AND User Modes (pic below)…
Ox64 boots the OpenSBI Supervisor Binary Interface in Machine Mode (Think BIOS for RISC-V Machines)
OpenSBI starts the U-Boot Bootloader in Supervisor Mode
U-Boot starts the NuttX Kernel, also in Supervisor Mode
And NuttX Kernel starts the NuttX Apps in User Mode
So we’ll boot NuttX Kernel in Supervisor Mode?
Yep we shall tweak TinyEMU to start NuttX in Supervisor Mode. (Instead of Machine Mode)
We’ll explain the details in the next article…
Any other gotchas?
There’s a tiny quirk: NuttX Kernel will make an ECALL too…
NuttX Kernel makes a System Call to OpenSBI to start the System Timer. (Pic above)
Will we run OpenSBI on TinyEMU?
That’s not necessary. We’ll emulate the System Timer in TinyEMU.
We’ll explain the details in the next article…
Today we created a barebones Ox64 BL808 Emulator that runs in the Web Browser…
We made it by (slightly) tweaking TinyEMU RISC-V Emulator and compiling to WebAssembly
Will it boot Apache NuttX RTOS unmodified? Yep NuttX Kernel boots all the way!
Though NuttX Apps will crash, because NuttX needs to boot in RISC-V Supervisor Mode (not Machine Mode)
And Console Input won’t work until we Emulate UART Interrupts
Up Next: Ox64 Emulator might Blink a Virtual LED. And it might get triggered daily for Automated NuttX Testing.
40 Years from Today: Maybe we’ll play with a Better Ox64 Emulator?
Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…
lupyuen.github.io/src/tinyemu2.md
Earlier we tweaked the RISC-V Addresses in TinyEMU, so that they match the Bouffalo Lab BL808 SoC (pic above): riscv_machine.c
// RISC-V Addresses for TinyEMU
// (modded for Ox64 BL808)
#define LOW_RAM_SIZE 0x00010000ul // 64 KB of Boot Code at Address 0x0
#define RAM_BASE_ADDR 0x50200000ul // Our Kernel boots here
#define PLIC_BASE_ADDR 0xe0000000ul // Platform-Level Interrupt Controller (PLIC)
#define PLIC_SIZE 0x00400000ul // Address Range of PLIC
#define CLINT_BASE_ADDR 0x02000000ul // CLINT is Unused
#define CLINT_SIZE 0x000c0000ul // CLINT is Unused
...
#define PLIC_HART_BASE 0x201000 // Hart 0 S-Mode Priority Threshold in PLIC
#define PLIC_HART_SIZE 0x1000 // Address Range of Hart 0 PLIC
This is how we derived the above RISC-V Addresses…
// 64 KB of Boot Code at Address 0x0
#define LOW_RAM_SIZE 0x00010000ul
Low RAM: This setting is specfic to TinyEMU, we left it unchanged. The Low RAM contains…
Address 0x1000: TinyEMU Boot Code
(Why not 0x0?)
Address 0x1040: Binary Device Tree
(NuttX doesn’t need the Device Tree)
// Our Kernel boots here
#define RAM_BASE_ADDR 0x50200000ul
RAM Base: NuttX (also Linux) boots at the above RAM Address (because of U-Boot Bootloader), as explained here…
// Platform-Level Interrupt Controller (PLIC)
// and Address Range of PLIC
#define PLIC_BASE_ADDR 0xe0000000ul
#define PLIC_SIZE 0x00400000ul
Platform-Level Interrupt Controller (PLIC): We documented the PLIC Addresses here…
// Hart 0 S-Mode Priority Threshold in PLIC
// and Address Range of Hart 0 PLIC
#define PLIC_HART_BASE 0x201000
#define PLIC_HART_SIZE 0x1000
PLIC Hart: We specify Hart 0, Supervisor Mode as explained here…
// Core-Local Interrupt Controller (CLINT) is Unused
#define CLINT_BASE_ADDR 0x02000000ul
#define CLINT_SIZE 0x000c0000ul
Core-Local Interrupt Controller (CLINT) is unused. We left the setting unchanged.