RISC-V Ox64 BL808 SBC: Starting Apache NuttX Real-Time Operating System

📝 12 Nov 2023

Booting Apache NuttX RTOS on Pine64 Ox64 64-bit RISC-V SBC (Bouffalo Lab BL808)

Last week we booted Linux on the Pine64 Ox64 64-bit RISC-V SBC (pic below), powered by Bouffalo Lab BL808 SoC…

And we wondered if a tiny 64-bit RTOS (Real-Time Operating System) like Apache NuttX RTOS might run more efficiently on Ox64.

(With only 64 MB of RAM)

Let’s make it happen! In this article we…

Pine64 Ox64 64-bit RISC-V SBC (Bouffalo Lab BL808)

§1 Begin with Star64 NuttX

We’re booting Star64 NuttX on Ox64? Unmodified?!

Yeah it feels like we’re Shredding a Toaster inside a Blender (with plenty of Smoke and Noise)…

But we’re starting with NuttX for Star64 JH7110 anyway! That’s because we have a very strong hunch (or just plainly stubborn) that NuttX will boot well across RISC-V SoCs.

(We ported NuttX QEMU to Star64 in only a few weeks!)

But Star64 runs on SiFive Cores. Ox64 uses T-Head Cores!

If RISC-V ain’t RISC-V on SiFive vs T-Head: We’ll find out!

This is how we download and build NuttX for Star64 JH7110 RISC-V SBC…

## Download WIP NuttX Source Code
git clone \
  --branch ox64 \
  https://github.com/lupyuen2/wip-nuttx \
  nuttx
git clone \
  --branch ox64 \
  https://github.com/lupyuen2/wip-nuttx-apps \
  apps

## Build NuttX for Star64
cd nuttx
tools/configure.sh star64:nsh
make

## Dump the RISC-V Disassembly for NuttX Kernel
riscv64-unknown-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  nuttx \
  >nuttx.S \
  2>&1

(Remember to install the Build Prerequisites and Toolchain)

(And enable Scheduler Info Output)

Next we prepare a Linux microSD for Ox64 as described in the previous article.

(Remember to flash OpenSBI and U-Boot Bootloader)

Then we do the Linux-To-NuttX Switcheroo: Overwrite the microSD Linux Image by the NuttX Kernel…

## Export the NuttX Kernel
## to `nuttx.bin`
riscv64-unknown-elf-objcopy \
  -O binary \
  nuttx \
  nuttx.bin

## Overwrite the Linux Image
## on Ox64 microSD
cp nuttx.bin \
  "/Volumes/NO NAME/Image"
diskutil unmountDisk /dev/disk2

Insert the microSD into Ox64 and power up Ox64.

Ox64 boots OpenSBI, which starts U-Boot Bootloader, which starts NuttX Kernel.

And we see… Absolutely Nothing!

Retrieving file: /extlinux/../Image
  append: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
Retrieving file: /extlinux/../bl808-pine64-ox64.dtb
  Flattened Device Tree blob at 51ff8000
  Booting using the fdt blob at 0x51ff8000
  Working FDT set to 51ff8000
  Loading Device Tree to 0000000053f22000, end 0000000053f25fab ... OK
  Working FDT set to 53f22000
Starting kernel...

(See the Complete Log)

Shouldn’t we see a Crash Dump?

Yeah we’re hoping that NuttX would crash and OpenSBI (Supervisor Binary Interface) could dump a meaningful Stack Trace. But nope!

Is NuttX alive? We can check…

Apache NuttX RTOS boots a tiny bit on Pine64 Ox64 64-bit RISC-V SBC (Bouffalo Lab BL808)

§2 Print to Serial Console

We have a strong hunch that NuttX is actually booting on Ox64… How to prove it?

We’ll print something in the NuttX Boot Code. Which is in RISC-V Assembly!

Ox64’s BL808 UART looks super familiar. When we compare these UARTs…

We discover that BL808 UART works the same way as BL602!

Thus we seek guidance from the NuttX Driver for BL602 UART.

Thanks! But how do we print to BL808 UART?

BL602 UART Driver prints to the Serial Console like so: bl602_serial.c

// Output FIFO Offset is 0x88
#define BL602_UART_FIFO_WDATA_OFFSET 0x000088
#define BL602_UART_FIFO_WDATA(n) (BL602_UART_BASE(n) + BL602_UART_FIFO_WDATA_OFFSET)

// Write a character to UART
void bl602_send(struct uart_dev_s *dev, int ch) {
  ...
  // Wait for FIFO to be empty
  while ((getreg32(BL602_UART_FIFO_CONFIG_1(uart_idx)) & \
    UART_FIFO_CONFIG_1_TX_CNT_MASK) == 0);

  // Write character to Output FIFO
  putreg32(ch, BL602_UART_FIFO_WDATA(uart_idx));
}

For BL808: We do the same. We simply write the character to…

Based on our Star64 Debug Code, we write this in RISC-V Assembly to print “123”…

/* Load UART3 Base Address to Register t0 */
li  t0, 0x30002000

/* Load `1` to Register t1 */
li  t1, 0x31
/* Store byte from Register t1 to UART3 Base Address, Offset 0x88 */
sb  t1, 0x88(t0)

/* Load `2` to Register t1 */
li  t1, 0x32
/* Store byte from Register t1 to UART3 Base Address, Offset 0x88 */
sb  t1, 0x88(t0)

/* Load `3` to Register t1 */
li  t1, 0x33
/* Store byte from Register t1 to UART3 Base Address, Offset 0x88 */
sb  t1, 0x88(t0)

(li loads a Value into a Register)

(sb stores a byte from a Register into an Address Offset)

We insert the code above into our NuttX Boot Code: jh7110_head.S

And we see (pic above)…

Starting kernel...
123

Our hunch is 100% correct, NuttX is ALIVE on Ox64 yay!

(See the Complete Log)

(Boot Code has moved here)

Anything else we changed in the NuttX Boot Code?

OpenSBI boots on Ox64 with Hart ID 0 (instead of 1). Which means we remove this adjustment for Hart ID: jh7110_head.S

/* We assume that OpenSBI has passed Hart ID (value 1) in Register a0.
 * But NuttX expects Hart ID to start at 0, so we subtract 1.
 * Previously: addi a0, a0, -1 */

§3 Update the Boot Address

Surely Ox64 boots at a different RAM Address from Star64?

Yep! Next we fix the NuttX Boot Address for Ox64.

From the U-Boot Bootloader we see that Ox64 boots Linux at this address…

$ printenv
kernel_addr_r=0x50200000

Based on the Boot Address, we define these Memory Regions for NuttX…

Memory RegionStart AddressSize
I/O Region0x0000 00000x5000 0000
Kernel Code0x5020 00002 MB
Kernel Data0x5040 00002 MB
Page Pool0x5060 00004 MB
RAM Disk0x5060 000016 MB

(Page Pool will be used by NuttX Apps)

(RAM Disk will contain the NuttX Shell and Apps)

We update the Memory Regions in the NuttX Linker Script: ld.script

MEMORY
{
  kflash (rx) :   ORIGIN = 0x50200000, LENGTH = 2048K /* w/ cache */
  ksram (rwx) :   ORIGIN = 0x50400000, LENGTH = 2048K /* w/ cache */
  pgram (rwx) :   ORIGIN = 0x50600000, LENGTH = 4096K /* w/ cache */
  ramdisk (rwx) : ORIGIN = 0x50A00000, LENGTH = 16M   /* w/ cache */
} /* TODO: Use up the entire 64 MB RAM */

(Moved here)

We make the same changes to the NuttX Build Configuration: nsh/defconfig

CONFIG_RAM_START=0x50200000
CONFIG_RAM_SIZE=1048576
CONFIG_ARCH_PGPOOL_PBASE=0x50600000
CONFIG_ARCH_PGPOOL_VBASE=0x50600000
CONFIG_ARCH_PGPOOL_SIZE=4194304

(Moved here)

And we update the NuttX Memory Map: jh7110_mm_init.c

// Map the whole I/O Memory
// with Virtual Address = Physical Address
// TODO: Interrupt Controller is missing!
#define MMU_IO_BASE (0x00000000ul)
#define MMU_IO_SIZE (0x50000000ul)

(Moved here)

What’s this Memory Map?

Inside the BL808 SoC is the Sv39 Memory Management Unit (MMU). (Same for Star64 JH7110)

The MMU maps Virtual Memory Addresses to Physical Memory Addresses. And stops the NuttX Kernel from accessing Invalid Addresses.

At startup, NuttX configures the MMU with the Memory Map, the Range of Memory Addresses that the NuttX Kernel is allowed to access.

The code above says that NuttX is allowed to access any address from 0x0000 0000 to 0x5000 0000. (Because of Memory-Mapped I/O)

Time to make NuttX talk…

(More about Memory Map)

NuttX prints our very first Stack Dump on Ox64 yay!

§4 Fix the UART Driver

NuttX on Ox64 has been awfully quiet…

How to fix the UART Driver so that NuttX can print things?

NuttX is still running the JH7110 UART Driver (16550).

To print to the Ox64 Serial Console, we make a quick patch to the NuttX UART Driver.

For now, we hardcode the UART3 Base Address (from above) and Output FIFO Offset: uart_16550.c

// Write one character to the UART
void u16550_putc(FAR struct u16550_s *priv, int ch) {

  // Hardcode the UART3 Base Address and Output FIFO Offset
  *(volatile uint8_t *) 0x30002088 = ch;

  // Previously:
  // while ((u16550_serialin(priv, UART_LSR_OFFSET) & UART_LSR_THRE) == 0);
  // u16550_serialout(priv, UART_THR_OFFSET, (uart_datawidth_t)ch);
}

(Moved here)

(Yeah the UART Buffer might overflow, we’ll fix later)

For Other UART Registers: We skip the reading and writing of the registers, because we’ll patch them later: uart_16550.c

// Read from UART Register
uart_datawidth_t u16550_serialin(FAR struct u16550_s *priv, int offset) {
  return 0;
  // Commented out the rest
}

// Write to UART Register
void u16550_serialout(FAR struct u16550_s *priv, int offset, uart_datawidth_t value) {
  // Commented out the rest
}

And we won’t wait for UART Ready, since we don’t access the Line Control Register: uart_16550.c

// Wait until UART is not busy. This is needed before writing to Line Control Register.
// Otherwise we will get spurious interrupts on Synopsys DesignWare 8250.
int u16550_wait(FAR struct u16550_s *priv) {
  // Nopez! No waiting for now
  return OK;
}

After these fixes, NuttX prints our very first Crash Dump on Ox64 yay! (Pic above)

Starting kernel...
123ABC
riscv_exception: 
  EXCEPTION: Load access fault
  MCAUSE: 5
  EPC:    50208086
  MTVAL:  0c002104
riscv_exception: PANIC!!! Exception = 0000000000000005
_assert: Current Version: NuttX  12.0.3 93a92a7-dirty Nov  5 2023 11:27:46 risc-v
_assert: Assertion failed panic: at file: common/riscv_exception.c:85 task: Idle_Task process: Kernel 0x50200e28
up_dump_register: EPC: 0000000050208086
up_dump_register: A0: 000000000c002104 A1: ffffffffffffffff A2: 0000000000000001 A3: 0000000000000003

(See the Complete Log)

MTVAL (Machine Trap Value) says that NuttX has crashed while reading the Invalid Data Address 0x0C00 2104. (Hence the “Load Access Fault”)

Why is Data Address 0x0C00 2104 causing unhappiness? First we learn about RISC-V Interrupts…

Platform-Level Interrupt Controller for Star64 JH7110

Platform-Level Interrupt Controller for Star64 JH7110

§5 Platform-Level Interrupt Controller

What’s this Platform-Level Interrupt Controller?

Inside our BL808 SoC, the Platform-Level Interrupt Controller (PLIC) is the hardware that receives External Interrupts and forwards them to our RISC-V CPU.

(Like for UART Interrupts, pic above)

Earlier we saw NuttX crashing with this RISC-V Exception…

EXCEPTION: Load access fault
MCAUSE: 5
EPC:    50208086
MTVAL:  0c002104

This says that NuttX crashed when it tried to access Invalid Data Address 0x0C00 2104 from Code Address 0x5020 8086.

We look up Code Address 0x5020 8086 in our RISC-V Disassembly for NuttX Kernel…

nuttx/arch/risc-v/src/common/riscv_modifyreg32.c:52
  regval  = getreg32(addr);
    50208086: 2701  sext.w a4,a4

Which points to this: riscv_modifyreg32.c

// Atomically modify the specified bits
// in a Memory-Mapped Register
void modifyreg32(uintptr_t addr, uint32_t clearbits, uint32_t setbits) {
  irqstate_t flags = spin_lock_irqsave(NULL);
  // Crashes here because `addr` is invalid...
  uint32_t regval = getreg32(addr);
  regval &= ~clearbits;
  regval |= setbits;
  putreg32(regval, addr);
  spin_unlock_irqrestore(NULL, flags);
}

Hence NuttX tried to modify a Memory-Mapped Register that doesn’t exist, and crashed.

But what Memory-Mapped Register?

The offending Data Address 0x0C00 2104 actually comes from the Star64 PLIC! (Platform-Level Interrupt Controller)

// Star64 PLIC Base Address
// From https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/hardware/jh7110_memorymap.h#L30
#define JH7110_PLIC_BASE 0x0c000000

// Star64 S-Mode Interrupt Enable
// From https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/hardware/jh7110_plic.h#L34-L49
#define JH7110_PLIC_ENABLE2 (JH7110_PLIC_BASE + 0x002104)

PLIC for Ox64 is in a different place, let’s change it.

What’s the PLIC Base Address for Ox64?

For Ox64, PLIC Base Address is 0xE000 0000, according to the Linux Device Tree: bl808-pine64-ox64.dts

interrupt-controller@e0000000 {
  compatible = "thead,c900-plic";
  reg = <0xe0000000 0x4000000>;
  interrupts-extended = <0x06 0xffffffff 0x06 0x09>;
  interrupt-controller;
  #address-cells = <0x00>;
  #interrupt-cells = <0x02>;
  riscv,ndev = <0x40>;
  phandle = <0x01>;
};

Based on the above, we change the PLIC Base Address for Ox64: jh7110_memorymap.h

#define JH7110_PLIC_BASE 0xe0000000ul

(Moved here)

(PLIC Offsets are in XuanTie OpenC906 User Manual, Page 77)

NuttX now crashes at a different place, with IRQ 15 (pic below)…

123ABC
nx_start: Entry
up_irqinitialize: a, b, c
riscv_dispatch_irq: irq=15
irq_unexpected_isr: ERROR irq: 15
_assert: Current Version: NuttX  12.0.3 910bfca-dirty Nov  6 2023 15:23:11 risc-v
_assert: Assertion failed panic: at file: irq/irq_unexpectedisr.c:54 task: Idle_Task process: Kernel 0x50200e50

(See the Complete Log)

But there’s something exceptional about IRQ 15…

NuttX crashes with IRQ 15

§6 Handle RISC-V Exceptions

What is IRQ 15? Who’s causing it? (Pic above)

From the XuanTie OpenC906 User Manual (Page 21)…

“Exception Vector ID 15: A Store / Atomic Instruction page error exception”

This says that NuttX tried to write to an Invalid Data Address.

And it failed due to an “Unexpected Interrupt”.

Something special about IRQ 15?

IRQ 15 is actually a RISC-V Exception!

Rightfully, NuttX should print a helpful RISC-V Exception Crash Dump with the offending Data Address. (Like this)

But NuttX wasn’t terribly helpful for this RISC-V Exception. Very odd!

Where did it crash?

Based on our Debug Log, NuttX crashes just before setting the PLIC: jh7110_irq.c

// Init the Interrupts
void up_irqinitialize(void) {
  ...
  // Disable S-Mode Interrupts
  _info("b\n");
  up_irq_save();

  // Disable all Global Interrupts
  _info("c\n");
  // Crashes here!
  putreg32(0x0, JH7110_PLIC_ENABLE1);
  putreg32(0x0, JH7110_PLIC_ENABLE2);
  ...
  // Attach the RISC-V Exception Handlers
  _info("f\n");
  riscv_exception_attach();

(Moved here)

Something doesn’t look right…

Yeah in the code above, we attach the RISC-V Exception Handlers (riscv_exception_attach)…

After the code has crashed! (putreg32)

Hence we attach the Exception Handlers earlier: jh7110_irq.c

// Init the Interrupts
void up_irqinitialize(void) {
  ...
  // Disable S-Mode Interrupts
  _info("b\n");
  up_irq_save();

  // Moved Here: Attach the RISC-V Exception Handlers
  _info("f\n");
  riscv_exception_attach();

  // Disable all Global Interrupts
  _info("c\n");
  // Crashes here!
  putreg32(0x0, JH7110_PLIC_ENABLE1);
  putreg32(0x0, JH7110_PLIC_ENABLE2);

(Moved here)

Then riscv_exception_attach will handle all RISC-V Exceptions correctly, including IRQ 15: riscv_exception.c

// Attach the RISC-V Exception Handlers
void riscv_exception_attach(void) {
  ...
  // IRQ 15: Store / AMO Page Fault
  irq_attach(RISCV_IRQ_STOREPF, riscv_exception, NULL);

Does it work?

Yep we see the Store / AMO Page Fault Exception! (Pic below)

up_irqinitialize: c
riscv_dispatch_irq: irq=15
riscv_exception: 
EXCEPTION: Store/AMO page fault
MCAUSE: f
EPC:    50207e6a
MTVAL:  e0002100

(See the Complete Log)

When we look up the NuttX Kernel Disassembly, the Exception Code Address 0x5020 7E6A (EPC) comes from our PLIC Code…

nuttx/arch/risc-v/src/chip/jh7110_irq.c:62
  putreg32(0x0, JH7110_PLIC_ENABLE1);
    50207e64: 700017b7  lui  a5,0x70001
    50207e68: 0786      slli a5,a5,0x1
    50207e6a: 1007a023  sw   zero,256(a5) # 70001100 <__ramdisk_end+0x1e601100>

(Moved here)

The offending Data Address (MTVAL) is 0xE000 2100.

Which is our Ox64 PLIC! We scrutinise PLIC again…

Store / AMO Page Fault Exception

§7 Add PLIC to Memory Map

But is 0xE000 2100 accessible?

Ah we forgot to add the Platform-Level Interrupt Controller (PLIC) to the Memory Map. This is how we fix it: jh7110_mm_init.c

// Map the whole I/O Memory
// with Virtual Address = Physical Address
// (Includes PLIC)
// TODO: This is mapping too much memory!
#define MMU_IO_BASE (0x00000000ul)
#define MMU_IO_SIZE (0xf0000000ul)

(Moved here)

(Memory Map doesn’t look right)

NuttX boots even further. And tries to register IRQ 57 for the Star64 UART Interrupt…

up_irqinitialize: c, d, e, g
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
irq_attach: irq=57
up_enable_irq: irq=57
riscv_exception: 
EXCEPTION: Load access fault
MCAUSE: 5
EPC:    50208342
MTVAL:  e0002104

(See the Complete Log)

But it crashes while accessing the PLIC at another Invalid Data Address: 0xE000 2104. (Sigh)

Ack! Enough with the PLIC already…

Yeah we’ll fix PLIC later. The entire UART Driver will be revamped anyway, including the UART Interrupt.

For now, we disable the UART Interrupt: uart_16550.c

// Attach the UART Interrupt for Star64
int u16550_attach(struct uart_dev_s *dev) {
  // Don't attach the interrupt
  // Previously:
  // ret = irq_attach(priv->irq, u16550_interrupt, dev);

  // Don't enable the interrupt
  // Previously:
  // up_enable_irq(priv->irq);

(Moved here)

NuttX hits another roadblock…

Initial RAM Disk for Star64 JH7110

Initial RAM Disk for Star64 JH7110

§8 Initial RAM Disk is Missing

We disabled the UART Interrupts. What happens now?

NuttX boots much further, but crashes in the NuttX Bringup…

up_irqinitialize: c, d, e, g
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
_assert: Current Version: NuttX  12.0.3 b244f85-dirty Nov  6 2023 17:35:34 risc-v
_assert: Assertion failed ret >= 0: at file: init/nx_bringup.c:283 task: AppBringUp process: Kernel 0x5020107e

(See the Complete Log)

That’s because NuttX couldn’t mount the Initial RAM Disk: nx_bringup.c

// Mount the File System containing
// the NuttX Shell (NSH)
ret = nx_mount(CONFIG_INIT_MOUNT_SOURCE, CONFIG_INIT_MOUNT_TARGET,
  CONFIG_INIT_MOUNT_FSTYPE, CONFIG_INIT_MOUNT_FLAGS,
  CONFIG_INIT_MOUNT_DATA);

// Fails here
DEBUGASSERT(ret >= 0);

That contains the Executable Binaries for NuttX Shell (NSH) and the NuttX Apps.

(More about Initial RAM Disk)

Why is the Initial RAM Disk missing?

That’s because we haven’t loaded the Initial RAM Disk into RAM!

We’ll modify the NuttX Kernel Image (or U-Boot Script) on the microSD Card, so that U-Boot Bootloader will load our Initial RAM Disk before starting NuttX…

Are we done yet?

That’s all for today! NuttX has booted so much code on Ox64. Here’s the flow of the NuttX Code that boots on Ox64 (pic below)…

NuttX Boot Flow for Ox64 BL808

Clickable Version of NuttX Boot Flow

§9 What’s Next

This week we made plenty of progress starting Apache NuttX RTOS on the tiny Ox64 BL808 RISC-V SBC…

We’ll do much more for NuttX on Ox64 BL808, stay tuned for updates!

Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/ox2.md

My soldering of Ox64 BL808 looks horrigible… But it boots NuttX!

My soldering of Ox64 BL808 looks horrigible… But it boots NuttX!

§10 Appendix: NuttX Boot Flow

What happens exactly when NuttX boots on Ox64?

In this article, NuttX has booted plenty of code on Ox64. Here’s the flow of the NuttX Code that boots on Ox64…

NuttX Boot Flow for Ox64 BL808

Clickable Version of NuttX Boot Flow

NuttX Boot Code: bl808_head calls…

Start Supervisor Mode: bl808_start_s prints “ABC” and calls…

Early Serial Init: riscv_earlyserialinit calls…

Memory Mgmt Init: bl808_mm_init inits the Memory Mgmt Unit by calling…

Start NuttX: nx_start does many things and calls…

IRQ Init: up_irqinitialize calls…

Init NuttX: up_initialize calls…

Bringup NuttX: nx_bringup calls…

Therefore we expect NuttX to boot completely on Ox64 when we’ve implemented…

How did we figure out the NuttX Boot Flow?

It’s awfully tricky to follow the Boot Flow by reading the NuttX Source Code. (So many C Macros!)

Instead we searched for Function Names in the NuttX Disassembly. Which has the C Macros completely expanded for us.

§11 Appendix: Memory Map for Ox64

Read the article…

What’s this Memory Map?

// Map the whole I/O Memory
// with Virtual Address = Physical Address
// TODO: Interrupt Controller is missing!
#define MMU_IO_BASE (0x00000000ul)
#define MMU_IO_SIZE (0x50000000ul)

(Source)

(Moved here)

Inside the BL808 SoC is the Sv39 Memory Management Unit (MMU) with 128 / 256 / 512 TLB table entries. (Same for Star64 JH7110)

The MMU maps Virtual Memory Addresses to Physical Memory Addresses. And stops the NuttX Kernel from accessing Invalid Addresses.

At startup, NuttX configures the MMU with the Memory Map, the Range of Memory Addresses that the NuttX Kernel is allowed to access.

The code above says that NuttX is allowed to access any address from 0x0000 0000 to 0x5000 0000. (Because of Memory-Mapped I/O)

(MMU appears in OpenC906 User Manual, Page 50)

But we forgot to add the PLIC to the Memory Map!

The Platform-Level Interrupt Controller (PLIC) is at 0xE000 0000.

Let’s add the PLIC to the Memory Map: jh7110_mm_init.c

// Map the whole I/O Memory
// with Virtual Address = Physical Address
// (Includes PLIC)
// TODO: This is mapping too much memory!
#define MMU_IO_BASE (0x00000000ul)
#define MMU_IO_SIZE (0xf0000000ul)

(Moved here)

This doesn’t look right…

Yeah when we substitute the above MMU_IO_BASE and MMU_IO_SIZE into the Memory Map: jh7110_mm_init.c

// Set up the Kernel MMU Memory Map
void jh7110_kernel_mappings(void) {
  ...
  // Map I/O Region, use enough large page tables for the I/O region
  // MMU_IO_BASE is 0x00000000
  // MMU_IO_SIZE is 0xf0000000
  mmu_ln_map_region(1, PGT_L1_VBASE, MMU_IO_BASE, MMU_IO_BASE, MMU_IO_SIZE, MMU_IO_FLAGS);

  // Map the Kernel Code for L2/L3
  // From https://github.com/lupyuen2/wip-nuttx/blob/ox64/boards/risc-v/jh7110/star64/scripts/ld.script#L20-L27
  // KFLASH_START is 0x50200000
  // KFLASH_SIZE  is 2 MB
  map_region(KFLASH_START, KFLASH_START, KFLASH_SIZE, MMU_KTEXT_FLAGS);

  // Map the Kernel Data for L2/L3
  // From https://github.com/lupyuen2/wip-nuttx/blob/ox64/boards/risc-v/jh7110/star64/scripts/ld.script#L20-L27
  // KSRAM_START is 0x50400000
  // KSRAM_SIZE  is 2 MB
  map_region(KSRAM_START, KSRAM_START, KSRAM_SIZE, MMU_KDATA_FLAGS);

  // Connect the L1 and L2 page tables for the kernel text and data
  mmu_ln_setentry(1, PGT_L1_VBASE, PGT_L2_PBASE, KFLASH_START, PTE_G);

  // Map the Page Pool for NuttX Apps
  // From https://github.com/lupyuen2/wip-nuttx/blob/ox64/boards/risc-v/jh7110/star64/scripts/ld.script#L20-L27
  // PGPOOL_START is 0x50600000
  // PGPOOL_SIZE  is 4 MB + 16 MB (including RAM Disk)
  mmu_ln_map_region(2, PGT_L2_VBASE, PGPOOL_START, PGPOOL_START, PGPOOL_SIZE, MMU_KDATA_FLAGS);
}

(Moved here)

We see a problem with the Memory Map…

Memory RegionStart AddressSize
I/O Region0x0000 00000xF000 0000
Kernel Code0x5020 00002 MB
Kernel Data0x5040 00002 MB
Page Pool0x5060 000020 MB

(Page Pool includes RAM Disk)

The I/O Region overlaps with the Kernel Code, Data and Page Pool!

This happens because the PLIC is located at 0xE000 0000. Which is AFTER the RAM Region…

Memory RegionStart AddressSize
I/O Region0x0000 00000x5000 0000
RAM0x5000 000064 MB
Apps0xC000 0000(See below)
PLIC0xE000 0000???

Also NuttX Apps will fail because they run in the (Virtual) User Address Space at 0xC000 0000: nsh/defconfig

CONFIG_ARCH_TEXT_VBASE=0xC0000000
CONFIG_ARCH_TEXT_NPAGES=128
CONFIG_ARCH_DATA_VBASE=0xC0100000
CONFIG_ARCH_DATA_NPAGES=128
CONFIG_ARCH_HEAP_VBASE=0xC0200000
CONFIG_ARCH_HEAP_NPAGES=128

(Moved here)

But our Kernel Memory Space already extends to 0xF000 0000!

Thus we might introduce another Memory Region, just to map the PLIC.

(Or should we move the User Address Space to 0xF000 0000? Which gives us max 256 MB for NuttX Apps?)

The OpenSBI Log might offer some hints on the Memory Map…

Firmware Base       : 0x3ef80000
Firmware Size       : 200 KB
Domain0 Region00    : 0xe4008000-0xe400bfff (I)
Domain0 Region01    : 0xe4000000-0xe4007fff (I)
Domain0 Region02    : 0x3ef80000-0x3efbffff ()
Domain0 Region03    : 0x00000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address: 0x50000000
Domain0 Next Arg1   : 0x51ff8000

(0x3EF8 0000 is probably protected because it contains the OpenSBI Firmware. What is “(I)”?)

Here’s how we fixed the Memory Map for Ox64 NuttX…

§12 Appendix: UART Driver for Ox64

Read the article…

How will we create the NuttX UART Driver for Ox64 BL808?

Today NuttX supports the 32-bit predecessor of BL808: Bouffalo Lab BL602.

When we compare these UARTs…

We discover that BL808 UART works the same way as BL602!

Thus we’ll simply copy the NuttX Driver for BL602 UART to Ox64.

UART Interrupts are mandatory: If UART Interrupts aren’t implemented, NuttX Shell (NSH) and NuttX Apps won’t print anything.

BL602 UART Driver has just been ported to Ox64! (Minus the UART Interrupts) Check our progress here…

What about other drivers: BL808 vs BL602?

The controllers below look highly similar on BL808 vs BL602. Which means we have plenty of NuttX Drivers to copy from BL602 to BL808!

ControllerBL808 RMBL602 RM
I2CPage 430Page 142
SPIPage 387Page 115
ADCPage 169Page 45
DACPage 180Page 66
DMAPage 187Page 70
InfraredPage 372Page 100
PWMPage 447Page 157
TimerPage 474Page 174

Our earlier experiments with BL602 NuttX proved that the drivers above work well. So we’re all set for BL808!

(BL602 NuttX is tested on Real Hardware every day)

(Still going strong!)

What about the drivers missing from BL602 NuttX?

We’ll port the missing BL808 Drivers from Bouffalo Lab’s BouffaloSDK to NuttX.

(BouffaloSDK is Apache 2.0 Licensed)

Initial RAM Disk for Star64 JH7110

Initial RAM Disk for Star64 JH7110

§13 Appendix: Initial RAM Disk

Read the article…

What’s this Initial RAM Disk?

The Initial RAM Disk contains the Executable Binaries for NuttX Shell (NSH) and NuttX Apps.

At startup, NuttX loads the Initial RAM Disk into RAM and mounts the File System, so that the NuttX Shell (and NuttX Apps) can be started later.

(More about Initial RAM Disk)

Why is the Initial RAM Disk missing from Ox64?

That’s because we haven’t loaded the Initial RAM Disk into RAM!

Two ways we can load the Initial RAM Disk…

  1. Load the Initial RAM Disk from a Separate File: initrd (similar to Star64)

    This means we need to modify the U-Boot Script: boot-pine64.scr

    And make it load the initrd file into RAM.

    (Which is good for separating the NuttX Kernel and NuttX Apps)

    OR…

  2. Append the Initial RAM Disk to the NuttX Kernel Image

    So the U-Boot Bootloader will load (one-shot into RAM) the NuttX Kernel + Initial RAM Disk.

    And we reuse the existing U-Boot Config on the microSD Card: extlinux/extlinux.conf

    (Which might be more efficient for our Limited RAM)

    (See the U-Boot Boot Flow)

    TODO: Can we mount the File System directly from the NuttX Kernel Image in RAM? Without copying to the RAM Disk Memory Region?

We’ll probably adopt the Second Method, since we are low on RAM. Like this…

## Export the NuttX Kernel to `nuttx.bin`
riscv64-unknown-elf-objcopy \
  -O binary \
  nuttx \
  nuttx.bin

## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero >/tmp/nuttx.pad

## Append Padding and Initial RAM Disk to NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
  >Image

## Overwrite the Linux Image on Ox64 microSD
cp Image "/Volumes/NO NAME/"

## U-Boot Bootloader will load NuttX Kernel and
## Initial RAM Disk into RAM

Here’s how we implemented the Initial RAM Disk…