RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk

📝 26 Nov 2023

NuttX App makes a System Call to NuttX Kernel

In Asia the wise folks say…

“One can hide on a certain day but cannot hide for a long time”

“躲过初一，躲不过十五”

In other words…

“Transformers? More than meets the eye!”

In this article, we go behind the shadow puppetry (wayang kulit) and deceptive simplicity of NuttX Applications inside Apache NuttX RTOS (Real-Time Operating System) for Pine64 Ox64 BL808 64-bit RISC-V SBC (pic below)…

Pine64 Ox64 64-bit RISC-V SBC (Bouffalo Lab BL808)

§1 Inside a NuttX App

What happens inside the simplest NuttX App?

// From https://github.com/apache/nuttx-apps/blob/master/examples/hello/hello_main.c#L36-L40
int main(int argc, FAR char *argv[]) {
  printf("Hello, World!!\n");
  return 0;
}

Let’s find out! First we build NuttX for Ox64 BL808 SBC.

Which produces this ELF Executable for our NuttX App…

## ELF Executable for `hello` looks big...
$ ls -l ../apps/bin/hello
-rwxr-xr-x  518,192  ../apps/bin/hello

## Though not much inside, mostly Debug Info...
$ riscv64-unknown-elf-size ../apps/bin/hello
   text  data  bss   dec  hex  filename
   3814     8    4  3826  ef2  ../apps/bin/hello

## Dump the RISC-V Disassembly to `hello.S`
$ riscv64-unknown-elf-objdump \
  --syms --source --reloc --demangle --line-numbers --wide \
  --debugging \
  ../apps/bin/hello \
  >hello.S \
  2>&1

(See the Build Outputs)

Here’s the RISC-V Disassembly of our NuttX App: hello.S

## Omitted: _start() prepares for signals (sig_trampoline) and calls main()

003e <main>:
int main(int argc, FAR char *argv[]) {
  3e: 1141      addi   sp,sp,-16  ## Subtract 16 from Stack Pointer

## Set Register A0 (Arg 0) to "Hello, World!!\n"
  40: 00000517  auipc  a0,0x0    40: R_RISCV_PCREL_HI20    .LC0
  44: 00050513  mv     a0,a0     44: R_RISCV_PCREL_LO12_I  .L0 

printf("Hello, World!!\n");
  48: e406      sd     ra,8(sp)  ## Save Return Address to Stack Pointer, Offset 8
  4a: 00000097  auipc  ra,0x0    4a: R_RISCV_CALL  puts
  4e: 000080e7  jalr   ra      # 4a <.LVL1+0x2>  ## Call puts()

return 0;
  52: 60a2      ld    ra,8(sp)  ## Load Return Address from Stack Pointer, Offset 8
  54: 4501      li    a0,0      ## Set Return Value to 0
  56: 0141      addi  sp,sp,16  ## Add 16 to Stack Pointer
  58: 8082      ret             ## Return to caller: _start()

## Followed by the code for puts(), lib_fwrite_unlocked(), write(), ...

In the RISC-V Disassembly, we see that main calls…

How will write call the NuttX Kernel? We’ll see soon!

This code looks broken…

printf("Hello, World!!\n");

  ## Load Register RA with Program Counter + 0x0
  4a: 00000097  auipc  ra, 0x0

  ## Call the function in Register RA: puts()
  4e: 000080e7  jalr   ra

We break it down…

Shouldn’t auipc add the Offset of puts?

Ah that’s because we’re looking at Relocatable Code!

The auipc Offset will be fixed up by the NuttX ELF Loader when it loads our NuttX App for execution.

In our RISC-V Disassembly, the Relocation Info shows that 0x0 will be replaced by the Offset of puts…

printf("Hello, World!!\n");

  ## Why load Register RA with Program Counter + 0x0?
  ## Gotcha! 0x0 will be changed to the Offset of puts()
  4a: 00000097  auipc  ra, 0x0  
  4a: R_RISCV_CALL     puts

  ## Call the function in Register RA: puts()
  ## Which will work when ELF Loader fixes the Offset of puts()
  4e: 000080e7  jalr   ra     # 4a <.LVL1+0x2>

Therefore we’re all good! (Eventually)

Why puts instead of printf?

The GCC Compiler has cleverly optimised away printf to become puts.

If we do this (and foil the GCC Compiler)…

// Nope, GCC Compiler won't change printf() to puts()
printf(
  "Hello, World %s!!\n",  // Meaningful Format String
  "Luppy"                 // Makes it complicated
);

Then printf will appear in our RISC-V Disassembly.

We circle back to write…

NuttX App calls NuttX Kernel

§2 NuttX App calls NuttX Kernel

Our app will print something to the console…

But NuttX Apps can’t write directly to the Serial Device right?

Nope!

That’s why “write” should trigger a System Call to the NuttX Kernel, jumping from RISC-V User Mode to Supervisor Mode.

(And write to the Serial Device, pic above)

Will NuttX Apps need Special Coding to make System Calls?

Not at all! The System Call is totally transparent to our app…

What’s this “forwarding” to a System Call?

This forwarding happens inside a Proxy Function that’s auto-generated during NuttX Build…

// From nuttx/syscall/proxies/PROXY_write.c
// Auto-Generated Proxy for `write`
// Looks like the Kernel `write`, though it's actually a System Call
ssize_t write(int parm1, FAR const void * parm2, size_t parm3) {

  // Make a System Call with 3 parameters...
  return (ssize_t) sys_call3(
    (unsigned int) SYS_write,  // System Call Number (63 = `write`)
    (uintptr_t) parm1,         // File Descriptor (1 = Standard Output)
    (uintptr_t) parm2,         // Buffer to be written
    (uintptr_t) parm3          // Number of bytes to write
  );
}

Our NuttX App (implicitly) calls this Proxy Version of “write” (that pretends to be the Kernel “write”)…

// Our App calls the Proxy Function...
int ret = write(
  1,                   // File Descriptor (1 = Standard Output)
  "Hello, World!!\n",  // Buffer to be written
  15                   // Number of bytes to write
);

Which triggers a System Call to the Kernel.

(Indeed “More than meets the eye!”)

What’s sys_call3?

It makes a System Call (to NuttX Kernel) with 3 Parameters: syscall.h

// Make a System Call with 3 parameters
uintptr_t sys_call3(
  unsigned int nbr,  // System Call Number (63 = `write`)
  uintptr_t parm1,   // First Parameter
  uintptr_t parm2,   // Second Parameter
  uintptr_t parm3    // Third Parameter
) {
  // Pass the Function Number and Parameters in
  // Registers A0 to A3
  register long r0 asm("a0") = (long)(nbr);
  register long r1 asm("a1") = (long)(parm1);
  register long r2 asm("a2") = (long)(parm2);
  register long r3 asm("a3") = (long)(parm3);

  // `ecall` will jump from RISC-V User Mode
  // to RISC-V Supervisor Mode
  // to execute the System Call.
  // Input + Output Registers: A0 to A3
  // Clobbers the Memory
  asm volatile
  (
    "ecall"
    :: "r"(r0), "r"(r1), "r"(r2), "r"(r3)
    : "memory"
  );

  // No-operation, does nothing
  asm volatile("nop" : "=r"(r0));

  // Return the result from Register A0
  return r0;
}

ecall is the RISC-V Instruction that jumps from RISC-V User Mode to Supervisor Mode…

That allows NuttX Kernel to execute the actual “write” function, with the real Serial Device.

(We’ll explain how)

Why the no-op after ecall?

We’re guessing: It might be reserved for special calls to NuttX Kernel in future.

(Similar to ebreak for Semihosting)

Every System Call to NuttX Kernel has its own Proxy Function?

Yep! We can see the Auto-Generated Proxy Functions for each System Call…

## Proxy Functions called by `hello` app
$ grep PROXY hello.S
PROXY__assert.c
PROXY__exit.c
PROXY_clock_gettime.c
PROXY_gettid.c
PROXY_lseek.c
PROXY_nxsem_wait.c
PROXY_sem_clockwait.c
PROXY_sem_destroy.c
PROXY_sem_post.c
PROXY_sem_trywait.c
PROXY_task_setcancelstate.c
PROXY_write.c

Next we figure out how System Calls will work…

NuttX Kernel handles System Call

§3 NuttX Kernel handles System Call

Our App makes an ecall to jump to NuttX Kernel (pic above)…

What happens on the other side?

Remember the Proxy Function from earlier? Now we do the exact opposite in our Stub Function (that runs in the Kernel)…

// From nuttx/syscall/stubs/STUB_write.c
// Auto-Generated Stub File for `write`
// This runs in NuttX Kernel triggered by `ecall`.
// We make the actual call to `write`.
// (`nbr` is Offset in Stub Lookup Table, unused)
uintptr_t STUB_write(int nbr, uintptr_t parm1, uintptr_t parm2, uintptr_t parm3) {

  // Call the Kernel version of `write`
  return (uintptr_t) write(  
    (int) parm1,    // File Descriptor (1 = Standard Output)
    (FAR const void *) parm2,  // Buffer to be written
    (size_t) parm3  // Number of bytes to write
  );                // Return the result to the App
}

Thus our NuttX Build auto-generates 2 things…

This happens for every System Call exposed by NuttX Kernel…

## Stub Functions in NuttX Kernel
$ grep STUB nuttx.S
STUB__assert.c
STUB__exit.c
STUB_boardctl.c
STUB_chmod.c
STUB_chown.c
...

(More about Proxy and Stub Functions)

Who calls STUB_write?

When our NuttX App makes an ecall, it triggers IRQ 8 (RISCV_IRQ_ECALLU) that’s handled by…

How will dispatch_syscall know which Stub Function to call?

Remember that our Proxy Function (in NuttX App) passes the System Call Number for “write”?

// From nuttx/syscall/proxies/PROXY_write.c
// Auto-Generated Proxy for `write`, called by NuttX App
ssize_t write(int parm1, FAR const void * parm2, size_t parm3) {

  // Make a System Call with 3 parameters...
  return (ssize_t) sys_call3(
    (unsigned int) SYS_write,  // System Call Number (63 = `write`)
    ...

dispatch_syscall (in NuttX Kernel) will look up the System Call Number in the Stub Lookup Table. And fetch the Stub Function to call.

How did we figure out that 63 is the System Call Number for “write”?

OK this gets tricky. Below is the Enum that defines all System Call Numbers: syscall.h and syscall_lookup.h

// System Call Enum sequentially assigns
// all System Call Numbers (8 to 147-ish)
enum {
  ...
  SYSCALL_LOOKUP(close, 1)  // 1 Parameter
  SYSCALL_LOOKUP(ioctl, 3)  // 3 Parameters
  SYSCALL_LOOKUP(read,  3)  // 3 Parameters
  SYSCALL_LOOKUP(write, 3)  // 3 Parameters
  ...
};

However it’s an Enum, numbered sequentially from 8 to 147-ish. We won’t literally see 63 in the NuttX Source Code.

Then we lookup the Debug Info in the RISC-V Disassembly for NuttX Kernel: nuttx.S

Abbrev Number: 6 (DW_TAG_enumerator)
  DW_AT_name        : SYS_write
  DW_AT_const_value : 63

Whoomp there it is! Says here that “write” is System Call #63.

That’s an odd way to define System Call Numbers…

Yeah it’s not strictly an immutable ABI like Linux, because our System Call Numbers may change! It depends on the Build Options that we select.

(ABI means Application Binary Interface)

Though there’s a jolly good thing: It’s super simple to experiment with new System Calls!

(Just add to NuttX System Calls)

(As explained here)

NuttX App calls NuttX Kernel

§4 System Call in Action

This looks complicated… It works right?

Yep we have solid evidence, from NuttX for Ox64 BL808 SBC!

Remember to enable System Call Logging in “make menuconfig”…

Build Setup 
  > Debug Options 
    > Syscall Debug Features 
      > Enable "Syscall Warning, Error and Info"

Watch what happens when we boot NuttX on Ox64 (pic above)…

riscv_dispatch_irq: irq=8
riscv_swint: Entry: regs: 0x5040bcb0 cmd: 63
EPC: 800019b2
A0: 003f A1: 0001 A2: 8000ad00 A3: 001e

(See the Complete Log)

The RISC-V Registers look familiar…

NuttX Kernel calls our Stub Function STUB_write…

riscv_swint: SWInt Return: 37
STUB_write: nbr=440, parm1=1, parm2=8000ad00, parm3=1e
NuttShell (NSH) NuttX-12.0.3

Which calls Kernel “write” and prints the text: “NuttShell”

Then NuttX Kernel completes the ecall…

riscv_swint: Entry: regs: 0x5040baa0 cmd: 3
EPC: 80001a6a
A0: 0003 A1: 5040bbec A2: 001e A3: 0000
riscv_swint: SWInt Return: 1e

And returns the result 0x1E to our NuttX App. (Via sret)

Our NuttX App has successfully made a System Call on Ox64 yay!

(See the Complete Log)

Virtual Memory for NuttX App

Virtual Memory for NuttX App

§5 Kernel Accesses App Memory

NuttX Kernel prints the buffer at 0x8000_AD00…

It doesn’t look like a RAM Address?

That’s a Virtual Memory Address…

TLDR? No worries…

Hence our NuttX App has passed a chunk of its own Virtual Memory. And NuttX Kernel happily prints it!

Huh? NuttX Kernel can access Virtual Memory?

  1. NuttX uses 2 sets of Page Tables: Kernel Page Table and User Page Table.

    (User Page Table defines the Virtual Memory for NuttX Apps)

  2. According to the NuttX Log, the Kernel swaps the RISC-V SATP Register from Kernel Page Table to User Page Table…

    And doesn’t swap back!

  3. Which means the User Page Table is still in effect!

    And the Virtual Memory at 0x8000_0000 is perfectly accessible by the Kernel.

  4. There’s a catch: RISC-V Supervisor Mode (NuttX Kernel) may access the Virtual Memory mapped to RISC-V User Mode (NuttX Apps)…

    Only if the SUM Bit is set in SSTATUS Register!

    (SUM Bit will permit Supervisor User Memory access)

  5. And that’s absolutely hunky dory because at NuttX Startup, nx_start calls…

    up_initial_state which calls…

    riscv_set_idleintctx to set the SUM Bit in SSTATUS Register

    (How NuttX calls nx_start)

That’s why NuttX Kernel can access Virtual Memory (passed by NuttX Apps) at 0x8000_0000!

Kernel Starts a NuttX App

Clickable Version of NuttX Flow

§6 Kernel Starts a NuttX App

Alrighty NuttX Apps can call NuttX Kernel…

But how does NuttX Kernel start a NuttX App?

Previously we walked through the Boot Sequence for NuttX…

Right after that, NuttX Bringup (nx_bringup) calls (pic above)…

To load a NuttX App module: load_module calls…

To load the ELF File: ELF Loader g_elfbinfmt calls…

There’s plenty happening inside Execute Module: exec_module. Too bad we won’t explore today.

(Clickable Version of NuttX Flow)

Initial RAM Disk for Star64 JH7110

Initial RAM Disk for Star64 JH7110

§7 Initial RAM Disk

OK we know how NuttX Kernel starts a NuttX App…

But where are the NuttX Apps stored?

Right now we’re working with the Early Port of NuttX to Ox64 BL808 SBC. We can’t access the File System in the microSD Card.

All we have: A File System that lives in RAM and contains our NuttX Shell + NuttX Apps.

That’s our Initial RAM Disk: initrd

## Build the Apps Filesystem
make -j 8 export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j 8 import
popd

## Generate the Initial RAM Disk `initrd`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
  -f initrd \
  -d ../apps/bin \
  -V "NuttXBootVol"

(Inside a ROM FS Filesystem)

How to load the Initial RAM Disk from microSD to RAM?

U-Boot Bootloader will do it for us!

Two ways that U-Boot can load the Initial RAM Disk from microSD…

  1. Load the Initial RAM Disk from a Separate File: initrd (similar to Star64, pic above)

    This means we modify the U-Boot Script: boot-pine64.scr

    And make it load the initrd file into RAM.

    (Which is good for separating the NuttX Kernel and NuttX Apps)

    OR…

  2. Append the Initial RAM Disk to the NuttX Kernel Image

    U-Boot Bootloader will load (one-shot into RAM) the NuttX Kernel + Initial RAM Disk.

    And we reuse the existing U-Boot Config on the microSD Card: extlinux/extlinux.conf

    (Which might be more efficient for our Limited RAM)

    (More about the U-Boot Boot Flow)

Since Ox64 is low on RAM, we’ll do the Second Method (Append to Kernel). Like this…

## Export the NuttX Kernel to `nuttx.bin`
riscv64-unknown-elf-objcopy \
  -O binary \
  nuttx \
  nuttx.bin

## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero >/tmp/nuttx.pad

## Append Padding and Initial RAM Disk to NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
  >Image

## Overwrite the Linux Image on Ox64 microSD
cp Image "/Volumes/NO NAME/"

## U-Boot Bootloader will load NuttX Kernel and
## Initial RAM Disk into RAM

This is how we made it work…

(Ox64 can boot NuttX from Flash Memory)

Initial RAM Disk for Ox64

§8 Mount the Initial RAM Disk

We appended the Initial RAM Disk to NuttX Kernel (pic above)…

U-Boot Bootloader loads the NuttX Kernel + Initial RAM Disk into RAM…

How in RAM will NuttX Kernel locate the Initial RAM Disk?

Our Initial RAM Disk follows the ROM File System Format (ROM FS). We search our RAM for the ROM File System by its Magic Number.

Then we copy it into the designated Memory Region for mounting: bl808_start.c

// Locate the Initial RAM Disk and copy to the designated Memory Region
void bl808_copy_ramdisk(void) {

  // After _edata, search for "-rom1fs-". This is the RAM Disk Address.
  // Limit search to 256 KB after Idle Stack Top.
  const char *header = "-rom1fs-";
  uint8_t *ramdisk_addr = NULL;
  for (uint8_t *addr = _edata; addr < (uint8_t *)BL808_IDLESTACK_TOP + (256 * 1024); addr++) {
    if (memcmp(addr, header, strlen(header)) == 0) {
      ramdisk_addr = addr;
      break;
    }
  }

  // Stop if RAM Disk is missing
  if (ramdisk_addr == NULL) { PANIC(); }

  // RAM Disk must be after Idle Stack, to prevent overwriting
  if (ramdisk_addr <= (uint8_t *)BL808_IDLESTACK_TOP) { PANIC(); }

  // Read the Filesystem Size from the next 4 bytes, in Big Endian
  // Add 0x1F0 to Filesystem Size
  const uint32_t size =
    (ramdisk_addr[8] << 24) + 
    (ramdisk_addr[9] << 16) + 
    (ramdisk_addr[10] << 8) + 
    ramdisk_addr[11] + 
    0x1F0;

  // Filesystem Size must be less than RAM Disk Memory Region
  if (size > (size_t)__ramdisk_size) { PANIC(); }

  // Copy the Filesystem bytes to RAM Disk Memory Region
  // Warning: __ramdisk_start overlaps with ramdisk_addr + size
  // Which doesn't work with memcpy.
  // Sadly memmove is aliased to memcpy, so we implement memmove ourselves
  bl808_copy_overlap((void *)__ramdisk_start, ramdisk_addr, size);
}

(More about edata, Idle Stack and bl808_copy_overlap in the next section)

Why did we copy Initial RAM Disk to ramdisk_start?

ramdisk_start points to the Memory Region that we reserved for mounting our RAM Disk.

It’s defined in the NuttX Linker Script: ld.script

/* Memory Region for Mounting RAM Disk */
ramdisk (rwx) : ORIGIN = 0x50A00000, LENGTH = 16M
...
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end  = ORIGIN(ramdisk) + LENGTH(ramdisk);

Who calls the code above?

We locate and copy the Initial RAM Disk at the very top of our NuttX Start Code.

This just after erasing the BSS (Global and Static Variables), in case we need to print some messages and it uses Global and Static Variables: bl808_start.c

// NuttX Start Code
void bl808_start(int mhartid) {

  // Clear the BSS for Global and Static Variables
  bl808_clear_bss();

  // Copy the RAM Disk
  bl808_copy_ramdisk();

Later during startup, we mount the RAM Disk from the Memory Region: bl808_appinit.c

// After NuttX has booted...
void board_late_initialize(void) {
  // Mount the RAM Disk
  mount_ramdisk();
}

// Mount the RAM Disk
int mount_ramdisk(void) {
  desc.minor    = RAMDISK_DEVICE_MINOR;
  desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
  desc.sectsize = SECTORSIZE;
  desc.image    = __ramdisk_start;
  ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);

(How NuttX calls board_late_initialize)

All this works great: NuttX mounts our RAM Disk successfully, and starts the ELF Executable for NuttX Shell!

bl808_copy_ramdisk:
  _edata=0x50400258, _sbss=0x50400290, _ebss=0x50407000, BL808_IDLESTACK_TOP=0x50407c00
  ramdisk_addr=0x50408288
  size=8192016
  Before Copy: ramdisk_addr=0x50408288
  After Copy: __ramdisk_start=0x50a00000
  ...
elf_initialize: Registering ELF
uart_register: Registering /dev/console
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
load_absmodule: Loading /system/bin/init
elf_loadbinary: Loading file: /system/bin/init
elf_init: filename: /system/bin/init loadinfo: 0x5040c618
elf_read: Read 64 bytes from offset 0

(“system/bin/init” is the NuttX Shell)

(See the Complete Log)

Last thing for today: The mysterious 64 KB padding…

Initial RAM Disk for Ox64

§9 Pad the Initial RAM Disk

Between NuttX Kernel and Initial RAM Disk…

Why did we pad 64 KB of zeroes? (Pic above)

## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero >/tmp/nuttx.pad

## Append Padding and Initial RAM Disk to NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
  >Image

## U-Boot Bootloader will load NuttX Kernel and
## Initial RAM Disk into RAM

U-Boot Bootloader will load our Initial RAM Disk into RAM. However it’s dangerously close to BSS Memory (Global and Static Variables) and Kernel Stack.

There’s a risk that our Initial RAM Disk will be contaminated by BSS and Stack. This is how we found a clean, safe space for our Initial RAM Disk (pic above)…

We inspect the NuttX Log and the NuttX Linker Script…

// End of Data Section
_edata=0x50400258

// Start of BSS Section
_sbss=0x50400290

// End of BSS Section
_ebss=0x50407000

// Top of Kernel Idle Stack
BL808_IDLESTACK_TOP=0x50407c00

// We located the initrd after the Top of Idle Stack
ramdisk_addr=0x50408288, size=8192016

// And we copied initrd to the Memory Region for the RAM Disk
__ramdisk_start=0x50a00000

Or graphically…

Memory RegionStartEnd
Data Section0x5040_0257
BSS Section0x5040_02900x5040_6FFF
Kernel Idle Stack0x5040_7BFF
Initial RAM Disk0x5040_82880x50BD_8297
RAM Disk Region0x50A0_00000x519F_FFFF

(NuttX will mount the RAM Disk from RAM Disk Region)

(Which overlaps with Initial RAM Disk!)

This says…

  1. NuttX Kernel nuttx.bin terminates at edata.

    (End of Data Section)

  2. If we append Initial RAM Disk initrd directly to the end of nuttx.bin…

    It will collide with the BSS Section and the Kernel Idle Stack.

    And initrd will get overwritten when NuttX runs the Boot Code and Start Code.

    (Boot Code uses the Kernel Idle Stack. Start Code erases the BSS)

  3. Best place to append initrd is after the Kernel Idle Stack.

    (Roughly 32 KB after edata)

  4. That’s why we inserted a padding of 64 KB between nuttx.bin and initrd.

    (Surely initrd won’t collide with BSS and Kernel Idle Stack)

  5. From the previous section, our code locates initrd.

    (Searching for the ROM FS Magic Number)

    And copies initrd to the RAM Disk Region.

  6. Finally NuttX mounts the RAM Disk from RAM Disk Region.

    NuttX Kernel starts the NuttX Shell correctly from the Mounted RAM Disk.

    (Everything goes well, nothing gets contaminated)

Yep our 64 KB Padding looks legit!

64 KB sounds arbitrary. What if the parameters change?

We have Runtime Checks to catch problems: bl808_start.c

// Stop if RAM Disk is missing
if (ramdisk_addr == NULL) { _err("Missing RAM Disk. Check the initrd padding."); PANIC(); }

// RAM Disk must be after Idle Stack, to prevent overwriting
if (ramdisk_addr <= (uint8_t *)BL808_IDLESTACK_TOP) { _err("RAM Disk must be after Idle Stack. Increase the initrd padding by %ul bytes.", (size_t)BL808_IDLESTACK_TOP - (size_t)ramdisk_addr); PANIC(): }

// Filesystem Size must be less than RAM Disk Memory Region
if (size > (size_t)__ramdisk_size) { _err("RAM Disk Region too small"); PANIC(); }

Why call bl808_copy_overlap to copy initrd to RAM Disk Region? Why not memcpy?

That’s because initrd overlaps with RAM Disk Region! (See above)

memcpy won’t work with Overlapping Memory Regions. Thus we added this: bl808_start.c

// Copy a chunk of memory from `src` to `dest`.
// `dest` overlaps with the end of `src`.
// From libs/libc/string/lib_memmove.c
void *bl808_copy_overlap(void *dest, const void *src, size_t count) {
  if (dest <= src) { _err("dest and src should overlap"); PANIC(); }
  char *d = (char *) dest + count;
  char *s = (char *) src + count;
  // TODO: This needs to be `volatile` or GCC Compiler will replace this by memcpy. Very strange. 
  while (count--) {
    d -= 1; s -= 1;
    volatile char c = *s;
    *d = c;
  }
  return dest;
}

We’re sure that it works?

We called verify_image to do a simple Integrity Check on initrd, before and after copying: jh7110_start.c

// Before Copy: Verify the RAM Disk Image to be copied
verify_image(ramdisk_addr);

// Copy the Filesystem bytes to RAM Disk Memory Region
// Warning: __ramdisk_start overlaps with ramdisk_addr + size
// Which doesn't work with memcpy.
// Sadly memmove is aliased to memcpy, so we implement memmove ourselves
bl808_copy_overlap((void *)__ramdisk_start, ramdisk_addr, size);

// After Copy: Verify the copied RAM Disk Image
verify_image(__ramdisk_start);

(verify_image searches for a specific byte)

That’s how we discovered that memcpy doesn’t work. And our bl808_copy_overlap works great for the Initial RAM Disk and NuttX Shell! (Pic below)

Ox64 boots to NuttX Shell

Ox64 boots to NuttX Shell

§10 What’s Next

Like we said at the top of the article…

“One can hide on the First of the Month… But not on the Fifteenth!”

Today we unravelled the inner workings of NuttX Applications for Ox64 BL808 RISC-V SBC…

We’ll do much more for NuttX on Ox64 BL808, stay tuned for updates!

(Like the fixing of UART Interrupts)

Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/app.md