Star64 JH7110 + NuttX RTOS: RISC-V Semihosting and Initial RAM Disk

📝 28 Jul 2023

Booting NuttX on Star64 with Initial RAM Disk

Once upon a time: There was a Very Naive Bloke (me!) who connected a Smartwatch to the internet…

Anyone in world could flash their own firmware on the watch, and watch it run on a Live Video Stream!

Until a Wise Person (politely) flashed some very clever firmware on the watch, that could access other devices connected to the watch…

All because of Semihosting!

Yep this really happened! (Thankfully it was a harmless experiment)

Three years later we’re still having Semihosting Problems, but on a different gadget: the Pine64 Star64 64-bit RISC-V Single-Board Computer. (Pic below)

(Based on StarFive JH7110, the same SoC in VisionFive2)

In this article, we find out…

Star64 RISC-V SBC

§1 NuttX Crashes On Star64

In the last article, we tried porting Apache NuttX RTOS from QEMU Emulator to Star64 JH7110 SBC…

NuttX seems to boot OK for a while…

123067DFHBC
qemu_rv_kernel_mappings: map I/O regions
qemu_rv_kernel_mappings: map kernel text
qemu_rv_kernel_mappings: map kernel data
qemu_rv_kernel_mappings: connect the L1 and L2 page tables
qemu_rv_kernel_mappings: map the page pool
qemu_rv_mm_init: mmu_enable: satp=1077956608
Inx_start: Entry
elf_initialize: Registering ELF
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
load_absmodule: Loading /system/bin/init
elf_loadbinary: Loading file: /system/bin/init
elf_init: filename: /system/bin/init loadinfo: 0x404069e8

(Source)

But then NuttX crashes with a RISC-V Exception…

EXCEPTION: Breakpoint
MCAUSE:    00000003
EPC:       40200434
MTVAL:     00000000

(Source)

Let’s find out why…

NuttX crashes due to a Semihosting Problem

§2 Decipher the RISC-V Exception

NuttX crashes with this RISC-V Exception…

What does it mean?

EXCEPTION: Breakpoint
MCAUSE:    00000003
EPC:       40200434
MTVAL:     00000000

(Source)

According to the Machine Cause Register (MCAUSE), value 3 says that it’s a “Machine Software Interrupt”.

Which means that NuttX has intentionally triggered a Software Interrupt. Probably to execute a Special Function.

Something special? Like what?

We look up the Exception Program Counter (EPC) 0x4020 0434 in our NuttX Disassembly…

nuttx/arch/risc-v/src/common/riscv_semihost.S:37
smh_call():
  // Register A0 contains the Semihosting Operation Number.
  // Register A1 contains the Semihosting Parameter.
  // Shift Left (does nothing)
  40200430: 01f01013  slli zero, zero, 0x1f

  // Crashes here:
  // Trigger Semihosting Breakpoint
  40200434: 00100073  ebreak

  // Shift Right (does nothing)
  // Encodes the Semihosting Call Number 7
  40200438: 40705013  srai zero, zero, 0x7

(Source)

The code above has a special RISC-V Instruction…

ebreak

What’s this ebreak?

From the RISC-V Spec…

“The EBREAK instruction is used to return control to a debugging environment”

“EBREAK was primarily designed to be used by a debugger to cause execution to stop and fall back into the debugger”

OK thanks but we’re not doing any debugging!

The next part is more helpful…

“Another use of EBREAK is to support Semihosting, where the execution environment includes a debugger that can provide services over an Alternate System Call Interface built around the EBREAK instruction”

Aha! NuttX is making a special System Call to Semihosting!

(We’ll see why)

“Because the RISC-V base ISA does not provide more than one EBREAK instruction, RISC-V Semihosting uses a special sequence of instructions to distinguish a Semihosting EBREAK from a Debugger Inserted EBREAK”

Which explains this (strange) preceding RISC-V Instruction…

// Shift Left the value 0x1F
// into Register X0...
// Which is always 0!
slli zero, zero, 0x1f

That doesn’t do anything meaningful!

Let’s talk about Semihosting…

NuttX calls Semihosting to read the Apps Filesystem

§3 NuttX Calls Semihosting

Who calls ebreak? And why?

ebreak is called by smh_call, which is called by host_call…

// NuttX calls Semihosting to
// access the Host Filesystem
static long host_call(
  unsigned int nbr,  // Semihosting Operation Number
  void *parm,        // Semihosting Parameter
  size_t size        // Size of Parameter
) {
  // Call Semihosting via `ebreak`
  long ret = smh_call(
    nbr,  // Semihosting Operation Number
    parm  // Semihosting Parameter
  );

(Source)

What’s this operation number?

The Semihosting Operation Numbers are defined here: riscv_hostfs.c

// Semihosting Operation Numbers
// (For File Operations)
#define HOST_OPEN   0x01
#define HOST_CLOSE  0x02
#define HOST_WRITE  0x05
#define HOST_READ   0x06
#define HOST_SEEK   0x0a
#define HOST_FLEN   0x0c
#define HOST_REMOVE 0x0e
#define HOST_RENAME 0x0f
#define HOST_ERROR  0x13

Aha! NuttX is calling Semihosting to access the File System!

Indeed! When we log host_call, we see…

host_call:
  nbr=0x1 (HOST_OPEN)
  parm=0x40406778
  size=24

Which calls Semihosting to open a file.

Open what file?

If we look back at the NuttX Crash Log…

nx_start_application: 
  Starting init task: /system/bin/init
load_absmodule: 
  Loading /system/bin/init
elf_loadbinary: 
  Loading file: /system/bin/init
elf_init: filename: 
  /system/bin/init loadinfo: 0x404069e8
riscv_exception:
  EXCEPTION: Breakpoint

(Source)

NuttX is trying to read the file /system/bin/init via Semihosting!

Why did it fail? Let’s find out…

§4 NuttX Apps Filesystem

What’s /system/bin/init?

Why is NuttX reading it at startup?

Remember we copied NuttX from QEMU and (naively) ran it on Star64?

We backtrack to the origin (NuttX on QEMU) and figure out what’s /system/bin/init…

## Build NuttX QEMU in Kernel Mode
tools/configure.sh rv-virt:knsh64
make V=1 -j7

## Build Apps Filesystem for NuttX QEMU
make export V=1
pushd ../apps
./tools/mkimport.sh \
  -z -x \
  ../nuttx/nuttx-export-*.tar.gz
make import V=1
popd

## Dump the `init` disassembly to `init.S`
riscv64-unknown-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  ../apps/bin/init \
  >init.S \
  2>&1

(Source)

(Why we use Kernel Mode)

The above commands will build the Apps Filesystem for NuttX QEMU.

Which includes /system/bin/init…

$ ls ../apps/bin       
getprime
hello
init
sh

Isn’t it supposed to be /system/bin/init? Not /apps/bin/init?

When we check the NuttX Build Configuration…

$ grep INIT .config
CONFIG_INIT_FILE=y
CONFIG_INIT_ARGS=""
CONFIG_INIT_FILEPATH="/system/bin/init"
CONFIG_INIT_MOUNT=y
CONFIG_INIT_MOUNT_SOURCE=""
CONFIG_INIT_MOUNT_TARGET="/system"
CONFIG_INIT_MOUNT_FSTYPE="hostfs"
CONFIG_INIT_MOUNT_FLAGS=0x1
CONFIG_INIT_MOUNT_DATA="fs=../apps"
CONFIG_PATH_INITIAL="/system/bin"
CONFIG_NSH_ARCHINIT=y

(Source)

We see that NuttX will mount the /apps filesystem as /system, via the Semihosting Host Filesystem.

That’s why it appears as /system/bin/init!

What’s inside /system/bin/init?

The RISC-V Disassembly of /system/bin/init shows this…

apps/system/nsh/nsh_main.c:52
  0000006e <main>:
    int main(int argc, FAR char *argv[]) {

(Source)

Yep it’s the Compiled ELF Executable of the NuttX Shell nsh!

Now everything makes sense…

  1. At Startup: NuttX tries to load /system/bin/init to start the NuttX Shell nsh

  2. But it Fails: Because /system/bin/init doesn’t exist in the Semihosting Filesystem on Star64!

This is why Semihosting won’t work on Star64…

QEMU reads the Apps Filesystem over Semihosting

§5 Semihosting on NuttX QEMU

Why Semihosting won’t work on Star64 SBC?

Semihosting was created for Hardware Debuggers and Virtual Machine Hypervisors, like QEMU Emulator.

The pic above shows how it works: Semihosting enables a Virtual Machine (like NuttX) to “Break Out” of its Sandbox to access the Filesystem on the Host Machine / Our Computer.

(Remember our story at the top of the article? Be careful with Semihosting!)

That’s why we Enable Semihosting when we run NuttX on QEMU…

## Start NuttX on QEMU
## with Semihosting Enabled
qemu-system-riscv64 \
  -kernel nuttx \
  -cpu rv64 \
  -M virt,aclint=on \
  -semihosting \
  -bios none \
  -nographic

(Source)

(Remove -bios none for newer versions of NuttX)

So that NuttX can access the Apps Filesystem (from previous section) as a Semihosting Filesystem! (Pic above)

(More about RISC-V Semihosting)

(See the Semihosting Spec)

This won’t work on Star64?

Semihosting won’t work because NuttX for Star64 runs on Real SBC Hardware (Bare Metal)…

There’s nothing to “break out” to!

Initial RAM Disk for NuttX

If not Semihosting… Then what?

In the world of Linux (and QEMU), there’s something cool called an Initial RAM Disk (initrd)…

Perfect for our NuttX Apps Filesystem!

That’s awesome but where do we start?

We begin by modding NuttX QEMU to load the Initial RAM Disk…

NuttX for QEMU will mount the Apps Filesystem from an Initial RAM Disk

§6 Modify NuttX QEMU for Initial RAM Disk

NuttX QEMU will load an Initial RAM Disk…

Instead of using Semihosting. How?

In the previous section, we said that…

To modify NuttX QEMU to load an Initial RAM Disk, we define the address of the RAM Disk Memory in the Linker Script: ld-kernel64.script

MEMORY
{
  ...
  /* Added RAM Disk Memory (Max 16 MB) */
  ramdisk (rwx) : ORIGIN = 0x80800000, LENGTH = 16M   /* w/ cache */
}

/* Increased Page Heap for RAM Disk */
__pgheap_size = LENGTH(pgram) + LENGTH(ramdisk);
/* Previously: __pgheap_size = LENGTH(pgram); */

/* Added RAM Disk Symbols */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size  = LENGTH(ramdisk);
__ramdisk_end   = ORIGIN(ramdisk) + LENGTH(ramdisk);

(0x8080 0000 is the next available RAM Address)

At NuttX Startup, we mount the RAM Disk: qemu_rv_appinit.c

// Called at NuttX Startup
void board_late_initialize(void) {

  // Mount the RAM Disk
  mount_ramdisk();

  // Perform board-specific initialization
#ifdef CONFIG_NSH_ARCHINIT
  mount(NULL, "/proc", "procfs", 0, NULL);
#endif
}

// Mount the RAM Disk
int mount_ramdisk(void) {

  // Define the ROMFS
  struct boardioc_romdisk_s desc;
  desc.minor    = RAMDISK_DEVICE_MINOR;
  desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
  desc.sectsize = SECTORSIZE;
  desc.image    = __ramdisk_start;

  // Mount the ROMFS
  int ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
  // Omitted: Handle Errors

(More about ROMFS in a while)

Before mounting, we copy the RAM Disk from 0x8400 0000 to ramdisk_start: qemu_rv_mm_init.c

void qemu_rv_kernel_mappings(void) {
  ...
  // Copy RAM Disk from 0x8400 0000 to
  // `__ramdisk_start` (`__ramdisk_size` bytes)
  // TODO: RAM Disk must not exceed `__ramdisk_size` bytes
  memcpy(                     // Copy the RAM Disk...
    (void *)__ramdisk_start,  // To RAM Disk Memory
    (void *)0x84000000,       // From QEMU initrd Address
    (size_t)__ramdisk_size    // For 16 MB
  );

(More about 0x8400 0000 in a while)

(Somehow map_region crashes when we map the RAM Disk Memory)

Things get really wonky when we exceed the bounds of the RAM Disk. So we validate the bounds: fs_romfsutil.c

// While reading from RAM Disk...
static uint32_t romfs_devread32(struct romfs_mountpt_s *rm, int ndx) {

  // If we're reading beyond the bounds of
  // RAM Disk Memory, halt (and catch fire)
  DEBUGASSERT(
    &rm->rm_buffer[ndx] <
      __ramdisk_start + (size_t)__ramdisk_size
  );

Finally we configure NuttX QEMU to mount the Initial RAM Disk as ROMFS (instead of Semihosting): knsh64/defconfig

CONFIG_BOARDCTL_ROMDISK=y
CONFIG_BOARD_LATE_INITIALIZE=y
CONFIG_FS_ROMFS=y
CONFIG_INIT_FILEPATH="/system/bin/init"
CONFIG_INIT_MOUNT=y
CONFIG_INIT_MOUNT_FLAGS=0x1
CONFIG_INIT_MOUNT_TARGET="/system/bin"

## We removed these...
## CONFIG_FS_HOSTFS=y
## CONFIG_RISCV_SEMIHOSTING_HOSTFS=y

(How we configured NuttX for RAM Disk)

That’s it! These are the files that we modified in NuttX QEMU to load the Initial RAM Disk (without Semihosting)…

What’s ROMFS?

ROMFS is the Filesystem Format of our Initial RAM Disk. (It defines how the Files and Directories are stored in the RAM Disk)

We could have used a FAT or EXT4 or NTFS Filesystem… But ROMFS is a lot simpler for NuttX.

(More about ROMFS in NuttX)

Why did we copy the RAM Disk from 0x8400 0000?

QEMU loads the Initial RAM Disk into RAM at 0x8400 0000…

That’s why we copied the RAM Disk from 0x8400 0000 to ramdisk_start.

Wow how did we figure out all this?

Actually we had plenty of guidance from NuttX on LiteX Arty-A7. Here’s our Detailed Analysis…

Booting NuttX QEMU with Initial RAM Disk

§7 Boot NuttX QEMU with Initial RAM Disk

We’re ready to run our modified NuttX QEMU… That loads the Initial RAM Disk!

We build NuttX QEMU in Kernel Mode (as before). Then we generate the Initial RAM Disk initrd…

## Omitted: Build NuttX QEMU in Kernel Mode
...
## Omitted: Build Apps Filesystem for NuttX QEMU
...
## Generate the Initial RAM Disk `initrd`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
  -f initrd \
  -d ../apps/bin \
  -V "NuttXBootVol"

(See the Build Steps)

(See the Build Log)

(genromfs generates a ROM FS Filesystem)

(Inside a ROM FS Filesystem)

This creates an Initial RAM Disk initrd (in ROMFS format) that’s 7.9 MB…

$ ls -l initrd
-rw-r--r--  1 7902208 initrd

(See the Build Outputs)

Finally we start QEMU and load our Initial RAM Disk…

## Start NuttX on QEMU
## with Initial RAM Disk `initrd`
qemu-system-riscv64 \
  -kernel nuttx \
  -initrd initrd \
  -cpu rv64 \
  -M virt,aclint=on \
  -semihosting \
  -bios none \
  -nographic

(Source)

(Remove -bios none for newer versions of NuttX)

And NuttX QEMU boots OK with our Initial RAM Disk yay! (Ignore the warnings)

ABC
nx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
board_late_initialize:
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
up_exit: TCB=0x802088d0 exiting

NuttShell (NSH) NuttX-12.0.3
nsh> nx_start: CPU0: Beginning Idle Loop
nsh>

(See the Run Log)

(See the Detailed Run Log)

We see exec_spawn warnings like this…

nsh> ls -l /system/bin/init
posix_spawn: pid=0xc0202978 path=ls file_actions=0xc0202980 attr=0xc0202988 argv=0xc0202a28
exec_spawn: ERROR: Failed to load program 'ls': -2
nxposix_spawn_exec: ERROR: exec failed: 2
 -r-xr-xr-x 3278720 /system/bin/init

But it’s OK to ignore them, because “ls” is a built-in Shell Command.

(Not an Executable File from our Apps Filesystem)

Now that we figured out Initial RAM Disk on QEMU, let’s do the same for Star64…

Booting NuttX on Star64 with Initial RAM Disk

§8 NuttX Star64 with Initial RAM Disk

One last thing for today: Booting NuttX on Star64 with Initial RAM Disk! (Instead of Semihosting)

We modify NuttX Star64 with the exact same steps as NuttX QEMU with Initial RAM Disk…

Note that we copy the Initial RAM Disk from 0x4610 0000 (instead of QEMU’s 0x8400 0000): jh7110_mm_init.c

// Copy RAM Disk from 0x4610 0000 to
// `__ramdisk_start` (`__ramdisk_size` bytes)
// TODO: RAM Disk must not exceed `__ramdisk_size` bytes
memcpy(                     // Copy the RAM Disk...
  (void *)__ramdisk_start,  // To RAM Disk Memory
  (void *)0x46100000,       // From U-Boot initrd Address
  (size_t)__ramdisk_size    // For 16 MB
);

(U-Boot Bootloader loads the RAM Disk at 0x4610 0000)

And the RAM Disk Memory is now located at 0x40A0 0000 (the next available RAM Address): ld.script

MEMORY
{
  ...
  /* Added RAM Disk Memory (Max 16 MB) */
  ramdisk (rwx) : ORIGIN = 0x40A00000, LENGTH = 16M   /* w/ cache */
}

/* Increased Page Heap for RAM Disk */
__pgheap_size = LENGTH(pgram) + LENGTH(ramdisk);
/* Previously: __pgheap_size = LENGTH(pgram); */

/* Added RAM Disk Symbols */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size  = LENGTH(ramdisk);
__ramdisk_end   = ORIGIN(ramdisk) + LENGTH(ramdisk);

The other modified files are the same as for NuttX QEMU with Initial RAM Disk.

(How to increase the RAM Disk Limit)

(NuttX Apps are limited to 4 MB RAM)

(How to increase the Page Heap Size)

How do we run this on Star64?

We build NuttX Star64, generate the Initial RAM Disk initrd and copy to our TFTP Folder (for Network Booting)…

## Omitted: Build NuttX Star64
...
## Omitted: Build Apps Filesystem for NuttX Star64
...
## Generate the Initial RAM Disk `initrd`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
  -f initrd \
  -d ../apps/bin \
  -V "NuttXBootVol"

## Copy NuttX Binary Image, Device Tree and
## Initial RAM Disk to TFTP Folder
cp nuttx.bin $HOME/tftproot/Image
cp jh7110-star64-pine64.dtb $HOME/tftproot
cp initrd $HOME/tftproot

(See the Build Steps)

(See the Build Log)

(genromfs generates a ROM FS Filesystem)

(Inside a ROM FS Filesystem)

Our Initial RAM Disk initrd (with ROMFS inside) is 7.9 MB (slightly bigger)…

$ ls -l initrd
-rw-r--r--  1 7930880 initrd

(See the Build Outputs)

And we boot NuttX on Star64 over TFTP or a microSD Card…

Does it work?

Now Star64 JH7110 boots OK with the Initial RAM Disk yay! (Not completely though)

Starting kernel ...
123067DFHBCI
nx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
board_late_initialize: 
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
nx_start_application: ret=3
up_exit: TCB=0x404088d0 exiting
nx_start: CPU0: Beginning Idle Loop

(See the Output Log)

So many questions (pic below)…

We’ll find out in the next article!

NuttX Star64 with Initial RAM Disk

§9 What’s Next

No more Semihosting Problems with NuttX on Star64 JH7110 SBC!

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/semihost.md

Booting NuttX on Star64 with Initial RAM Disk

§10 Appendix: Boot NuttX over TFTP with Initial RAM Disk

Previously we configured Star64’s U-Boot Bootloader to boot NuttX over TFTP…

Now we need to tweak the U-Boot Settings to boot with our Initial RAM Disk.

Star64’s U-Boot Bootloader loads our Initial RAM Disk at 0x4610 0000…

ramdisk_addr_r=0x46100000

(Source)

Which means that we need to add these TFTP Commands to U-Boot Bootloader…

## Added this: Assume Initial RAM Disk is max 16 MB
setenv ramdisk_size 0x1000000
printenv ramdisk_size
saveenv

## Load Kernel and Device Tree over TFTP
tftpboot ${kernel_addr_r} ${tftp_server}:Image
tftpboot ${fdt_addr_r} ${tftp_server}:jh7110-star64-pine64.dtb
fdt addr ${fdt_addr_r}

## Added this: Load Initial RAM Disk over TFTP
tftpboot ${ramdisk_addr_r} ${tftp_server}:initrd

## Changed this: Replaced `-` by `ramdisk_addr_r:ramdisk_size`
booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r}

Which will change our U-Boot Boot Script to…

## Load the NuttX Image from TFTP Server
## kernel_addr_r=0x40200000
## tftp_server=192.168.x.x
if tftpboot ${kernel_addr_r} ${tftp_server}:Image;
then

  ## Load the Device Tree from TFTP Server
  ## fdt_addr_r=0x46000000
  if tftpboot ${fdt_addr_r} ${tftp_server}:jh7110-star64-pine64.dtb;
  then

    ## Set the RAM Address of Device Tree
    ## fdt_addr_r=0x46000000
    if fdt addr ${fdt_addr_r};
    then

      ## Load the Intial RAM Disk from TFTP Server
      ## ramdisk_addr_r=0x46100000
      if tftpboot ${ramdisk_addr_r} ${tftp_server}:initrd;
      then

        ## Boot the NuttX Image with the Initial RAM Disk and Device Tree
        ## kernel_addr_r=0x40200000
        ## ramdisk_addr_r=0x46100000
        ## ramdisk_size=0x1000000
        ## fdt_addr_r=0x46000000
        booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r};
      fi;
    fi;
  fi;
fi

Which becomes…

## Assume Initial RAM Disk is max 16 MB
setenv ramdisk_size 0x1000000
## Check that it's correct
printenv ramdisk_size
## Save it for future reboots
saveenv

## Add the Boot Command for TFTP
setenv bootcmd_tftp 'if tftpboot ${kernel_addr_r} ${tftp_server}:Image ; then if tftpboot ${fdt_addr_r} ${tftp_server}:jh7110-star64-pine64.dtb ; then if fdt addr ${fdt_addr_r} ; then if tftpboot ${ramdisk_addr_r} ${tftp_server}:initrd ; then booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r} ; fi ; fi ; fi ; fi'
## Check that it's correct
printenv bootcmd_tftp
## Save it for future reboots
saveenv

(Remember to set tftp_server and boot_targets)

Run the above commands in U-Boot.

Copy the Initial RAM Disk initrd to the TFTP Folder…

## Copy NuttX Binary Image, Device Tree and
## Initial RAM Disk to TFTP Folder
cp nuttx.bin $HOME/tftproot/Image
cp jh7110-star64-pine64.dtb $HOME/tftproot
cp initrd $HOME/tftproot

(Source)

Power Star64 off and on.

NuttX now boots with our Initial RAM Disk over TFTP…

Here’s the U-Boot Log…

TFTP from server 192.168.x.x; our IP address is 192.168.x.x
Filename 'Image'.
Load address: 0x40200000
Loading: 9 MiB/s
done
Bytes transferred = 2097800 (200288 hex)
Using ethernet@16030000 device
TFTP from server 192.168.x.x; our IP address is 192.168.x.x
Filename 'jh7110-star64-pine64.dtb'.
Load address: 0x46000000
Loading: 8 MiB/s
done
Bytes transferred = 50235 (c43b hex)
Using ethernet@16030000 device
TFTP from server 192.168.x.x; our IP address is 192.168.x.x
Filename 'initrd'.
Load address: 0x46100000
Loading: 371.1 KiB/s
done
Bytes transferred = 7930880 (790400 hex)
## Flattened Device Tree blob at 46000000
   Booting using the fdt blob at 0x46000000
   Using Device Tree in place at 0000000046000000, end 000000004600f43a
Starting kernel ...

What if we omit the RAM Disk Size?

U-Boot won’t boot NuttX if we omit the RAM Disk Size…

## If we omit RAM Disk Size:
## Boot Fails
$ booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
Wrong Ramdisk Image Format
Ramdisk image is corrupt or invalid

So we hardcode a maximum RAM Disk Size of 16 MB…

## If we assume RAM Disk Size is max 16 MB:
## Boots OK
$ booti ${kernel_addr_r} ${ramdisk_addr_r}:0x1000000 ${fdt_addr_r}

Let’s talk about the NuttX Configuration for Initial RAM Disk…

NuttX Star64 with Initial RAM Disk

§11 Appendix: Configure NuttX for Initial RAM Disk

Earlier we configured NuttX QEMU and NuttX Star64 to boot with our Initial RAM Disk…

Here are the steps for updating the NuttX Build Configuration in make menuconfig…

  1. Board Selection > Enable boardctl() interface > Enable application space creation of ROM disks

  2. RTOS Features > RTOS hooks > Custom board late initialization

  3. File Systems > ROMFS file system

  4. RTOS Features > Tasks and Scheduling > Auto-mount init file system

    Set to /system/bin

  5. Build Setup > Debug Options > File System Debug Features > File System Error, Warnings and Info Output

  6. Disable: File Systems > Host File System

  7. Manually delete from nsh/defconfig…

    CONFIG_HOST_MACOS=y
CONFIG_INIT_MOUNT_DATA="fs=../apps"
CONFIG_INIT_MOUNT_FSTYPE="hostfs"
CONFIG_INIT_MOUNT_SOURCE=""

The steps above will produce the updated Build Configuration Files…

§12 Appendix: RAM Disk Address for RISC-V QEMU

We need the RAM Disk Address for RISC-V QEMU…

Can we enable logging for RISC-V QEMU?

Yep we use this QEMU Option: -trace "*"

## Start NuttX on QEMU
## with Initial RAM Disk `initrd`
qemu-system-riscv64 \
  -semihosting \
  -M virt,aclint=on \
  -cpu rv64 \
  -bios none \
  -kernel nuttx \
  -initrd initrd \
  -nographic \
  -trace "*"

(Remove -bios none for newer versions of NuttX)

In the QEMU Command above, we load the Initial RAM Disk initrd.

To discover the RAM Address of the Initial RAM Disk, we check the QEMU Trace Log…

resettablloader_write_rom nuttx
  ELF program header segment 0:
  @0x80000000 size=0x2b374 ROM=0
loader_write_rom nuttx
  ELF program header segment 1:
  @0x80200000 size=0x2a1 ROM=0
loader_write_rom initrd:
  @0x84000000 size=0x2fc3e8 ROM=0
loader_write_rom fdt:
  @0x87000000 size=0x100000 ROM=0

This says that QEMU loads our Initial RAM Disk initrd at 0x8400 0000

(And QEMU loads our Kernel at 0x8000 0000, Device Tree at 0x8700 0000)

We set the RAM Address of the Initial RAM Disk here…

We thought the Initial RAM Disk Address could be discovered from the Device Tree for RISC-V QEMU. But nope it’s not there…

§13 Appendix: Device Tree for RISC-V QEMU

To dump the Device Tree for RISC-V QEMU, we specify dumpdtb…

## Dump Device Tree for RISC-V QEMU
## Remove `-bios none` for newer versions of NuttX
qemu-system-riscv64 \
  -semihosting \
  -M virt,aclint=on,dumpdtb=qemu-riscv64.dtb \
  -cpu rv64 \
  -bios none \
  -kernel nuttx \
  -nographic

## Convert Device Tree to text format
dtc \
  -o qemu-riscv64.dts \
  -O dts \
  -I dtb \
  qemu-riscv64.dtb

(dtc decompiles a Device Tree)

This produces the Device Tree for RISC-V QEMU…

Which is helpful for browsing the Memory Addresses of I/O Peripherals in QEMU.

§14 Appendix: Initial RAM Disk for LiteX Arty-A7

Earlier we modified NuttX QEMU and NuttX Star64 to load our Initial RAM Disk…

We did it with plenty of guidance from NuttX on LiteX Arty-A7, below is our Detailed Analysis.

To generate the RAM Disk for LiteX Arty-A7, we run this command…

## Generate the Initial RAM Disk `romfs.img`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
  -f romfs.img \
  -d ../apps/bin \ 
  -V "NuttXBootVol"

(Source)

(About genromfs)

(Inside a ROM FS Filesystem)

(About NuttX RAM Disks and ROM Disks)

LiteX Memory Map tells us where the RAM Disk is loaded: 0x40C0 0000

"romfs.img":   "0x40C00000",
"nuttx.bin":   "0x40000000",
"opensbi.bin": "0x40f00000"

This is the LiteX Build Configuration for mounting the RAM Disk: knsh/defconfig

CONFIG_BOARDCTL_ROMDISK=y
CONFIG_BOARD_LATE_INITIALIZE=y
CONFIG_BUILD_KERNEL=y
CONFIG_FS_ROMFS=y
CONFIG_INIT_FILEPATH="/system/bin/init"
CONFIG_INIT_MOUNT=y
CONFIG_INIT_MOUNT_FLAGS=0x1
CONFIG_INIT_MOUNT_TARGET="/system/bin"
CONFIG_LITEX_APPLICATION_RAMDISK=y
CONFIG_NSH_FILE_APPS=y
CONFIG_NSH_READLINE=y
CONFIG_PATH_INITIAL="/system/bin"
CONFIG_RAM_SIZE=4194304
CONFIG_RAM_START=0x40400000
CONFIG_RAW_BINARY=y
CONFIG_SYSTEM_NSH_PROGNAME="init"
CONFIG_TESTING_GETPRIME=y

Which is consistent with the NuttX Doc on NSH Start-Up Script…

CONFIG_DISABLE_MOUNTPOINT not set
CONFIG_FS_ROMFS enabled

We mount the RAM Disk at LiteX Startup: litex_appinit.c

void board_late_initialize(void)
{
#ifdef CONFIG_LITEX_APPLICATION_RAMDISK
  litex_mount_ramdisk();
#endif

  litex_bringup();
}

Which calls litex_mount_ramdisk to mount the RAM Disk: litex_ramdisk.c

#ifndef CONFIG_BUILD_KERNEL
#error "Ramdisk usage is intended to be used with kernel build only"
#endif

#define SECTORSIZE   512
#define NSECTORS(b)  (((b) + SECTORSIZE - 1) / SECTORSIZE)
#define RAMDISK_DEVICE_MINOR 0

// Mount a ramdisk defined in the ld-kernel.script to /dev/ramX.
// The ramdisk is intended to contain a romfs with applications which can
// be spawned at runtime.
int litex_mount_ramdisk(void)
{
  int ret;
  struct boardioc_romdisk_s desc;

  desc.minor    = RAMDISK_DEVICE_MINOR;
  desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
  desc.sectsize = SECTORSIZE;
  desc.image    = __ramdisk_start;

  ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
  if (ret < 0)
    {
      syslog(LOG_ERR, "Ramdisk register failed: %s\n", strerror(errno));
      syslog(LOG_ERR, "Ramdisk mountpoint /dev/ram%d\n",
                                          RAMDISK_DEVICE_MINOR);
      syslog(LOG_ERR, "Ramdisk length %u, origin %x\n",
                                          (ssize_t)__ramdisk_size,
                                          (uintptr_t)__ramdisk_start);
    }

  return ret;
}

ramdisk_start and ramdisk_size are defined in the LiteX Memory Map: board_memorymap.h

/* RAMDisk */
#define RAMDISK_START     (uintptr_t)__ramdisk_start
#define RAMDISK_SIZE      (uintptr_t)__ramdisk_size

/* ramdisk (RW) */
extern uint8_t          __ramdisk_start[];
extern uint8_t          __ramdisk_size[];

And also in the LiteX Linker Script: ld-kernel.script

MEMORY
{
  kflash (rx)   : ORIGIN = 0x40000000, LENGTH = 4096K   /* w/ cache */
  ksram (rwx)   : ORIGIN = 0x40400000, LENGTH = 4096K   /* w/ cache */
  pgram (rwx)   : ORIGIN = 0x40800000, LENGTH = 4096K   /* w/ cache */
  ramdisk (rwx) : ORIGIN = 0x40C00000, LENGTH = 4096K   /* w/ cache */
}
...
/* Page heap */
__pgheap_start = ORIGIN(pgram);
__pgheap_size = LENGTH(pgram) + LENGTH(ramdisk);

/* Application ramdisk */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end  = ORIGIN(ramdisk) + LENGTH(ramdisk);

Note that pgheap_size needs to include ramdisk size.