đź“ť 28 Jul 2023
Once upon a time: There was a Very Naive Bloke (me!) who connected a Smartwatch to the internet…
Anyone in world could flash their own firmware on the watch, and watch it run on a Live Video Stream!
Until a Wise Person (politely) flashed some very clever firmware on the watch, that could access other devices connected to the watch…
All because of Semihosting!
Yep this really happened! (Thankfully it was a harmless experiment)
Three years later we’re still having Semihosting Problems, but on a different gadget: the Pine64 Star64 64-bit RISC-V Single-Board Computer. (Pic below)
(Based on StarFive JH7110, the same SoC in VisionFive2)
In this article, we find out…
What’s RISC-V Semihosting
Why it crashes Apache NuttX RTOS on Star64
How it affects the Apps Filesystem in NuttX
How we replaced Semihosting by Initial RAM Disk “initrd” (pic above)
After testing on QEMU Emulator
Thanks to NuttX on LiteX Arty-A7 for the guidance!
In the last article, we tried porting Apache NuttX RTOS from QEMU Emulator to Star64 JH7110 SBC…
NuttX seems to boot OK for a while…
123067DFHBC
qemu_rv_kernel_mappings: map I/O regions
qemu_rv_kernel_mappings: map kernel text
qemu_rv_kernel_mappings: map kernel data
qemu_rv_kernel_mappings: connect the L1 and L2 page tables
qemu_rv_kernel_mappings: map the page pool
qemu_rv_mm_init: mmu_enable: satp=1077956608
Inx_start: Entry
elf_initialize: Registering ELF
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
load_absmodule: Loading /system/bin/init
elf_loadbinary: Loading file: /system/bin/init
elf_init: filename: /system/bin/init loadinfo: 0x404069e8
But then NuttX crashes with a RISC-V Exception…
EXCEPTION: Breakpoint
MCAUSE: 00000003
EPC: 40200434
MTVAL: 00000000
Let’s find out why…
NuttX crashes with this RISC-V Exception…
What does it mean?
EXCEPTION: Breakpoint
MCAUSE: 00000003
EPC: 40200434
MTVAL: 00000000
According to the Machine Cause Register (MCAUSE), value 3 says that it’s a “Machine Software Interrupt”.
Which means that NuttX has intentionally triggered a Software Interrupt. Probably to execute a Special Function.
Something special? Like what?
We look up the Exception Program Counter (EPC) 0x4020 0434 in our NuttX Disassembly…
nuttx/arch/risc-v/src/common/riscv_semihost.S:37
smh_call():
// Register A0 contains the Semihosting Operation Number.
// Register A1 contains the Semihosting Parameter.
// Shift Left (does nothing)
40200430: 01f01013 slli zero, zero, 0x1f
// Crashes here:
// Trigger Semihosting Breakpoint
40200434: 00100073 ebreak
// Shift Right (does nothing)
// Encodes the Semihosting Call Number 7
40200438: 40705013 srai zero, zero, 0x7
The code above has a special RISC-V Instruction…
ebreak
What’s this ebreak?
From the RISC-V Spec…
“The EBREAK instruction is used to return control to a debugging environment”
“EBREAK was primarily designed to be used by a debugger to cause execution to stop and fall back into the debugger”
OK thanks but we’re not doing any debugging!
The next part is more helpful…
“Another use of EBREAK is to support Semihosting, where the execution environment includes a debugger that can provide services over an Alternate System Call Interface built around the EBREAK instruction”
Aha! NuttX is making a special System Call to Semihosting!
(We’ll see why)
“Because the RISC-V base ISA does not provide more than one EBREAK instruction, RISC-V Semihosting uses a special sequence of instructions to distinguish a Semihosting EBREAK from a Debugger Inserted EBREAK”
Which explains this (strange) preceding RISC-V Instruction…
// Shift Left the value 0x1F
// into Register X0...
// Which is always 0!
slli zero, zero, 0x1f
That doesn’t do anything meaningful!
Let’s talk about Semihosting…
Who calls ebreak? And why?
ebreak is called by smh_call, which is called by host_call…
// NuttX calls Semihosting to
// access the Host Filesystem
static long host_call(
unsigned int nbr, // Semihosting Operation Number
void *parm, // Semihosting Parameter
size_t size // Size of Parameter
) {
// Call Semihosting via `ebreak`
long ret = smh_call(
nbr, // Semihosting Operation Number
parm // Semihosting Parameter
);
What’s this operation number?
The Semihosting Operation Numbers are defined here: riscv_hostfs.c
// Semihosting Operation Numbers
// (For File Operations)
#define HOST_OPEN 0x01
#define HOST_CLOSE 0x02
#define HOST_WRITE 0x05
#define HOST_READ 0x06
#define HOST_SEEK 0x0a
#define HOST_FLEN 0x0c
#define HOST_REMOVE 0x0e
#define HOST_RENAME 0x0f
#define HOST_ERROR 0x13
Aha! NuttX is calling Semihosting to access the File System!
Indeed! When we log host_call, we see…
host_call:
nbr=0x1 (HOST_OPEN)
parm=0x40406778
size=24
Which calls Semihosting to open a file.
Open what file?
If we look back at the NuttX Crash Log…
nx_start_application:
Starting init task: /system/bin/init
load_absmodule:
Loading /system/bin/init
elf_loadbinary:
Loading file: /system/bin/init
elf_init: filename:
/system/bin/init loadinfo: 0x404069e8
riscv_exception:
EXCEPTION: Breakpoint
NuttX is trying to read the file /system/bin/init via Semihosting!
Why did it fail? Let’s find out…
What’s /system/bin/init?
Why is NuttX reading it at startup?
Remember we copied NuttX from QEMU and (naively) ran it on Star64?
We backtrack to the origin (NuttX on QEMU) and figure out what’s /system/bin/init…
## Build NuttX QEMU in Kernel Mode
tools/configure.sh rv-virt:knsh64
make V=1 -j7
## Build Apps Filesystem for NuttX QEMU
make export V=1
pushd ../apps
./tools/mkimport.sh \
-z -x \
../nuttx/nuttx-export-*.tar.gz
make import V=1
popd
## Dump the `init` disassembly to `init.S`
riscv64-unknown-elf-objdump \
-t -S --demangle --line-numbers --wide \
../apps/bin/init \
>init.S \
2>&1
The above commands will build the Apps Filesystem for NuttX QEMU.
Which includes /system/bin/init…
$ ls ../apps/bin
getprime
hello
init
sh
Isn’t it supposed to be /system/bin/init? Not /apps/bin/init?
When we check the NuttX Build Configuration…
$ grep INIT .config
CONFIG_INIT_FILE=y
CONFIG_INIT_ARGS=""
CONFIG_INIT_FILEPATH="/system/bin/init"
CONFIG_INIT_MOUNT=y
CONFIG_INIT_MOUNT_SOURCE=""
CONFIG_INIT_MOUNT_TARGET="/system"
CONFIG_INIT_MOUNT_FSTYPE="hostfs"
CONFIG_INIT_MOUNT_FLAGS=0x1
CONFIG_INIT_MOUNT_DATA="fs=../apps"
CONFIG_PATH_INITIAL="/system/bin"
CONFIG_NSH_ARCHINIT=y
We see that NuttX will mount the /apps filesystem as /system, via the Semihosting Host Filesystem.
That’s why it appears as /system/bin/init!
What’s inside /system/bin/init?
The RISC-V Disassembly of /system/bin/init shows this…
apps/system/nsh/nsh_main.c:52
0000006e <main>:
int main(int argc, FAR char *argv[]) {
Yep it’s the Compiled ELF Executable of the NuttX Shell nsh!
Now everything makes sense…
At Startup: NuttX tries to load /system/bin/init to start the NuttX Shell nsh
But it Fails: Because /system/bin/init doesn’t exist in the Semihosting Filesystem on Star64!
This is why Semihosting won’t work on Star64…
Why Semihosting won’t work on Star64 SBC?
Semihosting was created for Hardware Debuggers and Virtual Machine Hypervisors, like QEMU Emulator.
The pic above shows how it works: Semihosting enables a Virtual Machine (like NuttX) to “Break Out” of its Sandbox to access the Filesystem on the Host Machine / Our Computer.
(Remember our story at the top of the article? Be careful with Semihosting!)
That’s why we Enable Semihosting when we run NuttX on QEMU…
## Start NuttX on QEMU
## with Semihosting Enabled
qemu-system-riscv64 \
-kernel nuttx \
-cpu rv64 \
-smp 8 \
-M virt,aclint=on \
-semihosting \
-bios none \
-nographic
So that NuttX can access the Apps Filesystem (from previous section) as a Semihosting Filesystem! (Pic above)
(More about RISC-V Semihosting)
This won’t work on Star64?
Semihosting won’t work because NuttX for Star64 runs on Real SBC Hardware (Bare Metal)…
There’s nothing to “break out” to!
If not Semihosting… Then what?
In the world of Linux (and QEMU), there’s something cool called an Initial RAM Disk (initrd)…
It’s a RAM Disk, located in RAM (pic above)
But it’s an Initial RAM Disk. Which means there’s a Filesystem inside, preloaded with Files and Directories.
Perfect for our NuttX Apps Filesystem!
That’s awesome but where do we start?
We begin by modding NuttX QEMU to load the Initial RAM Disk…
NuttX QEMU will load an Initial RAM Disk…
Instead of using Semihosting. How?
In the previous section, we said that…
Initial RAM Disk (initrd) is a RAM Disk, located in RAM (pic above)
But it’s an Initial RAM Disk. Which means there’s a Filesystem inside, preloaded with Files and Directories.
To modify NuttX QEMU to load an Initial RAM Disk, we define the address of the RAM Disk Memory in the Linker Script: ld-kernel64.script
MEMORY
{
...
/* Added RAM Disk Memory (Max 16 MB) */
ramdisk (rwx) : ORIGIN = 0x80800000, LENGTH = 16M /* w/ cache */
}
/* Increased Page Heap for RAM Disk */
__pgheap_size = LENGTH(pgram) + LENGTH(ramdisk);
/* Previously: __pgheap_size = LENGTH(pgram); */
/* Added RAM Disk Symbols */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end = ORIGIN(ramdisk) + LENGTH(ramdisk);
(0x8080 0000 is the next available RAM Address)
At NuttX Startup, we mount the RAM Disk: qemu_rv_appinit.c
// Called at NuttX Startup
void board_late_initialize(void) {
// Mount the RAM Disk
mount_ramdisk();
// Perform board-specific initialization
#ifdef CONFIG_NSH_ARCHINIT
mount(NULL, "/proc", "procfs", 0, NULL);
#endif
}
// Mount the RAM Disk
int mount_ramdisk(void) {
// Define the ROMFS
struct boardioc_romdisk_s desc;
desc.minor = RAMDISK_DEVICE_MINOR;
desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
desc.sectsize = SECTORSIZE;
desc.image = __ramdisk_start;
// Mount the ROMFS
int ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
// Omitted: Handle Errors
(More about ROMFS in a while)
Before mounting, we copy the RAM Disk from 0x8400 0000 to ramdisk_start: qemu_rv_mm_init.c
void qemu_rv_kernel_mappings(void) {
...
// Copy RAM Disk from 0x8400 0000 to
// `__ramdisk_start` (`__ramdisk_size` bytes)
// TODO: RAM Disk must not exceed `__ramdisk_size` bytes
memcpy( // Copy the RAM Disk...
(void *)__ramdisk_start, // To RAM Disk Memory
(void *)0x84000000, // From QEMU initrd Address
(size_t)__ramdisk_size // For 16 MB
);
(More about 0x8400 0000 in a while)
(Somehow map_region crashes when we map the RAM Disk Memory)
Things get really wonky when we exceed the bounds of the RAM Disk. So we validate the bounds: fs_romfsutil.c
// While reading from RAM Disk...
static uint32_t romfs_devread32(struct romfs_mountpt_s *rm, int ndx) {
// If we're reading beyond the bounds of
// RAM Disk Memory, halt (and catch fire)
DEBUGASSERT(
&rm->rm_buffer[ndx] <
__ramdisk_start + (size_t)__ramdisk_size
);
Finally we configure NuttX QEMU to mount the Initial RAM Disk as ROMFS (instead of Semihosting): knsh64/defconfig
CONFIG_BOARDCTL_ROMDISK=y
CONFIG_BOARD_LATE_INITIALIZE=y
CONFIG_FS_ROMFS=y
CONFIG_INIT_FILEPATH="/system/bin/init"
CONFIG_INIT_MOUNT=y
CONFIG_INIT_MOUNT_FLAGS=0x1
CONFIG_INIT_MOUNT_TARGET="/system/bin"
## We removed these...
## CONFIG_FS_HOSTFS=y
## CONFIG_RISCV_SEMIHOSTING_HOSTFS=y
(How we configured NuttX for RAM Disk)
That’s it! These are the files that we modified in NuttX QEMU to load the Initial RAM Disk (without Semihosting)…
What’s ROMFS?
ROMFS is the Filesystem Format of our Initial RAM Disk. (It defines how the Files and Directories are stored in the RAM Disk)
We could have used a FAT or EXT4 or NTFS Filesystem… But ROMFS is a lot simpler for NuttX.
Why did we copy the RAM Disk from 0x8400 0000?
QEMU loads the Initial RAM Disk into RAM at 0x8400 0000…
That’s why we copied the RAM Disk from 0x8400 0000 to ramdisk_start.
Wow how did we figure out all this?
Actually we had plenty of guidance from NuttX on LiteX Arty-A7. Here’s our Detailed Analysis…
We’re ready to run our modified NuttX QEMU… That loads the Initial RAM Disk!
We build NuttX QEMU in Kernel Mode (as before). Then we generate the Initial RAM Disk initrd…
## Omitted: Build NuttX QEMU in Kernel Mode
...
## Omitted: Build Apps Filesystem for NuttX QEMU
...
## Generate the Initial RAM Disk `initrd`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
-f initrd \
-d ../apps/bin \
-V "NuttXBootVol"
(genromfs generates a ROM FS Filesystem)
This creates an Initial RAM Disk initrd (in ROMFS format) that’s 7.9 MB…
$ ls -l initrd
-rw-r--r-- 1 7902208 initrd
Finally we start QEMU and load our Initial RAM Disk…
## Start NuttX on QEMU
## with Initial RAM Disk `initrd`
qemu-system-riscv64 \
-kernel nuttx \
-initrd initrd \
-cpu rv64 \
-smp 8 \
-M virt,aclint=on \
-semihosting \
-bios none \
-nographic
And NuttX QEMU boots OK with our Initial RAM Disk yay! (Ignore the warnings)
ABC
nx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
board_late_initialize:
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
up_exit: TCB=0x802088d0 exiting
NuttShell (NSH) NuttX-12.0.3
nsh> nx_start: CPU0: Beginning Idle Loop
nsh>
We see exec_spawn warnings like this…
nsh> ls -l /system/bin/init
posix_spawn: pid=0xc0202978 path=ls file_actions=0xc0202980 attr=0xc0202988 argv=0xc0202a28
exec_spawn: ERROR: Failed to load program 'ls': -2
nxposix_spawn_exec: ERROR: exec failed: 2
-r-xr-xr-x 3278720 /system/bin/init
But it’s OK to ignore them, because “ls” is a built-in Shell Command.
(Not an Executable File from our Apps Filesystem)
Now that we figured out Initial RAM Disk on QEMU, let’s do the same for Star64…
One last thing for today: Booting NuttX on Star64 with Initial RAM Disk! (Instead of Semihosting)
We modify NuttX Star64 with the exact same steps as NuttX QEMU with Initial RAM Disk…
qemu_rv_mm_init.c: Copy RAM Disk at Startup
qemu_rv_appinit.c: Mount RAM Disk at Startup
fs_romfsutil.c: Validate RAM Disk Bounds
ld-kernel64.script: Linker Script with RAM Disk Memory
knsh64/defconfig: Build Configuration for RAM Disk
Note that we copy the Initial RAM Disk from 0x4610 0000 (instead of QEMU’s 0x8400 0000): jh7110_mm_init.c
// Copy RAM Disk from 0x4610 0000 to
// `__ramdisk_start` (`__ramdisk_size` bytes)
// TODO: RAM Disk must not exceed `__ramdisk_size` bytes
memcpy( // Copy the RAM Disk...
(void *)__ramdisk_start, // To RAM Disk Memory
(void *)0x46100000, // From U-Boot initrd Address
(size_t)__ramdisk_size // For 16 MB
);
(U-Boot Bootloader loads the RAM Disk at 0x4610 0000)
And the RAM Disk Memory is now located at 0x40A0 0000 (the next available RAM Address): ld.script
MEMORY
{
...
/* Added RAM Disk Memory (Max 16 MB) */
ramdisk (rwx) : ORIGIN = 0x40A00000, LENGTH = 16M /* w/ cache */
}
/* Increased Page Heap for RAM Disk */
__pgheap_size = LENGTH(pgram) + LENGTH(ramdisk);
/* Previously: __pgheap_size = LENGTH(pgram); */
/* Added RAM Disk Symbols */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end = ORIGIN(ramdisk) + LENGTH(ramdisk);
The other modified files are the same as for NuttX QEMU with Initial RAM Disk.
(How to increase the RAM Disk Limit)
(NuttX Apps are limited to 4 MB RAM)
(How to increase the Page Heap Size)
How do we run this on Star64?
We build NuttX Star64, generate the Initial RAM Disk initrd and copy to our TFTP Folder (for Network Booting)…
## Omitted: Build NuttX Star64
...
## Omitted: Build Apps Filesystem for NuttX Star64
...
## Generate the Initial RAM Disk `initrd`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
-f initrd \
-d ../apps/bin \
-V "NuttXBootVol"
## Copy NuttX Binary Image, Device Tree and
## Initial RAM Disk to TFTP Folder
cp nuttx.bin $HOME/tftproot/Image
cp jh7110-star64-pine64.dtb $HOME/tftproot
cp initrd $HOME/tftproot
(genromfs generates a ROM FS Filesystem)
Our Initial RAM Disk initrd (with ROMFS inside) is 7.9 MB (slightly bigger)…
$ ls -l initrd
-rw-r--r-- 1 7930880 initrd
And we boot NuttX on Star64 over TFTP or a microSD Card…
Does it work?
Now Star64 JH7110 boots OK with the Initial RAM Disk yay! (Not completely though)
Starting kernel ...
123067DFHBCI
nx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
board_late_initialize:
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
nx_start_application: ret=3
up_exit: TCB=0x404088d0 exiting
nx_start: CPU0: Beginning Idle Loop
So many questions (pic below)…
Why no NuttX Shell?
Was it started correctly?
(nx_start_application returned Process ID 3, seems OK)
Is Console Output working in NuttX Shell?
(Highly sus!)
Is our Interrupt Controller OK?
Are we using the right User Address Space?
And the right I/O Address Space?
How to handle RISC-V Timers in Supervisor Mode?
Do we need OpenSBI Timers?
We’ll find out in the next article!
No more Semihosting Problems with NuttX on Star64 JH7110 SBC!
We discovered that NuttX calls RISC-V Semihosting
(To access the Apps Filesystem)
But it crashes NuttX on Star64
(Because Semihosting won’t work on Bare Metal)
NuttX Shell lives in the NuttX Apps Filesystem
(So it’s mandatory for booting NuttX)
Thus we replaced Semihosting by Initial RAM Disk “initrd”
(And it works on Star64!)
By adapting the code from NuttX on LiteX Arty-A7
(Which we also tested on QEMU Emulator)
Now we need to figure out why NuttX Shell won’t appear…
“Star64 JH7110 + NuttX RTOS: RISC-V PLIC Interrupts and Serial I/O”
Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…
lupyuen.github.io/src/semihost.md
Previously we configured Star64’s U-Boot Bootloader to boot NuttX over TFTP…
Now we need to tweak the U-Boot Settings to boot with our Initial RAM Disk.
Star64’s U-Boot Bootloader loads our Initial RAM Disk at 0x4610 0000…
ramdisk_addr_r=0x46100000
Which means that we need to add these TFTP Commands to U-Boot Bootloader…
## Added this: Assume Initial RAM Disk is max 16 MB
setenv ramdisk_size 0x1000000
printenv ramdisk_size
saveenv
## Load Kernel and Device Tree over TFTP
tftpboot ${kernel_addr_r} ${tftp_server}:Image
tftpboot ${fdt_addr_r} ${tftp_server}:jh7110-star64-pine64.dtb
fdt addr ${fdt_addr_r}
## Added this: Load Initial RAM Disk over TFTP
tftpboot ${ramdisk_addr_r} ${tftp_server}:initrd
## Changed this: Replaced `-` by `ramdisk_addr_r:ramdisk_size`
booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r}
Which will change our U-Boot Boot Script to…
## Load the NuttX Image from TFTP Server
## kernel_addr_r=0x40200000
## tftp_server=192.168.x.x
if tftpboot ${kernel_addr_r} ${tftp_server}:Image;
then
## Load the Device Tree from TFTP Server
## fdt_addr_r=0x46000000
if tftpboot ${fdt_addr_r} ${tftp_server}:jh7110-star64-pine64.dtb;
then
## Set the RAM Address of Device Tree
## fdt_addr_r=0x46000000
if fdt addr ${fdt_addr_r};
then
## Load the Intial RAM Disk from TFTP Server
## ramdisk_addr_r=0x46100000
if tftpboot ${ramdisk_addr_r} ${tftp_server}:initrd;
then
## Boot the NuttX Image with the Initial RAM Disk and Device Tree
## kernel_addr_r=0x40200000
## ramdisk_addr_r=0x46100000
## ramdisk_size=0x1000000
## fdt_addr_r=0x46000000
booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r};
fi;
fi;
fi;
fi
Which becomes…
## Assume Initial RAM Disk is max 16 MB
setenv ramdisk_size 0x1000000
## Check that it's correct
printenv ramdisk_size
## Save it for future reboots
saveenv
## Add the Boot Command for TFTP
setenv bootcmd_tftp 'if tftpboot ${kernel_addr_r} ${tftp_server}:Image ; then if tftpboot ${fdt_addr_r} ${tftp_server}:jh7110-star64-pine64.dtb ; then if fdt addr ${fdt_addr_r} ; then if tftpboot ${ramdisk_addr_r} ${tftp_server}:initrd ; then booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r} ; fi ; fi ; fi ; fi'
## Check that it's correct
printenv bootcmd_tftp
## Save it for future reboots
saveenv
(Remember to set tftp_server and boot_targets)
Run the above commands in U-Boot.
Copy the Initial RAM Disk initrd to the TFTP Folder…
## Copy NuttX Binary Image, Device Tree and
## Initial RAM Disk to TFTP Folder
cp nuttx.bin $HOME/tftproot/Image
cp jh7110-star64-pine64.dtb $HOME/tftproot
cp initrd $HOME/tftproot
Power Star64 off and on.
NuttX now boots with our Initial RAM Disk over TFTP…
Here’s the U-Boot Log…
TFTP from server 192.168.x.x; our IP address is 192.168.x.x
Filename 'Image'.
Load address: 0x40200000
Loading: 9 MiB/s
done
Bytes transferred = 2097800 (200288 hex)
Using ethernet@16030000 device
TFTP from server 192.168.x.x; our IP address is 192.168.x.x
Filename 'jh7110-star64-pine64.dtb'.
Load address: 0x46000000
Loading: 8 MiB/s
done
Bytes transferred = 50235 (c43b hex)
Using ethernet@16030000 device
TFTP from server 192.168.x.x; our IP address is 192.168.x.x
Filename 'initrd'.
Load address: 0x46100000
Loading: 371.1 KiB/s
done
Bytes transferred = 7930880 (790400 hex)
## Flattened Device Tree blob at 46000000
Booting using the fdt blob at 0x46000000
Using Device Tree in place at 0000000046000000, end 000000004600f43a
Starting kernel ...
What if we omit the RAM Disk Size?
U-Boot won’t boot NuttX if we omit the RAM Disk Size…
## If we omit RAM Disk Size:
## Boot Fails
$ booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
Wrong Ramdisk Image Format
Ramdisk image is corrupt or invalid
So we hardcode a maximum RAM Disk Size of 16 MB…
## If we assume RAM Disk Size is max 16 MB:
## Boots OK
$ booti ${kernel_addr_r} ${ramdisk_addr_r}:0x1000000 ${fdt_addr_r}
Let’s talk about the NuttX Configuration for Initial RAM Disk…
Earlier we configured NuttX QEMU and NuttX Star64 to boot with our Initial RAM Disk…
Here are the steps for updating the NuttX Build Configuration in make menuconfig…
Board Selection > Enable boardctl() interface > Enable application space creation of ROM disks
RTOS Features > RTOS hooks > Custom board late initialization
File Systems > ROMFS file system
RTOS Features > Tasks and Scheduling > Auto-mount init file system
Set to /system/bin
Build Setup > Debug Options > File System Debug Features > File System Error, Warnings and Info Output
Disable: File Systems > Host File System
Manually delete from nsh/defconfig…
CONFIG_HOST_MACOS=y
CONFIG_INIT_MOUNT_DATA="fs=../apps"
CONFIG_INIT_MOUNT_FSTYPE="hostfs"
CONFIG_INIT_MOUNT_SOURCE=""
The steps above will produce the updated Build Configuration Files…
NuttX for QEMU: knsh64/defconfig
NuttX for Star64: nsh/defconfig
We need the RAM Disk Address for RISC-V QEMU…
Can we enable logging for RISC-V QEMU?
Yep we use this QEMU Option: -trace "*"
## Start NuttX on QEMU
## with Initial RAM Disk `initrd`
qemu-system-riscv64 \
-semihosting \
-M virt,aclint=on \
-cpu rv64 \
-smp 8 \
-bios none \
-kernel nuttx \
-initrd initrd \
-nographic \
-trace "*"
In the QEMU Command above, we load the Initial RAM Disk initrd.
To discover the RAM Address of the Initial RAM Disk, we check the QEMU Trace Log…
resettablloader_write_rom nuttx
ELF program header segment 0:
@0x80000000 size=0x2b374 ROM=0
loader_write_rom nuttx
ELF program header segment 1:
@0x80200000 size=0x2a1 ROM=0
loader_write_rom initrd:
@0x84000000 size=0x2fc3e8 ROM=0
loader_write_rom fdt:
@0x87000000 size=0x100000 ROM=0
This says that QEMU loads our Initial RAM Disk initrd at 0x8400 0000
(And QEMU loads our Kernel at 0x8000 0000, Device Tree at 0x8700 0000)
We set the RAM Address of the Initial RAM Disk here…
We thought the Initial RAM Disk Address could be discovered from the Device Tree for RISC-V QEMU. But nope it’s not there…
To dump the Device Tree for RISC-V QEMU, we specify dumpdtb…
## Dump Device Tree for RISC-V QEMU
qemu-system-riscv64 \
-semihosting \
-M virt,aclint=on,dumpdtb=qemu-riscv64.dtb \
-cpu rv64 \
-smp 8 \
-bios none \
-kernel nuttx \
-nographic
## Convert Device Tree to text format
dtc \
-o qemu-riscv64.dts \
-O dts \
-I dtb \
qemu-riscv64.dtb
(dtc decompiles a Device Tree)
This produces the Device Tree for RISC-V QEMU…
Which is helpful for browsing the Memory Addresses of I/O Peripherals in QEMU.
Earlier we modified NuttX QEMU and NuttX Star64 to load our Initial RAM Disk…
We did it with plenty of guidance from NuttX on LiteX Arty-A7, below is our Detailed Analysis.
To generate the RAM Disk for LiteX Arty-A7, we run this command…
## Generate the Initial RAM Disk `romfs.img`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
-f romfs.img \
-d ../apps/bin \
-V "NuttXBootVol"
(About NuttX RAM Disks and ROM Disks)
LiteX Memory Map tells us where the RAM Disk is loaded: 0x40C0 0000
"romfs.img": "0x40C00000",
"nuttx.bin": "0x40000000",
"opensbi.bin": "0x40f00000"
This is the LiteX Build Configuration for mounting the RAM Disk: knsh/defconfig
CONFIG_BOARDCTL_ROMDISK=y
CONFIG_BOARD_LATE_INITIALIZE=y
CONFIG_BUILD_KERNEL=y
CONFIG_FS_ROMFS=y
CONFIG_INIT_FILEPATH="/system/bin/init"
CONFIG_INIT_MOUNT=y
CONFIG_INIT_MOUNT_FLAGS=0x1
CONFIG_INIT_MOUNT_TARGET="/system/bin"
CONFIG_LITEX_APPLICATION_RAMDISK=y
CONFIG_NSH_FILE_APPS=y
CONFIG_NSH_READLINE=y
CONFIG_PATH_INITIAL="/system/bin"
CONFIG_RAM_SIZE=4194304
CONFIG_RAM_START=0x40400000
CONFIG_RAW_BINARY=y
CONFIG_SYSTEM_NSH_PROGNAME="init"
CONFIG_TESTING_GETPRIME=y
Which is consistent with the NuttX Doc on NSH Start-Up Script…
CONFIG_DISABLE_MOUNTPOINT not set
CONFIG_FS_ROMFS enabled
We mount the RAM Disk at LiteX Startup: litex_appinit.c
void board_late_initialize(void)
{
#ifdef CONFIG_LITEX_APPLICATION_RAMDISK
litex_mount_ramdisk();
#endif
litex_bringup();
}
Which calls litex_mount_ramdisk to mount the RAM Disk: litex_ramdisk.c
#ifndef CONFIG_BUILD_KERNEL
#error "Ramdisk usage is intended to be used with kernel build only"
#endif
#define SECTORSIZE 512
#define NSECTORS(b) (((b) + SECTORSIZE - 1) / SECTORSIZE)
#define RAMDISK_DEVICE_MINOR 0
// Mount a ramdisk defined in the ld-kernel.script to /dev/ramX.
// The ramdisk is intended to contain a romfs with applications which can
// be spawned at runtime.
int litex_mount_ramdisk(void)
{
int ret;
struct boardioc_romdisk_s desc;
desc.minor = RAMDISK_DEVICE_MINOR;
desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
desc.sectsize = SECTORSIZE;
desc.image = __ramdisk_start;
ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
if (ret < 0)
{
syslog(LOG_ERR, "Ramdisk register failed: %s\n", strerror(errno));
syslog(LOG_ERR, "Ramdisk mountpoint /dev/ram%d\n",
RAMDISK_DEVICE_MINOR);
syslog(LOG_ERR, "Ramdisk length %u, origin %x\n",
(ssize_t)__ramdisk_size,
(uintptr_t)__ramdisk_start);
}
return ret;
}
ramdisk_start and ramdisk_size are defined in the LiteX Memory Map: board_memorymap.h
/* RAMDisk */
#define RAMDISK_START (uintptr_t)__ramdisk_start
#define RAMDISK_SIZE (uintptr_t)__ramdisk_size
/* ramdisk (RW) */
extern uint8_t __ramdisk_start[];
extern uint8_t __ramdisk_size[];
And also in the LiteX Linker Script: ld-kernel.script
MEMORY
{
kflash (rx) : ORIGIN = 0x40000000, LENGTH = 4096K /* w/ cache */
ksram (rwx) : ORIGIN = 0x40400000, LENGTH = 4096K /* w/ cache */
pgram (rwx) : ORIGIN = 0x40800000, LENGTH = 4096K /* w/ cache */
ramdisk (rwx) : ORIGIN = 0x40C00000, LENGTH = 4096K /* w/ cache */
}
...
/* Page heap */
__pgheap_start = ORIGIN(pgram);
__pgheap_size = LENGTH(pgram) + LENGTH(ramdisk);
/* Application ramdisk */
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end = ORIGIN(ramdisk) + LENGTH(ramdisk);
Note that pgheap_size needs to include ramdisk size.