📝 26 Nov 2023
In Asia the wise folks say…
“One can hide on a certain day but cannot hide for a long time”
“躲过初一，躲不过十五”
In other words…
“Transformers? More than meets the eye!”
In this article, we go behind the shadow puppetry (wayang kulit) and deceptive simplicity of NuttX Applications inside Apache NuttX RTOS (Real-Time Operating System) for Pine64 Ox64 BL808 64-bit RISC-V SBC (pic below)…
What’s inside the simplest NuttX App
How NuttX Apps make RISC-V System Calls to NuttX Kernel
Virtual Memory for NuttX Apps
Loading of ELF Executables by NuttX Kernel
Bundling of NuttX Apps into the Initial RAM Disk
How we found the right spot to park our Initial RAM Disk
What happens inside the simplest NuttX App?
// From https://github.com/apache/nuttx-apps/blob/master/examples/hello/hello_main.c#L36-L40
int main(int argc, FAR char *argv[]) {
printf("Hello, World!!\n");
return 0;
}
Let’s find out! First we build NuttX for Ox64 BL808 SBC.
Which produces this ELF Executable for our NuttX App…
## ELF Executable for `hello` looks big...
$ ls -l ../apps/bin/hello
-rwxr-xr-x 518,192 ../apps/bin/hello
## Though not much inside, mostly Debug Info...
$ riscv64-unknown-elf-size ../apps/bin/hello
text data bss dec hex filename
3814 8 4 3826 ef2 ../apps/bin/hello
## Dump the RISC-V Disassembly to `hello.S`
$ riscv64-unknown-elf-objdump \
--syms --source --reloc --demangle --line-numbers --wide \
--debugging \
../apps/bin/hello \
>hello.S \
2>&1
Here’s the RISC-V Disassembly of our NuttX App: hello.S
## Omitted: _start() prepares for signals (sig_trampoline) and calls main()
003e <main>:
int main(int argc, FAR char *argv[]) {
3e: 1141 addi sp,sp,-16 ## Subtract 16 from Stack Pointer
## Set Register A0 (Arg 0) to "Hello, World!!\n"
40: 00000517 auipc a0,0x0 40: R_RISCV_PCREL_HI20 .LC0
44: 00050513 mv a0,a0 44: R_RISCV_PCREL_LO12_I .L0
printf("Hello, World!!\n");
48: e406 sd ra,8(sp) ## Save Return Address to Stack Pointer, Offset 8
4a: 00000097 auipc ra,0x0 4a: R_RISCV_CALL puts
4e: 000080e7 jalr ra # 4a <.LVL1+0x2> ## Call puts()
return 0;
52: 60a2 ld ra,8(sp) ## Load Return Address from Stack Pointer, Offset 8
54: 4501 li a0,0 ## Set Return Value to 0
56: 0141 addi sp,sp,16 ## Add 16 to Stack Pointer
58: 8082 ret ## Return to caller: _start()
## Followed by the code for puts(), lib_fwrite_unlocked(), write(), ...
In the RISC-V Disassembly, we see that main calls…
puts which calls…
lib_fwrite_unlocked which calls…
write which calls…
NuttX Kernel to print “Hello World”
How will write call the NuttX Kernel? We’ll see soon!
This code looks broken…
printf("Hello, World!!\n");
## Load Register RA with Program Counter + 0x0
4a: 00000097 auipc ra, 0x0
## Call the function in Register RA: puts()
4e: 000080e7 jalr ra
We break it down…
auipc sets Register RA to…
Program Counter + 0x0
jalr jumps to the Function pointed by Register RA…
Which we expect to be puts
Shouldn’t auipc add the Offset of puts?
Ah that’s because we’re looking at Relocatable Code!
The auipc Offset will be fixed up by the NuttX ELF Loader when it loads our NuttX App for execution.
In our RISC-V Disassembly, the Relocation Info shows that 0x0 will be replaced by the Offset of puts…
printf("Hello, World!!\n");
## Why load Register RA with Program Counter + 0x0?
## Gotcha! 0x0 will be changed to the Offset of puts()
4a: 00000097 auipc ra, 0x0
4a: R_RISCV_CALL puts
## Call the function in Register RA: puts()
## Which will work when ELF Loader fixes the Offset of puts()
4e: 000080e7 jalr ra # 4a <.LVL1+0x2>
Therefore we’re all good! (Eventually)
Why puts instead of printf?
The GCC Compiler has cleverly optimised away printf to become puts.
If we do this (and foil the GCC Compiler)…
// Nope, GCC Compiler won't change printf() to puts()
printf(
"Hello, World %s!!\n", // Meaningful Format String
"Luppy" // Makes it complicated
);
Then printf will appear in our RISC-V Disassembly.
We circle back to write…
Our app will print something to the console…
But NuttX Apps can’t write directly to the Serial Device right?
Nope!
NuttX Apps run in RISC-V User Mode…
Which can’t access the Serial Device (and other resources) controlled by NuttX Kernel…
Which runs in RISC-V Supervisor Mode
That’s why “write” should trigger a System Call to the NuttX Kernel, jumping from RISC-V User Mode to Supervisor Mode.
(And write to the Serial Device, pic above)
Will NuttX Apps need Special Coding to make System Calls?
Not at all! The System Call is totally transparent to our app…
Our NuttX App will call a normal function named “write”…
That pretends to be the actual “write” function in NuttX Kernel…
By forwarding the “write” function call (and parameters)…
Through a RISC-V System Call
What’s this “forwarding” to a System Call?
This forwarding happens inside a Proxy Function that’s auto-generated during NuttX Build…
// From nuttx/syscall/proxies/PROXY_write.c
// Auto-Generated Proxy for `write`
// Looks like the Kernel `write`, though it's actually a System Call
ssize_t write(int parm1, FAR const void * parm2, size_t parm3) {
// Make a System Call with 3 parameters...
return (ssize_t) sys_call3(
(unsigned int) SYS_write, // System Call Number (63 = `write`)
(uintptr_t) parm1, // File Descriptor (1 = Standard Output)
(uintptr_t) parm2, // Buffer to be written
(uintptr_t) parm3 // Number of bytes to write
);
}
Our NuttX App (implicitly) calls this Proxy Version of “write” (that pretends to be the Kernel “write”)…
// Our App calls the Proxy Function...
int ret = write(
1, // File Descriptor (1 = Standard Output)
"Hello, World!!\n", // Buffer to be written
15 // Number of bytes to write
);
Which triggers a System Call to the Kernel.
(Indeed “More than meets the eye!”)
What’s sys_call3?
It makes a System Call (to NuttX Kernel) with 3 Parameters: syscall.h
// Make a System Call with 3 parameters
uintptr_t sys_call3(
unsigned int nbr, // System Call Number (63 = `write`)
uintptr_t parm1, // First Parameter
uintptr_t parm2, // Second Parameter
uintptr_t parm3 // Third Parameter
) {
// Pass the Function Number and Parameters in
// Registers A0 to A3
register long r0 asm("a0") = (long)(nbr);
register long r1 asm("a1") = (long)(parm1);
register long r2 asm("a2") = (long)(parm2);
register long r3 asm("a3") = (long)(parm3);
// `ecall` will jump from RISC-V User Mode
// to RISC-V Supervisor Mode
// to execute the System Call.
// Input + Output Registers: A0 to A3
// Clobbers the Memory
asm volatile
(
"ecall"
:: "r"(r0), "r"(r1), "r"(r2), "r"(r3)
: "memory"
);
// No-operation, does nothing
asm volatile("nop" : "=r"(r0));
// Return the result from Register A0
return r0;
}
ecall is the RISC-V Instruction that jumps from RISC-V User Mode to Supervisor Mode…
That allows NuttX Kernel to execute the actual “write” function, with the real Serial Device.
(We’ll explain how)
Why the no-op after ecall?
We’re guessing: It might be reserved for special calls to NuttX Kernel in future.
(Similar to ebreak for Semihosting)
Every System Call to NuttX Kernel has its own Proxy Function?
Yep! We can see the Auto-Generated Proxy Functions for each System Call…
## Proxy Functions called by `hello` app
$ grep PROXY hello.S
PROXY__assert.c
PROXY__exit.c
PROXY_clock_gettime.c
PROXY_gettid.c
PROXY_lseek.c
PROXY_nxsem_wait.c
PROXY_sem_clockwait.c
PROXY_sem_destroy.c
PROXY_sem_post.c
PROXY_sem_trywait.c
PROXY_task_setcancelstate.c
PROXY_write.c
Next we figure out how System Calls will work…
Our App makes an ecall to jump to NuttX Kernel (pic above)…
What happens on the other side?
Remember the Proxy Function from earlier? Now we do the exact opposite in our Stub Function (that runs in the Kernel)…
// From nuttx/syscall/stubs/STUB_write.c
// Auto-Generated Stub File for `write`
// This runs in NuttX Kernel triggered by `ecall`.
// We make the actual call to `write`.
// (`nbr` is Offset in Stub Lookup Table, unused)
uintptr_t STUB_write(int nbr, uintptr_t parm1, uintptr_t parm2, uintptr_t parm3) {
// Call the Kernel version of `write`
return (uintptr_t) write(
(int) parm1, // File Descriptor (1 = Standard Output)
(FAR const void *) parm2, // Buffer to be written
(size_t) parm3 // Number of bytes to write
); // Return the result to the App
}
Thus our NuttX Build auto-generates 2 things…
Proxy Function (runs in NuttX Apps)
Stub Function (runs in NuttX Kernel)
This happens for every System Call exposed by NuttX Kernel…
## Stub Functions in NuttX Kernel
$ grep STUB nuttx.S
STUB__assert.c
STUB__exit.c
STUB_boardctl.c
STUB_chmod.c
STUB_chown.c
...
(More about Proxy and Stub Functions)
Who calls STUB_write?
When our NuttX App makes an ecall, it triggers IRQ 8 (RISCV_IRQ_ECALLU) that’s handled by…
riscv_swint which calls…
dispatch_syscall which calls the Kernel Stub Function (STUB_write) and…
sys_call2 with A0 set to SYS_syscall_return (3) which calls…
riscv_perform_syscall which calls…
riscv_swint with IRQ 0, to return from the ecall
How will dispatch_syscall know which Stub Function to call?
Remember that our Proxy Function (in NuttX App) passes the System Call Number for “write”?
// From nuttx/syscall/proxies/PROXY_write.c
// Auto-Generated Proxy for `write`, called by NuttX App
ssize_t write(int parm1, FAR const void * parm2, size_t parm3) {
// Make a System Call with 3 parameters...
return (ssize_t) sys_call3(
(unsigned int) SYS_write, // System Call Number (63 = `write`)
...
dispatch_syscall (in NuttX Kernel) will look up the System Call Number in the Stub Lookup Table. And fetch the Stub Function to call.
How did we figure out that 63 is the System Call Number for “write”?
OK this gets tricky. Below is the Enum that defines all System Call Numbers: syscall.h and syscall_lookup.h
// System Call Enum sequentially assigns
// all System Call Numbers (8 to 147-ish)
enum {
...
SYSCALL_LOOKUP(close, 1) // 1 Parameter
SYSCALL_LOOKUP(ioctl, 3) // 3 Parameters
SYSCALL_LOOKUP(read, 3) // 3 Parameters
SYSCALL_LOOKUP(write, 3) // 3 Parameters
...
};
However it’s an Enum, numbered sequentially from 8 to 147-ish. We won’t literally see 63 in the NuttX Source Code.
Then we lookup the Debug Info in the RISC-V Disassembly for NuttX Kernel: nuttx.S
Abbrev Number: 6 (DW_TAG_enumerator)
DW_AT_name : SYS_write
DW_AT_const_value : 63
Whoomp there it is! Says here that “write” is System Call #63.
That’s an odd way to define System Call Numbers…
Yeah it’s not strictly an immutable ABI like Linux, because our System Call Numbers may change! It depends on the Build Options that we select.
(ABI means Application Binary Interface)
Though there’s a jolly good thing: It’s super simple to experiment with new System Calls!
(Just add to NuttX System Calls)
This looks complicated… It works right?
Yep we have solid evidence, from NuttX for Ox64 BL808 SBC!
Remember to enable System Call Logging in “make menuconfig”…
Build Setup
> Debug Options
> Syscall Debug Features
> Enable "Syscall Warning, Error and Info"
Watch what happens when we boot NuttX on Ox64 (pic above)…
Our app (NuttX Shell) begins by printing something to the console.
It makes an ecall for System Call #63 “write”.
Which triggers IRQ 8 and jumps to NuttX Kernel
riscv_dispatch_irq: irq=8
riscv_swint: Entry: regs: 0x5040bcb0 cmd: 63
EPC: 800019b2
A0: 003f A1: 0001 A2: 8000ad00 A3: 001e
The RISC-V Registers look familiar…
A0 is 0x3F
(System Call #63 for “write”)
A1 is 1
(File Descriptor #1 for Standard Output)
A2 is 0x8000_AD00
(Buffer to be written)
A3 is 0x1E
(Number of bytes to write)
NuttX Kernel calls our Stub Function STUB_write…
riscv_swint: SWInt Return: 37
STUB_write: nbr=440, parm1=1, parm2=8000ad00, parm3=1e
NuttShell (NSH) NuttX-12.0.3
Which calls Kernel “write” and prints the text: “NuttShell”
Then NuttX Kernel completes the ecall…
riscv_swint: Entry: regs: 0x5040baa0 cmd: 3
EPC: 80001a6a
A0: 0003 A1: 5040bbec A2: 001e A3: 0000
riscv_swint: SWInt Return: 1e
A0 is 3
(Return from System Call: SYS_syscall_return)
A2 is 0x1E
(Number of bytes written)
And returns the result 0x1E to our NuttX App. (Via sret)
Our NuttX App has successfully made a System Call on Ox64 yay!
NuttX Kernel prints the buffer at 0x8000_AD00…
It doesn’t look like a RAM Address?
That’s a Virtual Memory Address…
TLDR? No worries…
Kernel RAM is at 0x5000_0000
Which gets dished out dynamically to NuttX Apps
And becomes Virtual Memory at 0x8000_0000 (pic above)
Hence our NuttX App has passed a chunk of its own Virtual Memory. And NuttX Kernel happily prints it!
Huh? NuttX Kernel can access Virtual Memory?
NuttX uses 2 sets of Page Tables: Kernel Page Table and User Page Table.
(User Page Table defines the Virtual Memory for NuttX Apps)
According to the NuttX Log, the Kernel swaps the RISC-V SATP Register from Kernel Page Table to User Page Table…
And doesn’t swap back!
Which means the User Page Table is still in effect!
And the Virtual Memory at 0x8000_0000 is perfectly accessible by the Kernel.
There’s a catch: RISC-V Supervisor Mode (NuttX Kernel) may access the Virtual Memory mapped to RISC-V User Mode (NuttX Apps)…
Only if the SUM Bit is set in SSTATUS Register!
And that’s absolutely hunky dory because at NuttX Startup, nx_start calls…
up_initial_state which calls…
riscv_set_idleintctx to set the SUM Bit in SSTATUS Register
That’s why NuttX Kernel can access Virtual Memory (passed by NuttX Apps) at 0x8000_0000!
Clickable Version of NuttX Flow
Alrighty NuttX Apps can call NuttX Kernel…
But how does NuttX Kernel start a NuttX App?
Previously we walked through the Boot Sequence for NuttX…
Right after that, NuttX Bringup (nx_bringup) calls (pic above)…
Create Init Thread: nx_create_initthread (to create the Init Thread) which calls…
Start App: nx_start_application (to start NuttX Shell) which calls…
Exec Spawn: exec_spawn (to start the app) which calls…
Exec Internal: exec_internal (to start the app) which calls…
Load Module: load_module (to load the app, see below) and…
Execute Module: exec_module (to execute the app)
To load a NuttX App module: load_module calls…
Load Absolute Module: load_absmodule (to load an absolute path) which calls…
Load Binary Format: binfmt_s.load (to load a binary module) which calls…
ELF Loader: g_elfbinfmt (to load the ELF File, see below)
To load the ELF File: ELF Loader g_elfbinfmt calls…
Load ELF Binary: elf_loadbinary (to load the ELF Binary) which calls…
Load ELF: elf_load (to load the ELF Binary) which calls…
Allocate Address Env: elf_addrenv_alloc (to allocate the Address Env) which calls…
Create Address Env: up_addrenv_create (to create the Address Env) which calls…
(Also calls mmu_satp_reg to set SATP Register)
Create MMU Region: create_region (to create the MMU Region) which calls…
Set MMU Page Table Entry: mmu_ln_setentry (to populate the Page Table Entries)
There’s plenty happening inside Execute Module: exec_module. Too bad we won’t explore today.
(Clickable Version of NuttX Flow)
Initial RAM Disk for Star64 JH7110
OK we know how NuttX Kernel starts a NuttX App…
But where are the NuttX Apps stored?
Right now we’re working with the Early Port of NuttX to Ox64 BL808 SBC. We can’t access the File System in the microSD Card.
All we have: A File System that lives in RAM and contains our NuttX Shell + NuttX Apps.
That’s our Initial RAM Disk: initrd
## Build the Apps Filesystem
make -j 8 export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j 8 import
popd
## Generate the Initial RAM Disk `initrd`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
-f initrd \
-d ../apps/bin \
-V "NuttXBootVol"
How to load the Initial RAM Disk from microSD to RAM?
U-Boot Bootloader will do it for us!
Two ways that U-Boot can load the Initial RAM Disk from microSD…
Load the Initial RAM Disk from a Separate File: initrd (similar to Star64, pic above)
This means we modify the U-Boot Script: boot-pine64.scr
And make it load the initrd file into RAM.
(Which is good for separating the NuttX Kernel and NuttX Apps)
OR…
Append the Initial RAM Disk to the NuttX Kernel Image
U-Boot Bootloader will load (one-shot into RAM) the NuttX Kernel + Initial RAM Disk.
And we reuse the existing U-Boot Config on the microSD Card: extlinux/extlinux.conf
(Which might be more efficient for our Limited RAM)
Since Ox64 is low on RAM, we’ll do the Second Method (Append to Kernel). Like this…
## Export the NuttX Kernel to `nuttx.bin`
riscv64-unknown-elf-objcopy \
-O binary \
nuttx \
nuttx.bin
## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero >/tmp/nuttx.pad
## Append Padding and Initial RAM Disk to NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
>Image
## Overwrite the Linux Image on Ox64 microSD
cp Image "/Volumes/NO NAME/"
## U-Boot Bootloader will load NuttX Kernel and
## Initial RAM Disk into RAM
This is how we made it work…
(Ox64 can boot NuttX from Flash Memory)
We appended the Initial RAM Disk to NuttX Kernel (pic above)…
U-Boot Bootloader loads the NuttX Kernel + Initial RAM Disk into RAM…
How in RAM will NuttX Kernel locate the Initial RAM Disk?
Our Initial RAM Disk follows the ROM File System Format (ROM FS). We search our RAM for the ROM File System by its Magic Number.
Then we copy it into the designated Memory Region for mounting: bl808_start.c
// Locate the Initial RAM Disk and copy to the designated Memory Region
void bl808_copy_ramdisk(void) {
// After _edata, search for "-rom1fs-". This is the RAM Disk Address.
// Limit search to 256 KB after Idle Stack Top.
const char *header = "-rom1fs-";
uint8_t *ramdisk_addr = NULL;
for (uint8_t *addr = _edata; addr < (uint8_t *)BL808_IDLESTACK_TOP + (256 * 1024); addr++) {
if (memcmp(addr, header, strlen(header)) == 0) {
ramdisk_addr = addr;
break;
}
}
// Stop if RAM Disk is missing
if (ramdisk_addr == NULL) { PANIC(); }
// RAM Disk must be after Idle Stack, to prevent overwriting
if (ramdisk_addr <= (uint8_t *)BL808_IDLESTACK_TOP) { PANIC(); }
// Read the Filesystem Size from the next 4 bytes, in Big Endian
// Add 0x1F0 to Filesystem Size
const uint32_t size =
(ramdisk_addr[8] << 24) +
(ramdisk_addr[9] << 16) +
(ramdisk_addr[10] << 8) +
ramdisk_addr[11] +
0x1F0;
// Filesystem Size must be less than RAM Disk Memory Region
if (size > (size_t)__ramdisk_size) { PANIC(); }
// Copy the Filesystem bytes to RAM Disk Memory Region
// Warning: __ramdisk_start overlaps with ramdisk_addr + size
// Which doesn't work with memcpy.
// Sadly memmove is aliased to memcpy, so we implement memmove ourselves
bl808_copy_overlap((void *)__ramdisk_start, ramdisk_addr, size);
}
(More about edata, Idle Stack and bl808_copy_overlap in the next section)
Why did we copy Initial RAM Disk to ramdisk_start?
ramdisk_start points to the Memory Region that we reserved for mounting our RAM Disk.
It’s defined in the NuttX Linker Script: ld.script
/* Memory Region for Mounting RAM Disk */
ramdisk (rwx) : ORIGIN = 0x50A00000, LENGTH = 16M
...
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end = ORIGIN(ramdisk) + LENGTH(ramdisk);
Who calls the code above?
We locate and copy the Initial RAM Disk at the very top of our NuttX Start Code.
This just after erasing the BSS (Global and Static Variables), in case we need to print some messages and it uses Global and Static Variables: bl808_start.c
// NuttX Start Code
void bl808_start(int mhartid) {
// Clear the BSS for Global and Static Variables
bl808_clear_bss();
// Copy the RAM Disk
bl808_copy_ramdisk();
Later during startup, we mount the RAM Disk from the Memory Region: bl808_appinit.c
// After NuttX has booted...
void board_late_initialize(void) {
// Mount the RAM Disk
mount_ramdisk();
}
// Mount the RAM Disk
int mount_ramdisk(void) {
desc.minor = RAMDISK_DEVICE_MINOR;
desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
desc.sectsize = SECTORSIZE;
desc.image = __ramdisk_start;
ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);
(How NuttX calls board_late_initialize)
All this works great: NuttX mounts our RAM Disk successfully, and starts the ELF Executable for NuttX Shell!
bl808_copy_ramdisk:
_edata=0x50400258, _sbss=0x50400290, _ebss=0x50407000, BL808_IDLESTACK_TOP=0x50407c00
ramdisk_addr=0x50408288
size=8192016
Before Copy: ramdisk_addr=0x50408288
After Copy: __ramdisk_start=0x50a00000
...
elf_initialize: Registering ELF
uart_register: Registering /dev/console
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
load_absmodule: Loading /system/bin/init
elf_loadbinary: Loading file: /system/bin/init
elf_init: filename: /system/bin/init loadinfo: 0x5040c618
elf_read: Read 64 bytes from offset 0
(“system/bin/init” is the NuttX Shell)
Last thing for today: The mysterious 64 KB padding…
Between NuttX Kernel and Initial RAM Disk…
Why did we pad 64 KB of zeroes? (Pic above)
## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero >/tmp/nuttx.pad
## Append Padding and Initial RAM Disk to NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
>Image
## U-Boot Bootloader will load NuttX Kernel and
## Initial RAM Disk into RAM
U-Boot Bootloader will load our Initial RAM Disk into RAM. However it’s dangerously close to BSS Memory (Global and Static Variables) and Kernel Stack.
There’s a risk that our Initial RAM Disk will be contaminated by BSS and Stack. This is how we found a clean, safe space for our Initial RAM Disk (pic above)…
We inspect the NuttX Log and the NuttX Linker Script…
// End of Data Section
_edata=0x50400258
// Start of BSS Section
_sbss=0x50400290
// End of BSS Section
_ebss=0x50407000
// Top of Kernel Idle Stack
BL808_IDLESTACK_TOP=0x50407c00
// We located the initrd after the Top of Idle Stack
ramdisk_addr=0x50408288, size=8192016
// And we copied initrd to the Memory Region for the RAM Disk
__ramdisk_start=0x50a00000
Or graphically…
| Memory Region | Start | End |
|---|---|---|
| Data Section | 0x5040_0257 | |
| BSS Section | 0x5040_0290 | 0x5040_6FFF |
| Kernel Idle Stack | 0x5040_7BFF | |
| Initial RAM Disk | 0x5040_8288 | 0x50BD_8297 |
| RAM Disk Region | 0x50A0_0000 | 0x519F_FFFF |
(NuttX will mount the RAM Disk from RAM Disk Region)
(Which overlaps with Initial RAM Disk!)
This says…
NuttX Kernel nuttx.bin terminates at edata.
(End of Data Section)
If we append Initial RAM Disk initrd directly to the end of nuttx.bin…
It will collide with the BSS Section and the Kernel Idle Stack.
And initrd will get overwritten when NuttX runs the Boot Code and Start Code.
(Boot Code uses the Kernel Idle Stack. Start Code erases the BSS)
Best place to append initrd is after the Kernel Idle Stack.
(Roughly 32 KB after edata)
That’s why we inserted a padding of 64 KB between nuttx.bin and initrd.
(Surely initrd won’t collide with BSS and Kernel Idle Stack)
From the previous section, our code locates initrd.
(Searching for the ROM FS Magic Number)
And copies initrd to the RAM Disk Region.
Finally NuttX mounts the RAM Disk from RAM Disk Region.
NuttX Kernel starts the NuttX Shell correctly from the Mounted RAM Disk.
(Everything goes well, nothing gets contaminated)
Yep our 64 KB Padding looks legit!
64 KB sounds arbitrary. What if the parameters change?
We have Runtime Checks to catch problems: bl808_start.c
// Stop if RAM Disk is missing
if (ramdisk_addr == NULL) { _err("Missing RAM Disk. Check the initrd padding."); PANIC(); }
// RAM Disk must be after Idle Stack, to prevent overwriting
if (ramdisk_addr <= (uint8_t *)BL808_IDLESTACK_TOP) { _err("RAM Disk must be after Idle Stack. Increase the initrd padding by %ul bytes.", (size_t)BL808_IDLESTACK_TOP - (size_t)ramdisk_addr); PANIC(): }
// Filesystem Size must be less than RAM Disk Memory Region
if (size > (size_t)__ramdisk_size) { _err("RAM Disk Region too small"); PANIC(); }
Why call bl808_copy_overlap to copy initrd to RAM Disk Region? Why not memcpy?
That’s because initrd overlaps with RAM Disk Region! (See above)
memcpy won’t work with Overlapping Memory Regions. Thus we added this: bl808_start.c
// Copy a chunk of memory from `src` to `dest`.
// `dest` overlaps with the end of `src`.
// From libs/libc/string/lib_memmove.c
void *bl808_copy_overlap(void *dest, const void *src, size_t count) {
if (dest <= src) { _err("dest and src should overlap"); PANIC(); }
char *d = (char *) dest + count;
char *s = (char *) src + count;
// TODO: This needs to be `volatile` or GCC Compiler will replace this by memcpy. Very strange.
while (count--) {
d -= 1; s -= 1;
volatile char c = *s;
*d = c;
}
return dest;
}
We’re sure that it works?
We called verify_image to do a simple Integrity Check on initrd, before and after copying: jh7110_start.c
// Before Copy: Verify the RAM Disk Image to be copied
verify_image(ramdisk_addr);
// Copy the Filesystem bytes to RAM Disk Memory Region
// Warning: __ramdisk_start overlaps with ramdisk_addr + size
// Which doesn't work with memcpy.
// Sadly memmove is aliased to memcpy, so we implement memmove ourselves
bl808_copy_overlap((void *)__ramdisk_start, ramdisk_addr, size);
// After Copy: Verify the copied RAM Disk Image
verify_image(__ramdisk_start);
(verify_image searches for a specific byte)
That’s how we discovered that memcpy doesn’t work. And our bl808_copy_overlap works great for the Initial RAM Disk and NuttX Shell! (Pic below)
Like we said at the top of the article…
“One can hide on the First of the Month… But not on the Fifteenth!”
Today we unravelled the inner workings of NuttX Applications for Ox64 BL808 RISC-V SBC…
We studied the internals of the simplest NuttX App
How NuttX Apps make System Calls with ecall, Proxy Functions and Stub Functions
Why NuttX Kernel can access the Virtual Memory of NuttX Apps
How NuttX Kernel loads ELF Executables
Bundling of NuttX Apps into the Initial RAM Disk in ROM FS Format
And making sure our RAM Disk is safe and sound after loading by U-Boot Bootloader
We’ll do much more for NuttX on Ox64 BL808, stay tuned for updates!
(Like the fixing of UART Interrupts)
Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…