đź“ť 7 Aug 2023
Apache NuttX Real-Time Operating System (RTOS) now officially supports Pine64 Star64 64-bit RISC-V Single-Board Computer! (Pic below)
(Works on StarFive VisionFive2 SBC too, since both SBCs are based on StarFive JH7110 SoC)
In this article we explain how we created the First Release of NuttX for Star64 JH7110…
Building NuttX for Star64
Creating a Bootable microSD with NuttX Kernel and Initial RAM Disk
What happens during NuttX Startup
Adding the NuttX Arch (JH7110) and NuttX Board (Star64)
Upcoming Features for NuttX on Star64
Which is probably helpful for folks who wish to…
Add a new NuttX Arch (SoC) or NuttX Board
Create NuttX Drivers (or NuttX Apps) for Star64 (or VisionFive2)
Or simply understand how we boot a Modern SBC from scratch!
(Watch the Demo Video on YouTube)
Let’s walk through the steps to build NuttX for Star64…
Install the NuttX Build Prerequisites, skip the RISC-V Toolchain…
Download the RISC-V Toolchain riscv64-unknown-elf…
“Download Toolchain for 64-bit RISC-V”
Add the downloaded toolchain to the PATH Environment Variable.
Check the RISC-V Toolchain:
$ riscv64-unknown-elf-gcc -v
Download the NuttX Repositories…
$ mkdir nuttx
$ cd nuttx
$ git clone https://github.com/apache/nuttx.git nuttx
$ git clone https://github.com/apache/nuttx-apps apps
Configure and build the NuttX Project…
$ cd nuttx
$ tools/configure.sh star64:nsh
$ make
$ riscv64-unknown-elf-objcopy -O binary nuttx nuttx.bin
This produces the NuttX Kernel nuttx.bin
Build the NuttX Apps Filesystem…
$ make export
$ pushd ../apps
$ tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
$ make import
$ popd
$ genromfs -f initrd -d ../apps/bin -V "NuttXBootVol"
This generates the Initial RAM Disk initrd
Download the Device Tree jh7110-visionfive-v2.dtb from StarFive VisionFive2 Software Releases.
Save it into the nuttx folder. Or do this…
$ wget https://github.com/starfive-tech/VisionFive2/releases/download/VF2_v3.1.5/jh7110-visionfive-v2.dtb
(NuttX doesn’t need a Device Tree, but U-Boot Bootloader needs it)
(Optional) For easier debugging, we might create the following…
## Copy the config
$ cp .config nuttx.config
## Dump the Kernel disassembly to `nuttx.S`
$ riscv64-unknown-elf-objdump \
-t -S --demangle --line-numbers --wide \
nuttx \
>nuttx.S \
2>&1
## Dump the NSH `init` disassembly to `init.S`
$ riscv64-unknown-elf-objdump \
-t -S --demangle --line-numbers --wide \
../
Now we create a Bootable microSD…
NuttX goes into the FAT Partition on the microSD
How do we create a Bootable microSD for NuttX?
From the previous section, we have…
NuttX Kernel: nuttx.bin
Initial RAM Disk: initrd
Device Tree: jh7110-visionfive-v2.dtb
Now we pack all 3 files into a Flat Image Tree (FIT)…
Inside the nuttx folder, create a Text File named nuttx.its with the following content: nuttx.its
/dts-v1/;
/ {
description = "NuttX FIT image";
#address-cells = <2>;
images {
vmlinux {
description = "vmlinux";
data = /incbin/("./nuttx.bin");
type = "kernel";
arch = "riscv";
os = "linux";
load = <0x0 0x40200000>;
entry = <0x0 0x40200000>;
compression = "none";
};
ramdisk {
description = "buildroot initramfs";
data = /incbin/("./initrd");
type = "ramdisk";
arch = "riscv";
os = "linux";
load = <0x0 0x46100000>;
compression = "none";
hash-1 {
algo = "sha256";
};
};
fdt {
data = /incbin/("./jh7110-visionfive-v2.dtb");
type = "flat_dt";
arch = "riscv";
load = <0x0 0x46000000>;
compression = "none";
hash-1 {
algo = "sha256";
};
};
};
configurations {
default = "nuttx";
nuttx {
description = "NuttX";
kernel = "vmlinux";
fdt = "fdt";
loadables = "ramdisk";
};
};
};
Or do this…
$ wget https://raw.githubusercontent.com/lupyuen/nuttx-star64/main/nuttx.its
Package the NuttX Kernel, Initial RAM Disk and Device Tree into a Flat Image Tree…
## For macOS:
$ brew install u-boot-tools
## For Linux:
$ sudo apt install u-boot-tools
## Generate FIT Image from `nuttx.bin`,
## `initrd` and `jh7110-visionfive-v2.dtb`.
## `nuttx.its` must be in the same
## directory as the NuttX binaries!
$ mkimage \
-f nuttx.its \
-A riscv \
-O linux \
-T flat_dt \
starfiveu.fit
## To check FIT image
$ mkimage -l starfiveu.fit
We’ll see the NuttX Kernel…
→ mkimage -f nuttx.its -A riscv -O linux -T flat_dt starfiveu.fit
FIT description: NuttX FIT image
Created: Fri Aug 4 23:20:52 2023
Image 0 (vmlinux)
Description: vmlinux
Created: Fri Aug 4 23:20:52 2023
Type: Kernel Image
Compression: uncompressed
Data Size: 2097800 Bytes = 2048.63 KiB = 2.00 MiB
Architecture: RISC-V
OS: Linux
Load Address: 0x40200000
Entry Point: 0x40200000
Followed by the Initial RAM Disk (containing NuttX Apps)…
Image 1 (ramdisk)
Description: buildroot initramfs
Created: Fri Aug 4 23:20:52 2023
Type: RAMDisk Image
Compression: uncompressed
Data Size: 8086528 Bytes = 7897.00 KiB = 7.71 MiB
Architecture: RISC-V
OS: Linux
Load Address: 0x46100000
Entry Point: unavailable
Hash algo: sha256
Hash value: 44b3603e6e611ade7361a936aab09def23651399d4a0a3c284f47082d788e877
Finally the Device Tree (not used by NuttX)…
Image 2 (fdt)
Description: unavailable
Created: Fri Aug 4 23:20:52 2023
Type: Flat Device Tree
Compression: uncompressed
Data Size: 50235 Bytes = 49.06 KiB = 0.05 MiB
Architecture: RISC-V
Load Address: 0x46000000
Hash algo: sha256
Hash value: 42767c996f0544f513280805b41f996446df8b3956c656bdbb782125ae8ffeec
Default Configuration: 'nuttx'
Configuration 0 (nuttx)
Description: NuttX
Kernel: vmlinux
FDT: fdt
Loadables: ramdisk
This produces the Flat Image Tree starfiveu.fit, which we’ll copy later to a microSD Card.
To prepare the microSD Card, download the microSD Image sdcard.img from StarFive VisionFive2 Software Releases.
Write the downloaded image to a microSD Card with Balena Etcher or GNOME Disks. (Or use dd)
Copy the file starfiveu.fit from above and overwrite the file on the microSD Card…
## For macOS: Copy to microSD
cp starfiveu.fit "/Volumes/NO NAME"
## For macOS: Unmount the microSD
## TODO: Verify that /dev/disk2 is microSD
diskutil unmountDisk /dev/disk2
We’re ready to boot NuttX on Star64!
Connect Star64 to our computer with a USB Serial Adapter…
Insert the microSD Card into Star64 and power up Star64. NuttX boots on Star64 and NuttShell (nsh) appears in the Serial Console.
To see the available commands in NuttShell:
$ help
(Watch the Demo Video on YouTube)
Booting NuttX over TFTP is also supported on Star64.
Step by step, here’s everything that happens when NuttX boots on our SBC…
OpenSBI (Supervisor Binary Interface) is the first thing that boots on our RISC-V SBC.
OpenSBI provides Secure Access to the Low-Level System Functions (controlling CPUs, Timers, Interrupts) for the JH7110 SoC.
OpenSBI boots in RISC-V Machine Mode, the most powerful mode in a RISC-V system.
“OpenSBI Supervisor Binary Interface”
U-Boot Bootloader starts after OpenSBI, in RISC-V Supervisor Mode.
(Which is less powerful than Machine Mode)
“U-Boot Bootloader for Star64”
U-Boot Bootloader loads into RAM the NuttX Kernel, Device Tree and Initial RAM Disk.
Inside the Initial RAM Disk: NuttX Shell and NuttX Apps.
NuttX Kernel starts in RISC-V Supervisor Mode and executes the NuttX Boot Code (in RISC-V Assembly).
NuttX Start Code (in C) runs next.
It prepares the RISC-V Memory Management Unit, to protect the Kernel Memory and I/O Memory.
“Initialise RISC-V Supervisor Mode: jh7110_start” and jh7110_start_s
NuttX Kernel (nx_start) starts the NuttX Drivers and mounts the Initial RAM Disk (containing the NuttX Shell and Apps)
Followed by the NuttX Shell (NSH), for the Command-Line Interface
Finally we talk about the NuttX Shell and Apps…
NuttX Shell (NSH) and NuttX Apps (like “hello”) will run in RISC-V User Mode.
(Which is the least powerful mode)
They will make System Calls to NuttX Kernel, jumping from User Mode to Supervisor Mode. (And back)
System Calls will happen when NuttX Shell and Apps do Console Output and Console Input.
Which will trigger RISC-V Interrupts in the 16550 UART Controller.
“Platform-Level Interrupt Controller”
And that’s everything that happens when NuttX boots on Star64!
But NuttX doesn’t actually need a Device Tree!
Yeah but because the Flat Image Tree needs a Device Tree, we do it anyway.
We created all this code from scratch?
Actually most of the code came from NuttX for QEMU RISC-V! (In Kernel Mode)
It’s amazing that we reused so much code from NuttX QEMU. And ported everything to Star64 within 2 months!
What’s the catch?
We have some Size Limitations on the Initial RAM Disk, NuttX Apps and NuttX Stacks. Here are the workarounds…
Porting Linux / Unix / POSIX Apps to NuttX might need extra work, check out this example…
How did we add Star64 JH7110 to NuttX?
When we add a new board to NuttX, we do it in 4 steps…
Patch the NuttX Dependencies
Add the NuttX Arch (JH7110 SoC)
Add the NuttX Board (Star64 SBC)
Update the NuttX Documentation
This is how we did it for Star64 SBC (with JH7110 SoC) in 3 Pull Requests…
First we patch any NuttX Dependencies needed by Star64 JH7110.
We discovered that JH7110 triggers too many spurious UART interrupts…
JH7110 uses a Synopsys DesignWare 8250 UART that has a peculiar problem with the Line Control Register (LCR)… If we write to LCR while the UART is busy, it will trigger spurious UART Interrupts.
The fix is to wait for the UART to be not busy before writing to LCR. We submitted this Pull Request to fix the NuttX 16550 UART Driver…
Wait for the Pull Request to be merged. Then we add the NuttX Arch…
(How to submit a Pull Request for NuttX)
Next we submit the Pull Request that implements the support for JH7110 SoC as a NuttX Arch…
We insert JH7110 SoC into the Kconfig for the RISC-V SoCs: arch/risc-v/Kconfig
config ARCH_CHIP_JH7110
bool "StarFive JH7110"
select ARCH_RV64
select ARCH_RV_ISA_M
select ARCH_RV_ISA_A
select ARCH_RV_ISA_C
select ARCH_HAVE_FPU
select ARCH_HAVE_DPFPU
select ARCH_HAVE_MULTICPU
select ARCH_HAVE_MPU
select ARCH_HAVE_MMU
select ARCH_MMU_TYPE_SV39
select ARCH_HAVE_ADDRENV
select ARCH_NEED_ADDRENV_MAPPING
select ARCH_HAVE_S_MODE
select ONESHOT
select ALARM_ARCH
---help---
StarFive JH7110 SoC.
...
config ARCH_CHIP
...
default "jh7110" if ARCH_CHIP_JH7110
...
if ARCH_CHIP_JH7110
source "arch/risc-v/src/jh7110/Kconfig"
endif
(Remember to indent with Tabs, not Spaces!)
And we create a Kconfig for JH7110 SoC: arch/risc-v/src/jh7110/Kconfig
Then we add the NuttX Arch Source Files for JH7110 SoC at…
Finally we create the Pull Request that implements the support for Star64 SBC as a NuttX Board…
We insert Star64 SBC into the Kconfig for NuttX Boards: nuttx/boards/Kconfig
config ARCH_BOARD_JH7110_STAR64
bool "PINE64 Star64"
depends on ARCH_CHIP_JH7110
---help---
This options selects support for NuttX on PINE64 Star64 based
on StarFive JH7110 SoC.
...
config ARCH_BOARD
...
default "star64" if ARCH_BOARD_JH7110_STAR64
...
if ARCH_BOARD_JH7110_STAR64
source "boards/risc-v/jh7110/star64/Kconfig"
endif
(Remember to indent with Tabs, not Spaces!)
We create a Kconfig for Star64 SBC: nuttx/boards/risc-v/jh7110/star64/Kconfig
And we add the NuttX Board Source Files for Star64 SBC at…
But don’t submit the Pull Request yet! We’ll add the NuttX Documentation in the next section.
We’re good for RISC-V. What about Arm?
This is how we add a NuttX Arch and Board for Arm64…
Though we probably should have split it into multiple Pull Requests like for RISC-V.
Seems we need to copy a bunch of source files across branches?
No sweat! Suppose we created a staging Pull Request in our own repo…
This command produces a list of Modified Files in our Pull Request…
## TODO: Change this to your PR
pr=https://github.com/lupyuen2/wip-nuttx/pull/40
curl -L $pr.diff \
| grep "diff --git" \
| sort \
| cut -d" " -f3 \
| cut -c3-
Like this…
boards/risc-v/jh7110/star64/include/board.h
boards/risc-v/jh7110/star64/include/board_memorymap.h
boards/risc-v/jh7110/star64/scripts/Make.defs
boards/risc-v/jh7110/star64/scripts/ld.script
That we can copy to another branch in a simple script…
b=$HOME/new_branch
mkdir -p $b/boards/risc-v/jh7110/star64/include
mkdir -p $b/boards/risc-v/jh7110/star64/scripts
a=boards/risc-v/jh7110/star64/include/board.h
cp $a $b/$a
a=boards/risc-v/jh7110/star64/include/board_memorymap.h
cp $a $b/$a
a=boards/risc-v/jh7110/star64/scripts/Make.defs
cp $a $b/$a
a=boards/risc-v/jh7110/star64/scripts/ld.script
cp $a $b/$a
How did we generate the NuttX Build Configuration?
The NuttX Build Configuration for Star64 is at…
We generated the defconfig with this command…
make menuconfig \
&& make savedefconfig \
&& grep -v CONFIG_HOST defconfig \
>boards/risc-v/jh7110/star64/configs/nsh/defconfig
(How we computed the UART Clock)
During development, we should enable additional Debug Options…
CONFIG_DEBUG_ASSERTIONS=y
CONFIG_DEBUG_ASSERTIONS_EXPRESSION=y
CONFIG_DEBUG_BINFMT=y
CONFIG_DEBUG_BINFMT_ERROR=y
CONFIG_DEBUG_BINFMT_WARN=y
CONFIG_DEBUG_ERROR=y
CONFIG_DEBUG_FEATURES=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_FS_ERROR=y
CONFIG_DEBUG_FS_WARN=y
CONFIG_DEBUG_FULLOPT=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_MM=y
CONFIG_DEBUG_MM_ERROR=y
CONFIG_DEBUG_MM_WARN=y
CONFIG_DEBUG_SCHED=y
CONFIG_DEBUG_SCHED_ERROR=y
CONFIG_DEBUG_SCHED_INFO=y
CONFIG_DEBUG_SCHED_WARN=y
CONFIG_DEBUG_SYMBOLS=y
CONFIG_DEBUG_WARN=y
BINFMT is the Binary Loader, good for troubleshooting NuttX App ELF loading issues
SCHED is for Task Scheduler, which will show the spawning of NuttX App Tasks
MM is for Memory Management, for troubleshooting Memory Mapping issues
FS is for File System, like the Initial RAM Disk
Before merging with NuttX Mainline, remember to remove the Debug Options for BINFMT, FS, MM and SCHED.
Earlier we created a Pull Request to add a new NuttX Board.
In the same Pull Request, we update the NuttX Docs like so…
Create a page for the JH7110 NuttX Arch (pic above)…
Documentation/platforms/risc-v/ jh7110/index.rst
===============
StarFive JH7110
===============
`StarFive JH7110 <https://doc-en.rvspace.org/Doc_Center/jh7110.html>`_ is a 64-bit RISC-V SoC that features:
- **CPU:** SiFive RISC-V U74 Application Cores (4 cores, RV64GCB) and SiFive RISC-V S7 Monitor Core (single core, RV64IMACB)
...
(Which goes under “RISC-V” because it’s a RISC-V SoC)
Under JH7110, create a page for the Star64 NuttX Board (pic above)…
Documentation/platforms/risc-v/ jh7110/boards/star64/index.rst
=============
PINE64 Star64
=============
`Star64 <https://wiki.pine64.org/wiki/STAR64>`_ is a 64-bit RISC-V based
Single Board Computer powered by StarFive JH7110 Quad-Core SiFive U74 64-Bit CPU,
Imagination Technology BX-4-32 GPU and supports up to 8GB 1866MHz LPDDR4 memory.
...
On that page, remember to document the steps to Build and Boot NuttX…
To update and preview the NuttX Docs, follow the instructions here…
And now we’re ready to submit the Pull Request. That’s how we add a NuttX Arch and NuttX Board!
How will we create the missing drivers for Star64 JH7110?
We welcome your contribution to NuttX!
Based on the official docs…
We have started working on the HDMI Support for NuttX on Star64 JH7110…
Here are the relevant docs for the other JH7110 Peripherals…
GPIO:
I2C:
(Based on DesignWare I2C)
This NuttX I2C Driver might work: cxd56_i2c.c
RTC, SPI, UART, DMA, I2S, PWM:
USB, Ethernet:
Image Sensor Processor:
Display:
We hope to test NuttX soon on the PineTab-V RISC-V Tablet. Stay tuned for updates!
TODO: Fix the System Timer by calling OpenSBI, similar to Ox64 BL808
Today we finally have NuttX running on a Single-Board Computer: Star64 JH7110 SBC! (And StarFive VisionFive2, pic above)
We talked about building NuttX for Star64
Booting NuttX Kernel (and Initial RAM Disk) with a Bootable microSD
We stepped through everything that happens during NuttX Startup
Hopefully this article will be helpful if you’re adding a NuttX Arch or NuttX Board
Stay tuned for Upcoming Features on Star64 and VisionFive2
(Maybe PineTab-V too!)
Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…
lupyuen.github.io/src/release.md
Why did the NuttX Build fail with missing math.h?
$ sudo apt install \
gcc-riscv64-unknown-elf \
picolibc-riscv64-unknown-elf
$ make
./stdio/lib_dtoa_engine.c:40:10:
fatal error: math.h: No such file or directory
#include <math.h>
If the NuttX Build fails due to missing math.h, install the xPack GNU RISC-V Embedded GCC Toolchain…
Is there another solution?
Here’s a quick hack: Edit the file nuttx/.config and add…
NEED_MATH_H=y
CONFIG_LIBM=y
This fixes the NuttX Build to use the NuttX Version of math.h. (Instead of the System Version)
NuttX Kernel will boot OK if we don’t actually use any Math Functions. But NuttX Apps will fail to load if they call Math Functions. (Like floor)
(Thanks to Ken Dickey for the tip!)
The StarFive VisionFive2 Software Release was helpful for creating the Bootable microSD for NuttX.
The Software Release includes an SD Card Image that boots OK on Star64…
Login with…
buildroot login: root
Password: starfive
(We reused the SD Card Image for NuttX)
Based on the files above, we figured out how to generate the Flat Image Tree for NuttX…
Also we see the script that generates the SD Card Image: genimage.sh
genimage \
--rootpath "${ROOTPATH_TMP}" \
--tmppath "${GENIMAGE_TMP}" \
--inputpath "${INPUT_DIR}" \
--outputpath "${OUTPUT_DIR}" \
--config genimage-vf2.cfg
The SD Card Partitions are defined here: genimage-vf2.cfg
image sdcard.img {
hdimage {
gpt = true
}
partition spl {
image = "work/u-boot-spl.bin.normal.out"
partition-type-uuid = 2E54B353-1271-4842-806F-E436D6AF6985
offset = 2M
size = 2M
}
partition uboot {
image = "work/visionfive2_fw_payload.img"
partition-type-uuid = 5B193300-FC78-40CD-8002-E86C45580B47
offset = 4M
size = 4M
}
partition image {
# partition-type = 0xC
partition-type-uuid = EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
image = "work/starfive-visionfive2-vfat.part"
offset = 8M
size = 292M
}
partition root {
# partition-type = 0x83
partition-type-uuid = 0FC63DAF-8483-4772-8E79-3D69D8477DE4
image = "work/buildroot_rootfs/images/rootfs.ext4"
offset = 300M
bootable = true
}
}
Useful for creating our own SD Card Partitions!
(We won’t need the spl, uboot and root partitions for NuttX)
How did we figure out the UART Clock for JH7110?
CONFIG_16550_UART0_CLOCK=23040000
We logged the values of DLM and DLL in the UART Driver during startup…
uint32_t dlm = u16550_serialin(priv, UART_DLM_OFFSET);
uint32_t dll = u16550_serialin(priv, UART_DLL_OFFSET);
(We capture DLM and DLL only when DLAB=1)
(Be careful to print only when DLAB=0)
According to our log, DLM is 0 and DLL is 13. Which means..
dlm = 0 = (div >> 8)
dll = 13 = (div & 0xff)
Which gives div=13. Now since baud=115200 at startup…
div = (uartclk + (baud << 3)) / (baud << 4)
13 = (uartclk + 921600) / 1843200
uartclk = (13 * 1843200) - 921600
= 23040000
Thus uartclk=23040000. And that’s why we set…
CONFIG_16550_UART0_CLOCK=23040000
Shouldn’t we get the UART Clock from the SoC Datasheet?
Yep absolutely! For JH7110 there isn’t a complete Datasheet, the docs only point to the Linux Device Tree. That’s why we derived the UART Clock ourselves.
Sometimes we work backwards when porting NuttX to SBCs…
Assume that the UART Driver is already configured by U-Boot
Get the simple UART Driver working, without configuring the UART
Figure out the right values of DLL and DLM, so that UART Driver will configure the UART correctly
DLL and DLM will be reused by other UARTs: UART 1, 2, 3, …
For Ox64 BL808 SBC: We skipped the UART Configuration completely. Which is OK because we won’t use the other UART Ports anyway…
So if we’re building the UART Driver ourselves and it’s incomplete, it’s OK to upstream it first and complete it later. That’s how we did it for PinePhone on Allwinner A64…