Apache NuttX RTOS on Sophgo SG2000 RISC-V SoC (Milk-V Duo S SBC)

📝 19 May 2024

Milk-V Duo S SBC with SG2000 RISC-V SoC

UPDATE: NuttX Mainline now supports SG2000 and Milk-V Duo S!

Soon we’ll see many new 64-bit RISC-V SBCs based on the Sophgo SG2000 RISC-V SoC.

Will they work with Apache NuttX RTOS? (Real-Time Operating System) Let’s find out…

Something strangely satisfying about NuttX on RISC-V… We finished the port in Only 10 Days 🎉

(Is this a sponsored review? I was given a Milk-V Duo S, and I bought another. So it cancels out I guess?)

Sophgo SG2000 RISC-V SoC

§1 Sophgo SG2000 RISC-V SoC

Sophgo SG2000 SoC has a fascinating mix of 64-bit RISC-V Cores (Arm too)…

Plus a Low-Power 8051 MCU (for Wakeup Duties) and a Tensor Processing Unit (for Image Recognition, not LLM)

Sophgo SG2000 RISC-V SoC

(See the SG2000 Reference Manual)

(See the Cvitek SDK Docs)

Whoa RISC-V AND Arm CPUs in a single SoC?

Actually there’s a Physical Switch that selects the Main CPU: RISC-V OR Arm.

Don’t let yer pet hamster flip it… It will get super frustrating!

(Sophgo / Sophon.ai comes from 3 Body)

Milk-V Duo S SBC with SG2000 RISC-V SoC

§2 Boot Without MicroSD

What happens if we boot Milk-V Duo S? Fresh from the box?

Connect our USB UART Dongle according to the instructions (pic above)…

Milk-V Duo SUSB UART
GND (Pin 6)GND
TX (Pin 8)RX
RX (Pin 10)TX

USB UART Dongle must be CP2102, it doesn’t like CH340 😬

Switch to “RV” (RISC-V) instead of “Arm”

Flip the Switch so it’s set to “RV” (RISC-V) instead of “Arm”. (Pic above)

Power up the board via the USB-C Port. Connect to the USB UART at 115.2 kbps…

screen /dev/ttyUSB0 115200

Milk-V Duo S won’t boot because it doesn’t ship with U-Boot Bootloader in Flash Memory…

C.SCS/0/0.WD.URPL.USBI.USBEF.BS/EMMC.EMI/25000000/12000000. 
E:eMMC initializing failed
E:Boot failed
E:RESET:plat/mars/platform.c:114

We’ll need U-Boot on MicroSD, in the next section.

(platform.c might be here)

If we see “B.SCS” instead of “C.SCS”…

B.SCS/0/0.WD.URPL.USBI.USBEF.BS/EMMC.EMI/25000000/12000000.

Nope we’re in Arm Mode… Flip the switch back to RISC-V!

If we use CH340 (instead of CP2102): UART Output will be gloriously garbled.

Debian Image for Sophgo SG2000

§3 Download the Linux MicroSD

Milk-V Duo S won’t boot without MicroSD. How now?

We boot Linux on MicroSD, thanks to the awesome work by Justin Hammond (Fishwaldo)…

We download the Latest Release for Milk-V Duo S (SG2000)…

Uncompress the Debian Image…

## For Linux:
$ sudo apt install lz4

## For macOS:
$ brew install lz4

## Uncompress the download to get `duos_sd.img`
$ lz4 duos_sd.img.lz4

And write duos_sd.img to a MicroSD Card. Use Balena Etcher, GNOME Disks or dd.

We’ll see these MicroSD Files…

## MicroSD Root Folder
$ ls -l /Volumes/boot
-rwx 3494900 System.map-5.10.4-20240329-1+
-rwx  125534 config-5.10.4-20240329-1+
drwx    2048 extlinux
drwx    2048 fdt
-rwx  388608 fip.bin
-rwx 4937389 vmlinuz-5.10.4-20240329-1+

## U-Boot Bootloader Config
$ ls -l /Volumes/boot/extlinux
-rwx 749 extlinux.conf

## Linux Device Tree for SG2000
$ ls -l /Volumes/boot/fdt/linux-image-duos-5.10.4-20240329-1+
-rwx 21575 cv181x_milkv_duos_sd.dtb

We peek at the U-Boot Bootloader Config (which will boot NuttX with a tiny tweak)

$ cat /Volumes/boot/extlinux/extlinux.conf
...
menu label Debian GNU/Linux trixie/sid 5.10.4-20240329-1+
linux  /vmlinuz-5.10.4-20240329-1+
fdtdir /fdt/linux-image-duos-5.10.4-20240329-1+/
append root=/dev/root console=ttyS0,115200 earlycon=sbi root=/dev/mmcblk0p2 rootwait rw

Watch what happens when we boot the MicroSD…

OpenSBI and U-Boot Bootloader

§4 Boot the Linux MicroSD

Linux on MicroSD: Will it boot on Milk-V Duo S?

Yep Linux boots OK. First we see OpenSBI (Supervisor Binary Interface)…

OpenSBI v0.9
Platform Name       : Milk-V DuoS
Platform Features   : mfdeleg
Platform HART Count : 1
Platform Console Device : uart8250
Firmware Base       : 0x8000_0000
Firmware Size       : 132 KB
Runtime SBI Version : 0.3

Domain0 Region00 : 0x7400_0000-0x7400_ffff (I)
Domain0 Region01 : 0x8000_0000-0x8003_ffff ()
Domain0 Region02 : 0x0-0xffff_ffff_ffff_ffff (R,W,X)
Boot HART ISA      : rv64imafdcvsux
Boot HART Features : scounteren,mcounteren,time
Boot HART MIDELEG  : 0x0222
Boot HART MEDELEG  : 0xb109

## OpenSBI boots at 0x8000_0000.
## 0x7400_0000 looks interesting! We'll come back to this

(See the Complete Log)

Followed by the U-Boot Bootloader…

## U-Boot Boots
U-Boot 2021.10-ga57aa1f2-dirty (Apr 24 2024 - 11:24:46 +0000) cvitek_cv181x
Hit any key to stop autoboot:  0 
Scanning mmc 0:1...
Found /extlinux/extlinux.conf

## U-Boot Menu
1:.Debian GNU/Linux trixie/sid 5.10.4-20240329-1+
2:.Debian GNU/Linux trixie/sid 5.10.4-20240329-1+ (rescue target)
Enter choice: 1

## U-Boot boots Debian Linux
Retrieving file: /vmlinuz-5.10.4-20240329-1+
Retrieving file: /fdt/linux-image-duos-5.10.4-20240329-1+/cv181x_milkv_duos_sd.dtb
Booting using the fdt blob at 0x81200000

Finally we see Debian Linux…

Starting kernel ...
Linux version 5.10.4-20240329-1+ (root@3abcc283c6ba) (riscv64-unknown-linux-musl-gcc (Xuantie-900 linux-5.10.4 musl gcc Toolchain V2.6.1 B-20220906) 10.2.0, GNU ld (GNU Binutils) 2.35)
...
Debian GNU/Linux trixie/sid duos ttyS0
duos login:

Linux works great, we hop over to NuttX…

(Cvitek is the old name of Sophgo / Sophon)

U-Boot Bootloader

§5 Settings for U-Boot Bootloader

How will we boot NuttX?

We seek guidance from the U-Boot Bootloader.

As we power on Milk-V Duo S, hit Enter a few times to see the U-Boot Command Prompt…

U-Boot 2021.10-ga57aa1f2-dirty (May 07 2024 - 08:13:12 +0000) cvitek_cv181x
Loading Environment from FAT... mmc1 : finished tuning, code:53
Hit any key to stop autoboot:  0
cv181x_c906#

Enter printenv to dump the U-Boot Settings…

## U-Boot Settings
$ printenv
kernel_addr_r=0x80200000
kernel_comp_addr_r=0x81800000
kernel_comp_size=0x1000000
ramdisk_addr_r=0x84000000
uImage_addr=0x81800000
update_addr=0x9fe00000

(See the U-Boot Settings)

kernel_addr_r says that U-Boot will load Linux Kernel into RAM at Address 0x8020_0000. (We’ll set this in NuttX)

And the Ethernet Driver is fully operational in U-Boot. Which means we can boot NuttX over the Network…

$ net list
eth0: ethernet@4070000
00:00:00:00:00:00
active

Here’s how…

(See the U-Boot Commands)

Boot NuttX over TFTP

§6 Boot NuttX over TFTP

What’s the quickest way to port NuttX to SG2000?

Like Linux, we could copy NuttX to MicroSD, insert into Milk-V Duo S and power up. Again and again and again…

But there’s a quicker way: Boot NuttX over the Network, thanks to U-Boot Bootloader and TFTP (Trivial File Transfer Protocol)

Follow the instructions here to install our TFTP Server. Copy these files to our TFTP Server…

At the U-Boot Command Prompt: We configure our TFTP Server…

## Set the U-Boot TFTP Server
## TODO: Change to your TFTP Server
setenv tftp_server 192.168.31.10

## If Initial RAM Disk is needed (like for Linux, not for NuttX)...
## Set the RAM Disk Size (assume the max)
## setenv ramdisk_size 0x1000000

## Save the U-Boot Config for future reboots
saveenv

Then we load the NuttX Image into RAM over TFTP…

## Fetch the IP Address over DHCP
## Load the NuttX Image from TFTP Server
## kernel_addr_r=0x80200000
dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000

## Load the Device Tree from TFTP Server
## fdt_addr_r=0x81200000
## TODO: Fix the Device Tree, it's not needed by NuttX
tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb

## Set the RAM Address of Device Tree
## fdt_addr_r=0x81200000
## TODO: Fix the Device Tree, it's not needed by NuttX
fdt addr ${fdt_addr_r}

## If Initial RAM Disk is needed...
## Load the Intial RAM Disk from TFTP Server
## ramdisk_addr_r=0x81600000
## tftpboot ${ramdisk_addr_r} ${tftp_server}:initrd

And we boot NuttX from RAM…

## Boot the NuttX Image with the Device Tree
## kernel_addr_r=0x80200000
## fdt_addr_r=0x81200000
## TODO: Fix the Device Tree, it's not needed by NuttX
booti ${kernel_addr_r} - ${fdt_addr_r}

## For Linux: We need the RAM Disk Address
## ramdisk_addr_r=0x81600000
## ramdisk_size=0x1000000
## booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r}

What happens when we boot NuttX?

Absolutely nothing…

## Boot NuttX over TFTP, mashed up in a single line...
$ dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000 ; tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb ; fdt addr ${fdt_addr_r} ; booti ${kernel_addr_r} - ${fdt_addr_r}

Booting using the fdt blob at 0x81200000
Loading Ramdisk to 9e27f000, end 9f27f000 ... OK
Loading Device Tree to 9e26f000, end 9e27e43a ... OK
Starting kernel ...

But that’s OK, we haven’t modified NuttX Kernel for SG2000. We’ll print something in a while.

We type these commands EVERY TIME we boot?

We can automate: Just do this once, and NuttX will Auto-Boot whenever we power up…

## Add the Boot Command for TFTP
setenv bootcmd_tftp 'dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000 ; tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb ; fdt addr ${fdt_addr_r} ; booti ${kernel_addr_r} - ${fdt_addr_r}'

## Save it for future reboots
saveenv

## Test the Boot Command for TFTP, then reboot
run bootcmd_tftp

## Remember the Original Boot Targets: `mmc0 dhcp pxe`
setenv orig_boot_targets "$boot_targets"

## Prepend TFTP to the Boot Targets: `tftp mmc0 dhcp pxe`
setenv boot_targets "tftp $boot_targets"

## Save it for future reboots
saveenv

(What about Static IP?)

(How to Undo Auto-Boot)

UART Controller for SG2000

§7 UART Controller for SG2000

How will NuttX print to the Serial Console?

First we track down the UART Controller for SG2000.

From SG2000 Reference Manual (Page 638): The UART Controller is at these Base Addresses (we’ll talk to UART0)

UART ModuleBase Address
UART00x0414_0000
UART10x0415_0000
UART20x0416_0000
UART30x0417_0000
UART40x041C_0000
RTCSYS_UART0x0502_2000

What UART Controller is inside SG2000?

According to OpenSBI Log: The UART Controller is uart8250.

Which is supported by NuttX. We mod the NuttX Boot Code to print something…

Print to UART in RISC-V Assembly

§8 Print to UART in RISC-V Assembly

Printing in RISC-V Assembly? Why not C?

That’s because the very first thing that boots is the NuttX Boot Code in RISC-V Assembly (instead of C)…

SG2000 UART0 Controller is at 0x0414_0000 (previous section). To print something, we write to the UART Output Register at that address: bl808_head.S

/* RISC-V Boot Code for Apache NuttX RTOS */
real_start:

  /* Print `123` to UART */
  /* Load UART Base Address to Register t0 */
  li  t0, 0x04140000

  /* Load `1` to Register t1 */
  li  t1, 0x31
  /* Store byte from Register t1 to UART Base Address, Offset 0 */
  sb  t1, 0(t0)

  /* Load `2` to Register t1 */
  li  t1, 0x32
  /* Store byte from Register t1 to UART Base Address, Offset 0 */
  sb  t1, 0(t0)

  /* Load `3` to Register t1 */
  li  t1, 0x33
  /* Store byte from Register t1 to UART Base Address, Offset 0 */
  sb  t1, 0(t0)

(Moved here)

(li loads a Value into a Register)

(sb stores a byte from a Register into an Address)

Our code will print “123” when NuttX boots. We test this…

§9 NuttX Boots A Tiny Bit

Follow these steps to build Apache NuttX RTOS for SG2000 and Milk-V Duo S…

This produces the NuttX Image File: Image. Which we copy to our TFTP Server…

## Copy NuttX Image and Device Tree to TFTP Server
## TODO: Change `tftpserver` and `tftpboot` to our TFTP Server and Path
cp Image Image-sg2000
scp Image-sg2000 \
  tftpserver:/tftpboot/Image-sg2000
scp cv181x_milkv_duos_sd.dtb \
  tftpserver:/tftpboot/cv181x_milkv_duos_sd.dtb

(cv181x_milkv_duos_sd.dtb is here)

To Boot NuttX: Run these commands at the U-Boot Command Prompt…

## Load NuttX Image and Device Tree into RAM
$ dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000
$ tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb
$ fdt addr ${fdt_addr_r}

## Boot NuttX from RAM
$ booti ${kernel_addr_r} - ${fdt_addr_r}

Starting kernel ...
123

See the “123”? That’s proof that our NuttX Boot Code is actually running on SG2000 and Milk-V Duo S. We port some more…

(See the Complete Log)

NuttX Kernel Boots OK

§10 NuttX Kernel Boots OK

NuttX Kernel prints “123”. What about the rest?

More mods for NuttX Kernel…

  1. We set the NuttX Memory Map for SG2000: nsh/defconfig

    ## Kernel RAM
CONFIG_RAM_START=0x80200000
CONFIG_RAM_SIZE=1048576

    (Moved here)

    (Explained here)

  2. Also the NuttX Linker Script: ld.script

    MEMORY {
  kflash (rx) : ORIGIN = 0x80200000, LENGTH = 2048K   /* w/ cache */
  ...
SECTIONS {
  . = 0x80200000;

    (Moved here)

    (Explained here)

  3. We select the NuttX Driver for 16550 UART: nsh/defconfig

    CONFIG_16550_REGINCR=4
CONFIG_16550_UART0=y
CONFIG_16550_UART0_BASE=0x04140000
CONFIG_16550_UART0_SERIAL_CONSOLE=y
CONFIG_16550_UART=y
CONFIG_16550_WAIT_LCR=y
CONFIG_SERIAL_UART_ARCH_MMIO=y

    (Moved here)

    (Explained here)

  4. Enable Logging for NuttX Scheduler and Binary Loader: nsh/defconfig

    CONFIG_DEBUG_BINFMT=y
CONFIG_DEBUG_BINFMT_ERROR=y
CONFIG_DEBUG_BINFMT_WARN=y
CONFIG_DEBUG_SCHED=y
CONFIG_DEBUG_SCHED_ERROR=y
CONFIG_DEBUG_SCHED_INFO=y
CONFIG_DEBUG_SCHED_WARN=y

    (Explained here)

  5. And disable the PLIC Interrupt Controller (until we figure it out)

    (Explained here)

After applying the above fixes: NuttX Kernel boots successfully! (Pic above)

## Load NuttX Image and Device Tree into RAM
$ dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000
$ tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb
$ fdt addr ${fdt_addr_r}

## Boot NuttX from RAM
$ booti ${kernel_addr_r} - ${fdt_addr_r}

Starting kernel ...
123ABCnx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
nxtask_activate: lpwork pid=1,TCB=0x80408130
nxtask_activate: AppBringUp pid=2,TCB=0x80408740
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
nxtask_activate: /system/bin/init pid=3,TCB=0x80409140
nxtask_exit: AppBringUp pid=2,TCB=0x80408740

One last thing and we’re done…

(See the Complete Log)

(Watch the Demo on YouTube)

NuttX Kernel boots all the way to NuttX Shell

§11 NuttX Shell Too!

NuttX Kernel boots OK. Where’s the NuttX Shell?

We won’t see the NuttX Shell until we fix the Interrupt Controller for SG2000. Which is NOT documented. (Sigh)

That’s because NuttX Shell requires UART Input Interrupts AND UART Output Interrupts, to support Console Input / Output.

Thus we sniff around and find out how the Interrupt Controller works.

  1. We dumped the Linux Device Tree for SG2000…

    ## Convert the SG2000 Device Tree
dtc \
  -o cv181x_milkv_duos_sd.dts \
  -O dts \
  -I dtb \
  cv181x_milkv_duos_sd.dtb

    (Explained here)

  2. Snooped the PLIC Interrupt Controller in the Device Tree: cv181x_milkv_duos_sd.dts

    interrupt-controller@70000000 {
  riscv,ndev = <0x65>;
  riscv,max-priority = <0x07>;
  reg-names = "control";
  reg = <0x00 0x70000000 0x00 0x4000000>;
  interrupts-extended = <0x16 0xffffffff 0x16 0x09>;
  interrupt-controller;
  compatible = "riscv,plic0";

    (Explained here)

  3. And fixed the NuttX Driver for PLIC Interrupts: bl808_memorymap.h

    // Base Address of PLIC Interrupt Controller
#define BL808_PLIC_BASE 0x70000000ul

    (Moved here)

    (Explained here)

After fixing the Interrupt Controller and UART Interrupts: Our Final NuttX Image boots all the way to NuttX Shell! (Pic above)

## Load NuttX Image and Device Tree into RAM
$ dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000
$ tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb
$ fdt addr ${fdt_addr_r}

## Boot NuttX from RAM
$ booti ${kernel_addr_r} - ${fdt_addr_r}

Starting kernel ...
NuttShell (NSH) NuttX-12.4.0

nsh> uname -a
NuttX 12.4.0 122c717 May  8 2024 18:13:30 risc-v ox64

nsh> ls
/:
 dev/
 proc/
 system/

nsh> ls /dev
/dev:
 console
 null
 ram0
 ttyS0
 zero

(See the Complete Log)

What about the rest of NuttX?

NuttX OSTest is the perfect way to test everything in NuttX…

nsh> ostest
user_main: mutex test
riscv_exception:
  EXCEPTION: Load access fault
  MCAUSE:    5
  EPC:       802189ce
  MTVAL:     0000000000000000
  Segmentation fault in PID 7: ostest

Sadly we’re hitting a RISC-V Exception: Load Access Fault. Needs more troubleshooting alamak.

(See the Complete Log)

NuttX Boot Flow

What happens exactly when NuttX boots on SG2000?

Exact same thing as NuttX booting on Ox64 BL808 SBC (pic above)…

Milk-V Duo S SBC with SG2000 RISC-V SoC

§12 What’s Next

We’re eagerly awaiting the new 64-bit RISC-V SBCs based on the Sophgo SG2000 RISC-V SoC. Meanwhile we’re all prepped and ready…

Up Next…

  1. We’ll Upstream SG2000 to NuttX Mainline

    (So others may contribute their code)

    UPDATE: NuttX Mainline now supports SG2000 and Milk-V Duo S!

  2. Run Daily Automated Testing on a Real SG2000 SBC…

    “Daily Automated Testing for Milk-V Duo S RISC-V SBC (IKEA TRETAKT / Apache NuttX RTOS)”

  3. Create an SG2000 Emulator for easier testing…

    “Emulate Sophgo SG2000 SoC / Milk-V Duo S SBC with TinyEMU RISC-V Emulator”

  4. We might run NuttX on the SG2000 Co-Processor

    (Plus SG2002 with its upsized TPU / NPU)

  5. Join me online at the Apache NuttX International Workshop

    (We’ll Q&A about Ox64 BL808 and SG2000)

Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/sg2000.md

NuttX Mainline now supports SG2000

§13 Appendix: Apache NuttX RTOS for Pine64 Oz64 SBC

What about Pine64 Oz64 SBC? Which also runs on SG2000 SoC?

Yep Pine64 Oz64 SBC is supported by NuttX Mainline! (Pic above)

Just follow the exact same instructions that we covered earlier…

§13.1 Connect our USB UART Adapter

Connect our USB UART Dongle to Oz64 SBC…

Oz64 SBCUSB UART
GND (Pin 6)GND
TX (Pin 8)RX
RX (Pin 10)TX

USB UART Dongle must be CP2102, it doesn’t like CH340

§13.2 Download the Linux MicroSD

Based on the Linux MicroSD created by Justin Hammond (Fishwaldo)…

We download the Latest Release for Milk-V Duo S (SG2000)…

Uncompress the Debian Image…

## For Linux:
$ sudo apt install lz4

## For macOS:
$ brew install lz4

## Uncompress the download to get `duos_sd.img`
$ lz4 duos_sd.img.lz4

And write duos_sd.img to a MicroSD Card. Use Balena Etcher, GNOME Disks or dd.

Insert the MicroSD Card into Oz64, but don’t power up yet…

§13.3 Download the NuttX Image

Download the NuttX Image File Image from the Latest Daily Build of NuttX for SG2000…

And download the Oz64 Device Tree…

We’ll copy them to our TFTP Server…

§13.4 Boot NuttX over TFTP

Follow the instructions here to install our TFTP Server.

Copy the NuttX Image and Device Tree (previous section) to our TFTP Server…

## Copy NuttX Image and Device Tree to TFTP Server
## TODO: Change `tftpserver` and `tftpboot` to our TFTP Server and Path
cp Image Image-sg2000
scp Image-sg2000 \
  tftpserver:/tftpboot/Image-sg2000
scp cv181x_milkv_duos_sd.dtb \
  tftpserver:/tftpboot/cv181x_milkv_duos_sd.dtb

Power up the board via the Oz64 Power Adapter. Connect to the USB UART at 115.2 kbps…

screen /dev/ttyUSB0 115200

When Oz64 boots, press Enter to reveal the U-Boot Command Prompt.

Enter these commands to configure U-Boot to Auto-Boot NuttX on Oz64…

## Set the U-Boot TFTP Server
## TODO: Change to your TFTP Server
setenv tftp_server 192.168.31.10

## Add the Boot Command for TFTP
setenv bootcmd_tftp 'dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000 ; tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb ; fdt addr ${fdt_addr_r} ; booti ${kernel_addr_r} - ${fdt_addr_r}'

## Remember the Original Boot Targets: `mmc0 dhcp pxe`
setenv orig_boot_targets "$boot_targets"

## Prepend TFTP to the Boot Targets: `tftp mmc0 dhcp pxe`
setenv boot_targets "tftp $boot_targets"

## Save it for future reboots
saveenv

Power-off and power-on our SBC. NuttX now auto-boots on Oz64 whenever we power-on…

Starting kernel ...
NuttShell (NSH) NuttX-12.4.0

nsh> uname -a
NuttX 12.4.0 122c717 May  8 2024 18:13:30 risc-v ox64

(See the NuttX Log)

(What about Static IP?)

(How to Undo Auto-Boot)

TODO: Boot NuttX on MicroSD

NuttX Mainline now supports SG2000

§14 Appendix: NuttX Mainline now supports SG2000!

Sophgo SG2000 SoC, Milk-V Duo S SBC and Pine64 Oz64 SBC are now officially supported by Apache NuttX Mainline! 🎉

How did we prepare the Pull Requests for NuttX Mainline?

In this article we discussed the Modified Code for SG2000. We took the Modified Code and staged the changes into our NuttX Repo here…

Next we create a new branch. Inside the new branch: We copy the NuttX Source Folders for BL808 / Ox64, and paste the folders as SG2000 / Duo S…

  1. “Copy and rename BL808 to SG2000”

  2. “Copy and rename Ox64 to DuoS”

  3. “Copy and rename BL808 Include to SG2000 Include”

  4. “Rename BL808 AppInit to SG2000 AppInit”

  5. “Remove BL808 UART from SG2000”

Remember our Modified Code for SG2000? We copy the Modified Code into our new NuttX Source Folders (for SG2000 and Duo S)…

We create a new NuttX SoC (Arch) for SG2000. And a new NuttX Board for Milk-V Duo S…

We compute the UART Clock Frequency for SG2000 and configure it in NuttX…

Then we goofed and realise that NSH Shell won’t wait for commands to complete! (NSH Shell returns immediately) We restore the NuttX Configuration to wait for processes…

sleep() wasn’t sleeping with the right duration. Thus we adjust the RISC-V Timer Frequency…

We enable RAW_BINARY so that NuttX Build will generate nuttx.bin (otherwise we need to generate it ourselves)…

We added the NuttX Documentation for SG2000 and Duo S…

During PR Review: We realised that duos wasn’t a distinctive name. Hence we renamed it to milkv_duos

And finally we run a script like this, to split the above Modified Files into Two Pull Requests. We made it to NuttX Mainline!

NuttX Mainline changes every day. Can we be sure that NuttX will always run OK on SG2000?

That’s why we Build NuttX for SG2000 every day at GitHub Actions. And we test the Daily Build on a Real Milk-V Duo S SBC!

Build NuttX for SG2000

§15 Appendix: Build NuttX for SG2000

We may download the NuttX Image File Image from the Latest Daily Build of NuttX for SG2000…

If we prefer to build NuttX ourselves, please read on…

In this article we took NuttX for Ox64 BL808 RISC-V SBC. Then made a few tweaks, and it boots on SG2000 and Milk-V Duo S…

  1. “Set the NuttX Memory Map”

  2. “Select the 16550 UART Driver”

  3. “Enable Logging for Scheduler”

  4. “Fix the NuttX Driver for PLIC”

Follow these steps to build Apache NuttX RTOS (Mainline) for SG2000 and Milk-V Duo S, based on the Official NuttX Docs…

Install the Build Prerequisites, skip the RISC-V Toolchain…

Download the RISC-V Toolchain for riscv-none-elf (xPack)…

Then Download and Build NuttX…

set -e  #  Exit when any command fails
set -x  #  Echo commands

## Download the Source Code for NuttX Kernel and NuttX Apps
git clone https://github.com/apache/incubator-nuttx nuttx
git clone https://github.com/apache/incubator-nuttx-apps apps
cd nuttx

## Pull updates
git pull && git status && hash1=`git rev-parse HEAD`
pushd ../apps
git pull && git status && hash2=`git rev-parse HEAD`
popd
echo NuttX Source: https://github.com/apache/nuttx/tree/$hash1 >nuttx.hash
echo NuttX Apps: https://github.com/apache/nuttx-apps/tree/$hash2 >>nuttx.hash

## Show the version of GCC
riscv-none-elf-gcc -v

## Configure the build
tools/configure.sh milkv_duos:nsh

## Preserve the build config
cp .config nuttx.config

## Run the build
make

## Build the Apps Filesystem
make export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make import
popd

## Generate the Initial RAM Disk
genromfs -f initrd -d ../apps/bin -V "NuttXBootVol"

## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero >/tmp/nuttx.pad

## Append the Padding and Initial RAM Disk to the NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
  >Image

## Copy the NuttX Image and Device Tree to our TFTP Server
cp Image Image-sg2000
wget https://github.com/lupyuen2/wip-nuttx/releases/download/sg2000-1/cv181x_milkv_duos_sd.dtb
scp Image-sg2000 tftpserver:/tftpboot/Image-sg2000
scp cv181x_milkv_duos_sd.dtb tftpserver:/tftpboot/cv181x_milkv_duos_sd.dtb

## [For Debugging Only] Show the size
riscv-none-elf-size nuttx

## [For Debugging Only] Dump the NuttX Kernel disassembly to nuttx.S
riscv-none-elf-objdump \
  --syms --source --reloc --demangle --line-numbers --wide \
  --debugging \
  nuttx \
  >nuttx.S \
  2>&1

## [For Debugging Only] Dump the NSH Shell disassembly to init.S
riscv-none-elf-objdump \
  --syms --source --reloc --demangle --line-numbers --wide \
  --debugging \
  ../apps/bin/init \
  >init.S \
  2>&1

## [For Debugging Only] Dump the Hello App disassembly to hello.S
riscv-none-elf-objdump \
  --syms --source --reloc --demangle --line-numbers --wide \
  --debugging \
  ../apps/bin/hello \
  >hello.S \
  2>&1

(See the Build Script)

(See the Build Outputs)

The steps above assume that we’ve installed our TFTP Server, according to the instructions here.

Then follow these steps to boot NuttX on Milk-V Duo S…

Virtual Memory for NuttX Apps

Why the RAM Disk? Isn’t NuttX an RTOS?

SG2000 uses a RAM Disk because it runs in NuttX Kernel Mode (instead of the typical Flat Mode). This means we can do Memory Protection and Virtual Memory for Apps. (Pic above)

But it also means we need to bundle the NuttX Apps as ELF Files, hence the RAM Disk…

Most of the NuttX Platforms run on NuttX Flat Mode, which has NuttX Apps Statically-Linked into the NuttX Kernel.

NuttX Flat Mode works well for Small Microcontrollers. But SG2000 and other SoCs will need the more sophisticated NuttX Kernel Mode…

Porting NuttX to SG2000

§16 Appendix: Port NuttX to SG2000

How did we port NuttX to SG2000?

We started with NuttX for Ox64 BL808 RISC-V SBC. Then made a few tweaks, and it boots on SG2000 and Milk-V Duo S. This chapter explains the minor tweaks that we made. (Pic above)

Why did we start with NuttX for Ox64?

That’s because Ox64 BL808 runs on the same RISC-V Core as SG2000: T-Head C906.

What about the T-Head Extensions for C906?

Yep we copied (unchanged) the T-Head Extensions for C906 from Ox64 BL808 to SG2000. And they work hunky dory on SG2000…

Let’s talk about the tweaks…

§16.1 NuttX Memory Map

From U-Boot Bootloader Settings: We see that SG2000 boots at this address…

kernel_addr_r=0x80200000

Thus we define the NuttX Memory Map for SG2000 like so…

NuttX Kernel will boot at 0x8020_0000, NuttX Apps will run at Virtual Address 0xC000_0000.

Here’s the NuttX Config: nsh/defconfig

## Kernel RAM
## TODO: Fix the size
CONFIG_RAM_START=0x80200000
CONFIG_RAM_SIZE=1048576

## Kernel Paged Pool (Allocated to NuttX Apps)
## TODO: Fix the size
CONFIG_ARCH_PGPOOL_PBASE=0x80600000
CONFIG_ARCH_PGPOOL_VBASE=0x80600000
CONFIG_ARCH_PGPOOL_SIZE=4194304

## Virtual Memory for NuttX App Code
CONFIG_ARCH_TEXT_VBASE=0xC0000000
CONFIG_ARCH_TEXT_NPAGES=128

## Virtual Memory for NuttX App Data
CONFIG_ARCH_DATA_VBASE=0xC0100000
CONFIG_ARCH_DATA_NPAGES=128

## Virtual Memory for NuttX App Heap
CONFIG_ARCH_HEAP_VBASE=0xC0200000
CONFIG_ARCH_HEAP_NPAGES=128

(Moved here)

And here’s the NuttX Linker Script: ld.script

/* TODO: Fix the size */
MEMORY {
  kflash (rx) : ORIGIN = 0x80200000, LENGTH = 2048K   /* w/ cache */
  ksram (rwx) : ORIGIN = 0x80400000, LENGTH = 2048K   /* w/ cache */
  pgram (rwx) : ORIGIN = 0x80600000, LENGTH = 4096K   /* w/ cache */
  ramdisk (rwx) : ORIGIN = 0x80A00000, LENGTH = 16M   /* w/ cache */
}
...
SECTIONS {
  . = 0x80200000;

(Moved here)

§16.2 Select the 16550 UART Driver

From OpenSBI Log: We see that SG2000 runs with a 8250 UART Controller.

Thus we select the NuttX Driver for 16550 UART, which is compatible with 8250…

Here’s the NuttX Config: nsh/defconfig

CONFIG_16550_ADDRWIDTH=0
CONFIG_16550_REGINCR=4
CONFIG_16550_UART0=y
CONFIG_16550_UART0_BASE=0x04140000
CONFIG_16550_UART0_CLOCK=23040000
CONFIG_16550_UART0_SERIAL_CONSOLE=y
CONFIG_16550_UART=y
CONFIG_16550_WAIT_LCR=y
CONFIG_SERIAL_UART_ARCH_MMIO=y

(Moved here)

Don’t update the NuttX Config File directly! We ran make menuconfig to generate the above file…

## Update NuttX Config
make menuconfig \
  && make savedefconfig \
  && grep -v CONFIG_HOST defconfig \
  >boards/risc-v/bl808/ox64/configs/nsh/defconfig

To Find Menuconfig Settings: Press “/” and enter the name of the setting, like “16550_ADDRWIDTH”. This ensures that the Kconfig Dependencies are correctly updated.

UART IRQ

How did we get IRQ 69 for UART?

We set IRQ 69 for UART0…

That’s because the SG2000 Reference Manual (Page 13) says…

3.1 Interrupt Subsystem

Table 3.2: Interrupt number and Interrupt source mapping for Master RISCV C906 @ 1.0Ghz

Int #44: UART0

Linux Device Tree also says UART0 IRQ is 44 (0x2C)

serial@04140000 {
  compatible = "snps,dw-apb-uart";
  reg = <0x00 0x4140000 0x00 0x1000>;
  clock-frequency = <0x17d7840>;
  reg-shift = <0x02>;
  reg-io-width = <0x04>;
  status = "okay";
  interrupts = <0x2c 0x04>;
  interrupt-parent = <0x04>;
};

Thus we compute NuttX IRQ = 25 + RISC-V IRQ = 69

(We should fix the UART Clock: 16550_UART0_CLOCK)

Whither PLIC?

§16.3 Disable Interrupt Controller

Most RISC-V SBCs (Ox64, Star64) will manage Interrupts with a Platform-Level Interrupt Controller (PLIC). But PLIC isn’t documented for SG2000. (Pic above sigh)

Initially we disable PLIC in NuttX…

Later we’ll dump the SG2000 Linux Device Tree to understand the Interrupt Controller.

§16.4 Dump the Linux Device Tree

To understand the Interrupt Controller: We dump the Linux Device Tree for SG2000.

Based on the SG2000 Debian Image, thanks to Justin Hammond (Fishwaldo)…

We download the Latest Release for Milk-V Duo S (SG2000)…

Then copy out the SG2000 Device Tree Binary: cv181x_milkv_duos_sd.dtb

And convert it to Device Tree Source: cv181x_milkv_duos_sd.dts

## Convert the SG2000 Device Tree
dtc \
  -o cv181x_milkv_duos_sd.dts \
  -O dts \
  -I dtb \
  cv181x_milkv_duos_sd.dtb

We go inside the Device Tree…

§16.5 Interrupt Controller for SG2000

Earlier we dumped the Linux Device Tree for SG2000. We snoop inside to understand the Interrupt Controller: cv181x_milkv_duos_sd.dts

// PLIC Interrupt Controller for External Interrupts
interrupt-controller@70000000 {
  riscv,ndev = <0x65>;
  riscv,max-priority = <0x07>;
  reg-names = "control";
  reg = <0x00 0x70000000 0x00 0x4000000>;
  interrupts-extended = <0x16 0xffffffff 0x16 0x09>;
  interrupt-controller;
  compatible = "riscv,plic0";
  #interrupt-cells = <0x02>;
  #address-cells = <0x00>;
  phandle = <0x04>;
};

// CLINT Interrupt Controller for Internal Interrupts
clint@74000000 {
  interrupts-extended = <0x16 0x03 0x16 0x07>;
  reg = <0x00 0x74000000 0x00 0x10000>;
  compatible = "riscv,clint0";
  clint,has-no-64bit-mmio;
};

We see that PLIC (External Interrupts) is at 0x7000_0000, CLINT (Internal Interrupts) at 0x7400_0000…

Platform-Level Interrupt Controller

§16.6 Fix the NuttX Driver for PLIC

Based on the PLIC Address from above: We fix the Platform-Level Interrupt Controller (PLIC) for SG2000…

Now we see a bit more NuttX…

Starting kernel ...
123ABCnx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
nxtask_activate: lpwork pid=1,TCB=0x80409130
nxtask_activate: AppBringUp pid=2,TCB=0x80409740
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
nxtask_activate: /system/bin/init pid=3,TCB=0x8040b730
nxtask_exit: AppBringUp pid=2,TCB=0x80409740

Nuttnx_start: CPU0: Beginning Idle Loop

(See the Complete Log)

Why did it stop?

Duh we set the wrong UART0 IRQ! Here’s the fix…

§16.7 Enable Logging for Scheduler

For easier troubleshooting: We enable Logging for NuttX Scheduler and Binary Loader…

Here’s the NuttX Config: nsh/defconfig

CONFIG_DEBUG_BINFMT=y
CONFIG_DEBUG_BINFMT_ERROR=y
CONFIG_DEBUG_BINFMT_WARN=y
CONFIG_DEBUG_SCHED=y
CONFIG_DEBUG_SCHED_ERROR=y
CONFIG_DEBUG_SCHED_INFO=y
CONFIG_DEBUG_SCHED_WARN=y

Remember: Always use make menuconfig to update the settings!

§16.8 NuttX Crash Dump

What happens when something goes wrong in NuttX?

We’ll see a NuttX Crash Dump, like so…

Booting using the fdt blob at 0x81200000
Loading Ramdisk to 9fe00000, end 9fe00000 ... OK
Loading Device Tree to 000000009f26f000, end 000000009f27e43a ... OK
Starting kernel ...

123ABCnx_start: Entry
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
nxtask_activate: lpwork pid=1,TCB=0x80408130
nxtask_activate: AppBringUp pid=2,TCB=0x80408740
nx_start_application: Starting init task: /system/bin/init
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3

_assert: Current Version: NuttX  12.4.0 f37a380-dirty May  7 2024 10:31:33 risc-v
_assert: Assertion failed 0x17 == (insn & 0x7F): at file: machine/risc-v/arch_elf.c:494 task: AppBringUp process: Kernel 0x80200f34
up_dump_register: EPC: 000000008021087a
up_dump_register: A0: 0000000080401b70 A1: 00000000000001ee A2: 0000000080228ef8 A3: 0000000000000000
up_dump_register: A4: 0000000000000017 A5: 0000000000000002 A6: 000000000000ab9c A7: fffffffffffffff8
up_dump_register: T0: 000000000000002e T1: 0000000000000007 T2: 00000000000001ff T3: 000000008040c3fc
up_dump_register: T4: 000000008040c3f0 T5: 0000000000000009 T6: 000000000000002a
up_dump_register: S0: 0000000000000000 S1: 0000000080408740 S2: 0000000000000017 S3: 0000000000000000
up_dump_register: S4: 0000000080228ef8 S5: 0000000080228de8 S6: 0000000080401e10 S7: 8000000201842022
up_dump_register: S8: 00000000000001ee S9: 000000008040b9a0 S10: 0000000000000070 S11: 000000008040b990
up_dump_register: SP: 000000008040c330 FP: 0000000000000000 TP: 0000000000000000 RA: 000000008021087a
dump_stack: User Stack:
dump_stack:   base: 0x8040c030
dump_stack:   size: 00002000
dump_stack:     sp: 0x8040c330

(See the Complete Log)

What’s this Assertion Failure?

_assert: Assertion failed 0x17 == (insn & 0x7F):
at file: machine/risc-v/arch_elf.c:494
task: AppBringUp process: Kernel 0x80200f34

Oops we goofed and used the Wrong U-Boot Command…

## Nope! This won't work for NuttX. RAM Disk Address must be `-`!
setenv tftp_server 192.168.31.10 ; dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000 ;
tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb ; fdt addr ${fdt_addr_r} ;
booti ${kernel_addr_r} ${ramdisk_addr_r}:${ramdisk_size} ${fdt_addr_r}

Which overwrites the NuttX Image in RAM. Here’s the Correct U-Boot Command…

## This works OK for NuttX. RAM Disk Address must be `-`!
setenv tftp_server 192.168.31.10 ; dhcp ${kernel_addr_r} ${tftp_server}:Image-sg2000 ;
tftpboot ${fdt_addr_r} ${tftp_server}:cv181x_milkv_duos_sd.dtb ; fdt addr ${fdt_addr_r} ; 
booti ${kernel_addr_r} - ${fdt_addr_r}