📝 30 Jun 2023
Pine64 Star64 is a new 64-bit RISC-V Single-Board Computer, based on the StarFive JH7110 SoC.
(Star64 version 1.1 was released May 2023)
In this article we’ll…
Look inside the brand new Linux Images for Star64
Decompile (with Ghidra) the RISC-V Linux Kernel
Figure out how Apache NuttX RTOS might run on Star64
We won’t actually run anything on Star64 yet. We’ll save the fun parts for the next article!
What’s NuttX?
Apache NuttX is a Real-Time Operating System (RTOS) that runs on many kinds of devices, from 8-bit to 64-bit.
The analysis that we do today will be super helpful for porting NuttX to Star64.
Let’s inspect the microSD Images…
“All we need is a microSD”
According to Software Releases for Star64, we have these Linux Images…
Let’s inspect Armbian 23.8 Lunar (Minimal)
Yocto Images at pine64.my-ho.st
We pick star64-image-minimal 1.2
What about other Linux Distros?
Linux on RISC-V is in Active Development, many distros are not quite ready for the StarFive JH7110 SoC.
Check out the current state of RISC-V Linux…
We begin with the Armbian Image for Star64…
Uncompress the .xz file, mount the .img file on Linux / macOS / Windows as an ISO Volume.
The pic above shows that the Armbian Image contains 1 used partition: armbi_root (612 MB), that contains the Linux Root Filesystem.
Plus one unused partition (4 MB) at the top. (Partition Table)
What will happen when it boots?
Let’s check the configuration for U-Boot Bootloader at /boot/extlinux/extlinux.conf…
label Armbian
kernel /boot/Image
initrd /boot/uInitrd
fdt /boot/dtb/starfive/jh7110-star64-pine64.dtb
append root=UUID=99f62df4-be35-475c-99ef-2ba3f74fe6b5 console=ttyS0,115200n8 console=tty0 earlycon=sbi rootflags=data=writeback stmmaceth=chain_mode:1 rw rw no_console_suspend consoleblank=0 fsck.fix=yes fsck.repair=yes net.ifnames=0 splash plymouth.ignore-serial-consoles
(“extlinux/extlinux.conf” is specified by U-Boot’s boot_syslinux_conf)
This says that U-Boot will load the Linux Kernel Image from /boot/Image.
(Which is sym-linked to /boot/vmlinuz-5.15.0-starfive2)
Where in RAM will the Kernel Image be loaded?
According to kernel_addr_r from the Default U-Boot Settings, the Linux Kernel will be loaded at RAM Address 0x4020 0000…
kernel_addr_r=0x40200000
Everything looks hunky dory?
Nope the Flattened Device Tree (FDT) is missing!
fdt /boot/dtb/starfive/jh7110-star64-pine64.dtb
Which means that Armbian will fail to boot on Star64!
Retrieving file: /boot/uInitrd
10911538 bytes read in 466 ms (22.3 MiB/s)
Retrieving file: /boot/Image
22040576 bytes read in 936 ms (22.5 MiB/s)
Retrieving file: /boot/dtb/starfive/jh7110-star64-pine64.dtb
Failed to load '/boot/dtb/starfive/jh7110-star64-pine64.dtb'
The missing Device Tree is noted in this Pine64 Forum Post. So we might need to check back later for the Official Armbian Image, if it’s fixed.
(balbes150 suggests that we try this Armbian Image instead)
For Reference: Here’s the list of Supported Device Trees…
→ ls /Volumes/armbi_root/boot/dtb-5.15.0-starfive2/starfive
evb-overlay jh7110-evb-usbdevice.dtb
jh7110-evb-can-pdm-pwmdac.dtb jh7110-evb.dtb
jh7110-evb-dvp-rgb2hdmi.dtb jh7110-fpga.dtb
jh7110-evb-i2s-ac108.dtb jh7110-visionfive-v2-A10.dtb
jh7110-evb-pcie-i2s-sd.dtb jh7110-visionfive-v2-A11.dtb
jh7110-evb-spi-uart2.dtb jh7110-visionfive-v2-ac108.dtb
jh7110-evb-uart1-rgb2hdmi.dtb jh7110-visionfive-v2-wm8960.dtb
jh7110-evb-uart4-emmc-spdif.dtb jh7110-visionfive-v2.dtb
jh7110-evb-uart5-pwm-i2c-tdm.dtb vf2-overlay
And here are the other files in /boot…
→ ls -l /Volumes/armbi_root/boot
total 94416
lrwxrwxrwx 24 Image -> vmlinuz-5.15.0-starfive2
-rw-r--r-- 4276712 System.map-5.15.0-starfive2
-rw-r--r-- 1536 armbian_first_run.txt.template
-rw-r--r-- 38518 boot.bmp
-rw-r--r-- 144938 config-5.15.0-starfive2
lrwxrwxrwx 20 dtb -> dtb-5.15.0-starfive2
drwxr-xr-x 0 dtb-5.15.0-starfive2
drwxrwxr-x 0 extlinux
lrwxrwxrwx 27 initrd.img -> initrd.img-5.15.0-starfive2
-rw-r--r-- 10911474 initrd.img-5.15.0-starfive2
lrwxrwxrwx 27 initrd.img.old -> initrd.img-5.15.0-starfive2
-rw-rw-r-- 341 uEnv.txt
lrwxrwxrwx 24 uInitrd -> uInitrd-5.15.0-starfive2
-rw-r--r-- 10911538 uInitrd-5.15.0-starfive2
lrwxrwxrwx 24 vmlinuz -> vmlinuz-5.15.0-starfive2
-rw-r--r-- 22040576 vmlinuz-5.15.0-starfive2
lrwxrwxrwx 24 vmlinuz.old -> vmlinuz-5.15.0-starfive2
What’s initrd?
initrd /boot/uInitrd
initrd is the Initial RAM Disk that will be loaded into RAM while starting the Linux Kernel.
According to the U-Boot Bootloader Log…
Initial RAM Disk will be loaded first:
/boot/uInitrd
Followed by Linux Kernel:
/boot/Image
Then Device Tree
(Which is missing)
Let’s compare Armbian with Yocto…
The Yocto Image for Star64 looks more complicated than Armbian (but it works)…
Uncompress the .bz2 file, rename as .img.
(Balena Etcher won’t work with .bz2 files!)
Write the .img file to a microSD Card with Balena Etcher or GNOME Disks.
Insert the microSD Card into a Linux Computer. (Like Pinebook Pro)
From the pic above, we see 4 used partitions…
spl (2 MB): For Secondary Program Loader
uboot (4 MB): For U-Boot Bootloader
boot (380 MB): U-Boot Configuration and Linux Kernel Image
root (686 MB): Linux Root Filesystem
Plus one unused partition (2 MB) at the top. (Partition Table)
What will happen when it boots?
boot partition has 2 files…
$ ls -l /run/media/luppy/boot
total 14808
-rw-r--r-- 15151064 fitImage
-rw-r--r-- 1562 vf2_uEnv.txt
/boot/vf2_uEnv.txt contains the configuration for U-Boot Bootloader…
## This is the sample jh7110_uEnv.txt file for starfive visionfive U-boot
## The current convention (SUBJECT TO CHANGE) is that this file
## will be loaded from the third partition on the
## MMC card.
partnum=3
## The FIT file to boot from
fitfile=fitImage
## for addr info
fileaddr=0xa0000000
fdtaddr=0x46000000
## boot Linux flat or compressed 'Image' stored at 'kernel_addr_r'
kernel_addr_r=0x40200000
irdaddr=46100000
irdsize=5f00000
...
kernel_addr_r says that Linux Kernel will be loaded at RAM Address 0x4020 0000…
## boot Linux flat or compressed 'Image' stored at 'kernel_addr_r'
kernel_addr_r=0x40200000
Also different from Armbian: Yocto boots from the Flat Image Tree (FIT) at /boot/fitImage
## The FIT file to boot from
fitfile=fitImage
Which packs everything into a Single FIT File: Kernel Image, RAM Disk, Device Tree…
Loading kernel from FIT Image at a0000000 ...
Loading ramdisk from FIT Image at a0000000 ...
Loading fdt from FIT Image at a0000000 ...
Yocto’s /root/boot looks different from Armbian…
$ ls -l /run/media/luppy/root/boot
total 24376
lrwxrwxrwx 17 fitImage -> fitImage-5.15.107
-rw-r--r-- 9807808 fitImage-5.15.107
-rw-r--r-- 15151064 fitImage-initramfs-5.15.107
Yocto looks more complicated than Armbian, but it boots OK on Star64!
How will Star64 boot from the spl and uboot partitions?
Normally we don’t! (SPL and U-Boot from Star64’s Internal Flash Memory will work OK)
But if we need to (for testing), flip the DIP Switches and set GPIO 0 = High, GPIO 1 = Low.
(DIP Switch Labels are inverted: “ON” actually means “Low”)
When we port NuttX RTOS to Star64…
Will NuttX boot with Armbian or Yocto settings?
Armbian looks simpler than Yocto, since it uses a plain Kernel Image File /boot/Image.
(Instead of Yocto’s complicated Flat Image Tree)
(UPDATE: We switched to Flat Image Tree for NuttX)
Hence for NuttX we’ll adopt the Armbian Boot Settings, overwriting /boot/Image by our NuttX Kernel Image.
And hopefully U-Boot Bootloader will boot NuttX on Star64! Assuming that we fix these…
Compile NuttX Kernel to boot at 0x4020 0000
Add a placeholder for Device Tree (since it’s missing)
Use the special File Format for Linux Kernel Image (“MZ”)
Let’s figure out the special File Format for /boot/Image…
What’s inside the Linux Kernel Image?
Let’s look inside the Armbian Kernel Image at /boot/Image.
(Which is sym-linked to /boot/vmlinuz-5.15.0-starfive2)
Open the file with a Hex Editor. (Pic above)
See the “RISCV” at 0x30? That’s the Magic Number for the RISC-V Linux Image Header!
u32 code0; /* Executable code */
u32 code1; /* Executable code */
u64 text_offset; /* Image load offset, little endian */
u64 image_size; /* Effective Image size, little endian */
u64 flags; /* kernel flags, little endian */
u32 version; /* Version of this header */
u32 res1 = 0; /* Reserved */
u64 res2 = 0; /* Reserved */
u64 magic = 0x5643534952; /* Magic number, little endian, "RISCV" */
u32 magic2 = 0x05435352; /* Magic number 2, little endian, "RSC\x05" */
u32 res3; /* Reserved for PE COFF offset */
Our NuttX Kernel shall recreate this RISC-V Linux Image Header.
(Or U-Boot Bootloader might refuse to boot NuttX)
This is how we decode the RISC-V Linux Header…
Why does the pic show “MZ” at 0x0? Who is “MZ”?
To solve the “MZ” Mystery, we decompile the Linux Kernel…
Can we actually see the RISC-V Code inside the Linux Kernel?
Yep! Let’s decompile the Armbian Kernel with Ghidra, the popular tool for Reverse Engineering…
In Ghidra, create a New Project
Click File > Import File
Select boot/vmlinuz-5.15.0-starfive2 and enter these Import Options…
Format: Raw Binary
Language: RISCV > RV64GC (RISCV:LE:64:RV64GC:gcc)
(StarFive JH7110 has 4 × RV64GC U74 Application Cores)
(RV64GC is short for RV64IMAFDCZicsr_Zifencei)
Options > Base Address: 0x40200000
(Based on the U-Boot Configuration from above)
(Ghidra thinks it’s PE Format because of “MZ”… But it’s not!)
(TODO: Pic should show 0x4020 0000 instead)
In the Ghidra Project, double-click vmlinuz-5.15.0-starfive2
Analyse the file with the Default Options.
Wait a while and we’ll see the Decompiled Linux Kernel in Ghidra…
At Address 0x4020 0002 we see a Jump to FUN_402010c8.
Double-click FUN_402010c8 to see the Linux Boot Code…
The CSR Instructions look interesting, but we’ll skip them today.
When we match the RISC-V Instructions, the Linux Kernel Source File is probably this…
The first RISC-V Instruction looks kinda sus…
// Load -13 into Register S4
li s4,-0xd
// Jump to Actual Boot Code
j FUN_402010c8
It’s highly sus because the First Instruction doesn’t do anything meaningful!
Remember the “MZ” at the top of our Kernel Image?
For Legacy Reasons, the Linux Kernel embeds “MZ” to signify that it’s a PE / COFF File, to look like a UEFI Application.
The RISC-V Instruction li assembles into Machine Code as “MZ”. That’s why it’s the first instruction in the Linux Kernel!
We’ll recreate “MZ” in our NuttX Kernel.
(“MZ” refers to Mark Zbikowski)
(Linux Kernel pretends to be a DOS File… NuttX Kernel pretends to be Linux. Hilarious!)
Yocto Linux with KDE Plasma on Star64
Today we completed our Linux Homework… Without a Star64 SBC!
We inspected the brand new Linux Images for Star64
We decompiled with Ghidra the RISC-V Linux Kernel
And we figured out how Apache NuttX RTOS might boot on Star64
Please join me in the next article as we actually boot Linux on Star64! (Pic above)
Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…
lupyuen.github.io/src/star64.md
What’s inside the RISC-V Linux Header?
Earlier we downloaded the Armbian Kernel Image…
Let’s decode the RISC-V Linux Header at /boot/Image. (Pic above)
(Which is sym-linked to /boot/vmlinuz-5.15.0-starfive2)
We dump the bytes in the file…
hexdump vmlinuz-5.15.0-starfive2
Which will begin with this RISC-V Linux Image Header…
Here are the decoded bytes…
code0: Executable code
(4 bytes, offset 0x00)
4D 5A 6F 10
code1: Executable code
(4 bytes, offset 0x04)
60 0C 01 00
text_offset: Image load offset, little endian
(8 bytes, offset 0x08)
00 00 20 00 00 00 00 00
image_size: Effective Image size, little endian
(8 bytes, offset 0x10)
00 C0 56 01 00 00 00 00
flags: Kernel flags, little endian
(8 bytes, offset 0x18)
00 00 00 00 00 00 00 00
version: Version of this header (MinL MinM . MajL MajM)
(4 bytes, offset 0x20)
02 00 00 00
res1: Reserved
(4 bytes, offset 0x24)
00 00 00 00
res2: Reserved
(8 bytes, offset 0x28)
00 00 00 00 00 00 00 00
magic: Magic number, little endian, “RISCV\x00\x00\x00”
(8 bytes, offset 0x30)
52 49 53 43 56 00 00 00
magic2: Magic number 2, little endian, “RSC\x05”
(4 bytes, offset 0x38)
52 53 43 05
res3: Reserved for PE COFF offset
(4 bytes, offset 0x3C)
40 00 00 00
Our NuttX Kernel shall recreate this RISC-V Linux Image Header. (Total 0x40 bytes)
(Or U-Boot Bootloader might refuse to boot NuttX)
Why is the Image Load Offset set to 0x20 0000?
Image Load Offset (from Start of RAM) is set to 0x20 0000 because the Linux Kernel boots at 0x4020 0000.
The Image Load Offset is hardcoded in the Linux Kernel Boot Code for 64-bit RISC-V…
#ifdef CONFIG_RISCV_M_MODE
/* If running at Machine Privilege Level... */
/* Image Load Offset is 0 MB from Start of RAM */
.dword 0
#else
#if __riscv_xlen == 64
/* If running at 64-bit Supervisor Privilege Level... */
/* Image Load Offset is 2 MB from Start of RAM */
.dword 0x200000
#else
/* If running at 32-bit Supervisor Privilege Level... */
/* Image Load Offset is 4 MB from Start of RAM */
.dword 0x400000
#endif
#endif
We’ll do the same for NuttX.