Inspecting the RISC-V Linux Images for Star64 JH7110 SBC

📝 30 Jun 2023

Pine64 Star64 64-bit RISC-V SBC

Pine64 Star64 is a new 64-bit RISC-V Single-Board Computer, based on the StarFive JH7110 SoC.

(Star64 version 1.1 was released May 2023)

In this article we’ll…

We won’t actually run anything on Star64 yet. We’ll save the fun parts for the next article!

What’s NuttX?

Apache NuttX is a Real-Time Operating System (RTOS) that runs on many kinds of devices, from 8-bit to 64-bit.

The analysis that we do today will be super helpful for porting NuttX to Star64.

Let’s inspect the microSD Images…

“All we need is a microSD”

“All we need is a microSD”

1 Linux Images for Star64

According to Software Releases for Star64, we have these Linux Images…

What about other Linux Distros?

Linux on RISC-V is in Active Development, many distros are not quite ready for the StarFive JH7110 SoC.

Check out the current state of RISC-V Linux…

Armbian Image for Star64

2 Armbian Image for Star64

We begin with the Armbian Image for Star64

Uncompress the .xz file, mount the .img file on Linux / macOS / Windows as an ISO Volume.

The pic above shows that the Armbian Image contains 1 used partition: armbi_root (612 MB), that contains the Linux Root Filesystem.

Plus one unused partition (4 MB) at the top. (Partition Table)

What will happen when it boots?

Let’s check the configuration for U-Boot Bootloader at /boot/extlinux/extlinux.conf

label Armbian
  kernel /boot/Image
  initrd /boot/uInitrd
  fdt /boot/dtb/starfive/jh7110-star64-pine64.dtb
  append root=UUID=99f62df4-be35-475c-99ef-2ba3f74fe6b5 console=ttyS0,115200n8 console=tty0 earlycon=sbi rootflags=data=writeback stmmaceth=chain_mode:1 rw rw no_console_suspend consoleblank=0 fsck.fix=yes fsck.repair=yes net.ifnames=0 splash plymouth.ignore-serial-consoles

(“extlinux/extlinux.conf” is specified by U-Boot’s boot_syslinux_conf)

This says that U-Boot will load the Linux Kernel Image from /boot/Image.

(Which is sym-linked to /boot/vmlinuz-5.15.0-starfive2)

Where in RAM will the Kernel Image be loaded?

According to kernel_addr_r from the Default U-Boot Settings, the Linux Kernel will be loaded at RAM Address 0x4020 0000

kernel_addr_r=0x40200000

(Source)

Everything looks hunky dory?

Nope the Flattened Device Tree (FDT) is missing!

fdt /boot/dtb/starfive/jh7110-star64-pine64.dtb

Which means that Armbian will fail to boot on Star64!

Retrieving file: /boot/uInitrd
  10911538 bytes read in 466 ms (22.3 MiB/s)
Retrieving file: /boot/Image
  22040576 bytes read in 936 ms (22.5 MiB/s)
Retrieving file: /boot/dtb/starfive/jh7110-star64-pine64.dtb
  Failed to load '/boot/dtb/starfive/jh7110-star64-pine64.dtb'

(Source)

The missing Device Tree is noted in this Pine64 Forum Post. So we might need to check back later for the Official Armbian Image, if it’s fixed.

(balbes150 suggests that we try this Armbian Image instead)

For Reference: Here’s the list of Supported Device Trees

→ ls /Volumes/armbi_root/boot/dtb-5.15.0-starfive2/starfive
evb-overlay                      jh7110-evb-usbdevice.dtb
jh7110-evb-can-pdm-pwmdac.dtb    jh7110-evb.dtb
jh7110-evb-dvp-rgb2hdmi.dtb      jh7110-fpga.dtb
jh7110-evb-i2s-ac108.dtb         jh7110-visionfive-v2-A10.dtb
jh7110-evb-pcie-i2s-sd.dtb       jh7110-visionfive-v2-A11.dtb
jh7110-evb-spi-uart2.dtb         jh7110-visionfive-v2-ac108.dtb
jh7110-evb-uart1-rgb2hdmi.dtb    jh7110-visionfive-v2-wm8960.dtb
jh7110-evb-uart4-emmc-spdif.dtb  jh7110-visionfive-v2.dtb
jh7110-evb-uart5-pwm-i2c-tdm.dtb vf2-overlay

And here are the other files in /boot

→ ls -l /Volumes/armbi_root/boot
total 94416
lrwxrwxrwx       24 Image -> vmlinuz-5.15.0-starfive2
-rw-r--r--  4276712 System.map-5.15.0-starfive2
-rw-r--r--     1536 armbian_first_run.txt.template
-rw-r--r--    38518 boot.bmp
-rw-r--r--   144938 config-5.15.0-starfive2
lrwxrwxrwx       20 dtb -> dtb-5.15.0-starfive2
drwxr-xr-x        0 dtb-5.15.0-starfive2
drwxrwxr-x        0 extlinux
lrwxrwxrwx       27 initrd.img -> initrd.img-5.15.0-starfive2
-rw-r--r-- 10911474 initrd.img-5.15.0-starfive2
lrwxrwxrwx       27 initrd.img.old -> initrd.img-5.15.0-starfive2
-rw-rw-r--      341 uEnv.txt
lrwxrwxrwx       24 uInitrd -> uInitrd-5.15.0-starfive2
-rw-r--r-- 10911538 uInitrd-5.15.0-starfive2
lrwxrwxrwx       24 vmlinuz -> vmlinuz-5.15.0-starfive2
-rw-r--r-- 22040576 vmlinuz-5.15.0-starfive2
lrwxrwxrwx       24 vmlinuz.old -> vmlinuz-5.15.0-starfive2

What’s initrd?

initrd /boot/uInitrd

initrd is the Initial RAM Disk that will be loaded into RAM while starting the Linux Kernel.

According to the U-Boot Bootloader Log

  1. Initial RAM Disk will be loaded first:

    /boot/uInitrd

  2. Followed by Linux Kernel:

    /boot/Image

  3. Then Device Tree

    (Which is missing)

Let’s compare Armbian with Yocto…

Yocto Image for Star64

3 Yocto Image for Star64

The Yocto Image for Star64 looks more complicated than Armbian (but it works)…

Uncompress the .bz2 file, rename as .img.

(Balena Etcher won’t work with .bz2 files!)

Write the .img file to a microSD Card with Balena Etcher or GNOME Disks.

Insert the microSD Card into a Linux Computer. (Like Pinebook Pro)

From the pic above, we see 4 used partitions…

Plus one unused partition (2 MB) at the top. (Partition Table)

What will happen when it boots?

boot partition has 2 files…

$ ls -l /run/media/luppy/boot
total 14808
-rw-r--r-- 15151064 fitImage
-rw-r--r--     1562 vf2_uEnv.txt

/boot/vf2_uEnv.txt contains the configuration for U-Boot Bootloader

## This is the sample jh7110_uEnv.txt file for starfive visionfive U-boot
## The current convention (SUBJECT TO CHANGE) is that this file
## will be loaded from the third partition on the
## MMC card.
partnum=3

## The FIT file to boot from
fitfile=fitImage

## for addr info
fileaddr=0xa0000000
fdtaddr=0x46000000
## boot Linux flat or compressed 'Image' stored at 'kernel_addr_r'
kernel_addr_r=0x40200000
irdaddr=46100000
irdsize=5f00000
...

(See the Complete File)

kernel_addr_r says that Linux Kernel will be loaded at RAM Address 0x4020 0000

## boot Linux flat or compressed 'Image' stored at 'kernel_addr_r'
kernel_addr_r=0x40200000

Also different from Armbian: Yocto boots from the Flat Image Tree (FIT) at /boot/fitImage

## The FIT file to boot from
fitfile=fitImage

Which packs everything into a Single FIT File: Kernel Image, RAM Disk, Device Tree

Loading kernel from FIT Image at a0000000 ...
Loading ramdisk from FIT Image at a0000000 ...
Loading fdt from FIT Image at a0000000 ...

(Source)

Yocto’s /root/boot looks different from Armbian…

$ ls -l /run/media/luppy/root/boot
total 24376
lrwxrwxrwx       17 fitImage -> fitImage-5.15.107
-rw-r--r--  9807808 fitImage-5.15.107
-rw-r--r-- 15151064 fitImage-initramfs-5.15.107

Yocto looks more complicated than Armbian, but it boots OK on Star64!

How will Star64 boot from the spl and uboot partitions?

Normally we don’t! (SPL and U-Boot from Star64’s Internal Flash Memory will work OK)

But if we need to (for testing), flip the DIP Switches and set GPIO 0 = High, GPIO 1 = Low.

(DIP Switch Labels are inverted: “ON” actually means “Low”)

U-Boot Bootloader Log

4 Boot NuttX with U-Boot Bootloader

When we port NuttX RTOS to Star64…

Will NuttX boot with Armbian or Yocto settings?

Armbian looks simpler than Yocto, since it uses a plain Kernel Image File /boot/Image.

(Instead of Yocto’s complicated Flat Image Tree)

(UPDATE: We switched to Flat Image Tree for NuttX)

Hence for NuttX we’ll adopt the Armbian Boot Settings, overwriting /boot/Image by our NuttX Kernel Image.

And hopefully U-Boot Bootloader will boot NuttX on Star64! Assuming that we fix these…

Let’s figure out the special File Format for /boot/Image

Armbian Kernel Image

5 Inside the Kernel Image

What’s inside the Linux Kernel Image?

Let’s look inside the Armbian Kernel Image at /boot/Image.

(Which is sym-linked to /boot/vmlinuz-5.15.0-starfive2)

Open the file with a Hex Editor. (Pic above)

See the “RISCV” at 0x30? That’s the Magic Number for the RISC-V Linux Image Header!

u32 code0;                /* Executable code */
u32 code1;                /* Executable code */
u64 text_offset;          /* Image load offset, little endian */
u64 image_size;           /* Effective Image size, little endian */
u64 flags;                /* kernel flags, little endian */
u32 version;              /* Version of this header */
u32 res1 = 0;             /* Reserved */
u64 res2 = 0;             /* Reserved */
u64 magic = 0x5643534952; /* Magic number, little endian, "RISCV" */
u32 magic2 = 0x05435352;  /* Magic number 2, little endian, "RSC\x05" */
u32 res3;                 /* Reserved for PE COFF offset */

Our NuttX Kernel shall recreate this RISC-V Linux Image Header.

(Or U-Boot Bootloader might refuse to boot NuttX)

This is how we decode the RISC-V Linux Header…

Why does the pic show “MZ” at 0x0? Who is “MZ”?

To solve the “MZ” Mystery, we decompile the Linux Kernel…

6 Decompile the Kernel with Ghidra

Can we actually see the RISC-V Code inside the Linux Kernel?

Yep! Let’s decompile the Armbian Kernel with Ghidra, the popular tool for Reverse Engineering…

  1. In Ghidra, create a New Project

  2. Click File > Import File

  3. Select boot/vmlinuz-5.15.0-starfive2 and enter these Import Options…

    Format: Raw Binary

    Language: RISCV > RV64GC (RISCV:LE:64:RV64GC:gcc)

    (StarFive JH7110 has 4 × RV64GC U74 Application Cores)

    (RV64GC is short for RV64IMAFDCZicsr_Zifencei)

    Options > Base Address: 0x40200000

    (Based on the U-Boot Configuration from above)

    (Ghidra thinks it’s PE Format because of “MZ”… But it’s not!)

    Load the Armbian Linux Kernel Image into Ghidra

    Load the Armbian Linux Kernel Image into Ghidra

    (TODO: Pic should show 0x4020 0000 instead)

  4. In the Ghidra Project, double-click vmlinuz-5.15.0-starfive2

    Analyse the file with the Default Options.

Wait a while and we’ll see the Decompiled Linux Kernel in Ghidra…

Disassembled Linux Kernel in Ghidra

At Address 0x4020 0002 we see a Jump to FUN_402010c8.

Double-click FUN_402010c8 to see the Linux Boot Code…

Linux Boot Code in Ghidra

The CSR Instructions look interesting, but we’ll skip them today.

When we match the RISC-V Instructions, the Linux Kernel Source File is probably this…

The first RISC-V Instruction looks kinda sus…

// Load -13 into Register S4
li  s4,-0xd

// Jump to Actual Boot Code
j   FUN_402010c8

It’s highly sus because the First Instruction doesn’t do anything meaningful!

Remember the “MZ” at the top of our Kernel Image?

Armbian Kernel Image

For Legacy Reasons, the Linux Kernel embeds “MZ” to signify that it’s a PE / COFF File, to look like a UEFI Application.

The RISC-V Instruction li assembles into Machine Code as “MZ”. That’s why it’s the first instruction in the Linux Kernel!

We’ll recreate “MZ” in our NuttX Kernel.

(“MZ” refers to Mark Zbikowski)

(Linux Kernel pretends to be a DOS File… NuttX Kernel pretends to be Linux. Hilarious!)

Yocto Linux with KDE Plasma on Star64

Yocto Linux with KDE Plasma on Star64

7 What’s Next

Today we completed our Linux Homework… Without a Star64 SBC!

Please join me in the next article as we actually boot Linux on Star64! (Pic above)

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/star64.md

Armbian Kernel Image

8 Appendix: Decode the RISC-V Linux Header

What’s inside the RISC-V Linux Header?

Earlier we downloaded the Armbian Kernel Image

Let’s decode the RISC-V Linux Header at /boot/Image. (Pic above)

(Which is sym-linked to /boot/vmlinuz-5.15.0-starfive2)

We dump the bytes in the file…

hexdump vmlinuz-5.15.0-starfive2 

Which will begin with this RISC-V Linux Image Header

Here are the decoded bytes…

  1. code0: Executable code

    (4 bytes, offset 0x00)

    4D  5A  6F  10
    
  2. code1: Executable code

    (4 bytes, offset 0x04)

    60  0C  01  00  
    
  3. text_offset: Image load offset, little endian

    (8 bytes, offset 0x08)

    00  00  20  00  00  00  00  00
    
  4. image_size: Effective Image size, little endian

    (8 bytes, offset 0x10)

    00  C0  56  01  00  00  00  00
    
  5. flags: Kernel flags, little endian

    (8 bytes, offset 0x18)

    00  00  00  00  00  00  00  00
    
  6. version: Version of this header (MinL MinM . MajL MajM)

    (4 bytes, offset 0x20)

    02  00  00  00  
    
  7. res1: Reserved

    (4 bytes, offset 0x24)

    00  00  00  00  
    
  8. res2: Reserved

    (8 bytes, offset 0x28)

    00  00  00  00  00  00  00  00
    
  9. magic: Magic number, little endian, “RISCV\x00\x00\x00”

    (8 bytes, offset 0x30)

    52  49  53  43  56  00  00  00  
    
  10. magic2: Magic number 2, little endian, “RSC\x05”

    (4 bytes, offset 0x38)

    52  53  43  05
    
  11. res3: Reserved for PE COFF offset

    (4 bytes, offset 0x3C)

    40  00  00  00
    

Our NuttX Kernel shall recreate this RISC-V Linux Image Header. (Total 0x40 bytes)

(Or U-Boot Bootloader might refuse to boot NuttX)

Why is the Image Load Offset set to 0x20 0000?

Image Load Offset (from Start of RAM) is set to 0x20 0000 because the Linux Kernel boots at 0x4020 0000.

The Image Load Offset is hardcoded in the Linux Kernel Boot Code for 64-bit RISC-V…

#ifdef CONFIG_RISCV_M_MODE
  /* If running at Machine Privilege Level... */
  /* Image Load Offset is 0 MB from Start of RAM */
  .dword 0

#else
  #if __riscv_xlen == 64
    /* If running at 64-bit Supervisor Privilege Level... */
    /* Image Load Offset is 2 MB from Start of RAM */
    .dword 0x200000

  #else
    /* If running at 32-bit Supervisor Privilege Level... */
    /* Image Load Offset is 4 MB from Start of RAM */
    .dword 0x400000
  #endif
#endif

(Source)

We’ll do the same for NuttX.