Star64 JH7110 + NuttX RTOS: Creating the First Release for the RISC-V SBC

📝 7 Aug 2023

Apache NuttX RTOS boots OK on Star64 JH7110 SBC

(Also on Hackster.io)

Apache NuttX Real-Time Operating System (RTOS) now officially supports Pine64 Star64 64-bit RISC-V Single-Board Computer! (Pic below)

(Works on StarFive VisionFive2 SBC too, since both SBCs are based on StarFive JH7110 SoC)

In this article we explain how we created the First Release of NuttX for Star64 JH7110…

Which is probably helpful for folks who wish to…

(Watch the Demo Video on YouTube)

Star64 RISC-V SBC

§1 Build NuttX for Star64

Let’s walk through the steps to build NuttX for Star64

  1. Install the NuttX Build Prerequisites, skip the RISC-V Toolchain…

    “Install Prerequisites”

  2. Download the RISC-V Toolchain riscv64-unknown-elf

    “Download Toolchain for 64-bit RISC-V”

    Add the downloaded toolchain to the PATH Environment Variable.

    Check the RISC-V Toolchain:

    $ riscv64-unknown-elf-gcc -v
  3. Download the NuttX Repositories

    $ mkdir nuttx
    $ cd nuttx
    $ git clone https://github.com/apache/nuttx.git nuttx
    $ git clone https://github.com/apache/nuttx-apps apps
  4. Configure and build the NuttX Project

    $ cd nuttx
    $ tools/configure.sh star64:nsh
    $ make
    $ riscv64-unknown-elf-objcopy -O binary nuttx nuttx.bin

    This produces the NuttX Kernel nuttx.bin

    (Missing math.h? See this)

  5. Build the NuttX Apps Filesystem

    $ make export
    $ pushd ../apps
    $ tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
    $ make import
    $ popd
    $ genromfs -f initrd -d ../apps/bin -V "NuttXBootVol"

    This generates the Initial RAM Disk initrd

    (Inside a ROM FS Filesystem)

  6. Download the Device Tree jh7110-visionfive-v2.dtb from StarFive VisionFive2 Software Releases.

    Save it into the nuttx folder. Or do this…

    $ wget https://github.com/starfive-tech/VisionFive2/releases/download/VF2_v3.1.5/jh7110-visionfive-v2.dtb

    (NuttX doesn’t need a Device Tree, but U-Boot Bootloader needs it)

  7. (Optional) For easier debugging, we might create the following…

    ## Copy the config
    $ cp .config nuttx.config
    
    ## Dump the Kernel disassembly to `nuttx.S`
    $ riscv64-unknown-elf-objdump \
      -t -S --demangle --line-numbers --wide \
      nuttx \
      >nuttx.S \
      2>&1
    
    ## Dump the NSH `init` disassembly to `init.S`
    $ riscv64-unknown-elf-objdump \
      -t -S --demangle --line-numbers --wide \
      ../

    (See the Build Outputs)

    (See the Build Steps)

    (See the Build Log)

    (GitHub Actions Workflow)

    (Automated Daily Builds)

    (Shell Script to Build and Run NuttX)

Now we create a Bootable microSD…

NuttX goes into the FAT Partition on the microSD

NuttX goes into the FAT Partition on the microSD

§2 NuttX in a Bootable microSD

How do we create a Bootable microSD for NuttX?

From the previous section, we have…

Now we pack all 3 files into a Flat Image Tree (FIT)

Inside the nuttx folder, create a Text File named nuttx.its with the following content: nuttx.its

/dts-v1/;

/ {
  description = "NuttX FIT image";
  #address-cells = <2>;

  images {
    vmlinux {
      description = "vmlinux";
      data = /incbin/("./nuttx.bin");
      type = "kernel";
      arch = "riscv";
      os = "linux";
      load = <0x0 0x40200000>;
      entry = <0x0 0x40200000>;
      compression = "none";
    };

    ramdisk {
      description = "buildroot initramfs";
      data = /incbin/("./initrd");
      type = "ramdisk";
      arch = "riscv";
      os = "linux";
      load = <0x0 0x46100000>;
      compression = "none";
      hash-1 {
        algo = "sha256";
      };
    };

    fdt {
      data = /incbin/("./jh7110-visionfive-v2.dtb");
      type = "flat_dt";
      arch = "riscv";
      load = <0x0 0x46000000>;
      compression = "none";
      hash-1 {
        algo = "sha256";
      };
    };
  };

  configurations {
    default = "nuttx";

    nuttx {
      description = "NuttX";
      kernel = "vmlinux";
      fdt = "fdt";
      loadables = "ramdisk";
    };
  };
};

Or do this…

$ wget https://raw.githubusercontent.com/lupyuen/nuttx-star64/main/nuttx.its

Package the NuttX Kernel, Initial RAM Disk and Device Tree into a Flat Image Tree…

## For macOS:
$ brew install u-boot-tools
## For Linux:
$ sudo apt install u-boot-tools

## Generate FIT Image from `nuttx.bin`, 
## `initrd` and `jh7110-visionfive-v2.dtb`.
## `nuttx.its` must be in the same 
## directory as the NuttX binaries!
$ mkimage \
  -f nuttx.its \
  -A riscv \
  -O linux \
  -T flat_dt \
  starfiveu.fit

## To check FIT image
$ mkimage -l starfiveu.fit

We’ll see the NuttX Kernel

→ mkimage -f nuttx.its -A riscv -O linux -T flat_dt starfiveu.fit
FIT description: NuttX FIT image
Created:         Fri Aug  4 23:20:52 2023
 Image 0 (vmlinux)
  Description:  vmlinux
  Created:      Fri Aug  4 23:20:52 2023
  Type:         Kernel Image
  Compression:  uncompressed
  Data Size:    2097800 Bytes = 2048.63 KiB = 2.00 MiB
  Architecture: RISC-V
  OS:           Linux
  Load Address: 0x40200000
  Entry Point:  0x40200000

Followed by the Initial RAM Disk (containing NuttX Apps)…

 Image 1 (ramdisk)
  Description:  buildroot initramfs
  Created:      Fri Aug  4 23:20:52 2023
  Type:         RAMDisk Image
  Compression:  uncompressed
  Data Size:    8086528 Bytes = 7897.00 KiB = 7.71 MiB
  Architecture: RISC-V
  OS:           Linux
  Load Address: 0x46100000
  Entry Point:  unavailable
  Hash algo:    sha256
  Hash value:   44b3603e6e611ade7361a936aab09def23651399d4a0a3c284f47082d788e877

Finally the Device Tree (not used by NuttX)…

 Image 2 (fdt)
  Description:  unavailable
  Created:      Fri Aug  4 23:20:52 2023
  Type:         Flat Device Tree
  Compression:  uncompressed
  Data Size:    50235 Bytes = 49.06 KiB = 0.05 MiB
  Architecture: RISC-V
  Load Address: 0x46000000
  Hash algo:    sha256
  Hash value:   42767c996f0544f513280805b41f996446df8b3956c656bdbb782125ae8ffeec
 Default Configuration: 'nuttx'
 Configuration 0 (nuttx)
  Description:  NuttX
  Kernel:       vmlinux
  FDT:          fdt
  Loadables:    ramdisk

This produces the Flat Image Tree starfiveu.fit, which we’ll copy later to a microSD Card.

To prepare the microSD Card, download the microSD Image sdcard.img from StarFive VisionFive2 Software Releases.

Write the downloaded image to a microSD Card with Balena Etcher or GNOME Disks. (Or use dd)

Copy the file starfiveu.fit from above and overwrite the file on the microSD Card…

## For macOS: Copy to microSD
cp starfiveu.fit "/Volumes/NO NAME"

## For macOS: Unmount the microSD
## TODO: Verify that /dev/disk2 is microSD
diskutil unmountDisk /dev/disk2

We’re ready to boot NuttX on Star64!

(More about Flat Image Tree)

(How sdcard.img was created)

Apache NuttX RTOS boots OK on Star64 JH7110 SBC

§3 Boot NuttX on Star64

Connect Star64 to our computer with a USB Serial Adapter

Insert the microSD Card into Star64 and power up Star64. NuttX boots on Star64 and NuttShell (nsh) appears in the Serial Console.

To see the available commands in NuttShell:

$ help

(Watch the Demo Video on YouTube)

(See the NuttX Log)

Booting NuttX over TFTP is also supported on Star64.

Booting NuttX over the Network with TFTP

§4 NuttX Startup Explained

Step by step, here’s everything that happens when NuttX boots on our SBC…

OpenSBI and U-Boot Bootloader

  1. OpenSBI (Supervisor Binary Interface) is the first thing that boots on our RISC-V SBC.

    OpenSBI provides Secure Access to the Low-Level System Functions (controlling CPUs, Timers, Interrupts) for the JH7110 SoC.

    OpenSBI boots in RISC-V Machine Mode, the most powerful mode in a RISC-V system.

    “OpenSBI Supervisor Binary Interface”

    OpenSBI starts U-Boot Bootloader

  2. U-Boot Bootloader starts after OpenSBI, in RISC-V Supervisor Mode.

    (Which is less powerful than Machine Mode)

    “U-Boot Bootloader for Star64”

    Star64 boots with NuttX Kernel, Device Tree and Initial RAM Disk

  3. U-Boot Bootloader loads into RAM the NuttX Kernel, Device Tree and Initial RAM Disk.

    Inside the Initial RAM Disk: NuttX Shell and NuttX Apps.

  4. NuttX Kernel starts in RISC-V Supervisor Mode and executes the NuttX Boot Code (in RISC-V Assembly).

    “NuttX in Supervisor Mode (Boot Code)”

  5. NuttX Start Code (in C) runs next.

    It prepares the RISC-V Memory Management Unit, to protect the Kernel Memory and I/O Memory.

    “Initialise RISC-V Supervisor Mode: jh7110_start” and jh7110_start_s

  6. NuttX Kernel (nx_start) starts the NuttX Drivers and mounts the Initial RAM Disk (containing the NuttX Shell and Apps)

  7. Followed by the NuttX Shell (NSH), for the Command-Line Interface

    “NuttX Apps Filesystem: init / nsh”

Finally we talk about the NuttX Shell and Apps

  1. NuttX Shell (NSH) and NuttX Apps (like “hello”) will run in RISC-V User Mode.

    (Which is the least powerful mode)

  2. They will make System Calls to NuttX Kernel, jumping from User Mode to Supervisor Mode. (And back)

    “ECALL from RISC-V User Mode to Supervisor Mode”

  3. System Calls will happen when NuttX Shell and Apps do Console Output and Console Input.

    “Serial Output in NuttX”

    “Serial Input in NuttX”

  4. Which will trigger RISC-V Interrupts in the 16550 UART Controller.

    “Platform-Level Interrupt Controller”

    UART Interrupts are handled by the RISC-V Platform-Level Interrupt Controller (PLIC)

And that’s everything that happens when NuttX boots on Star64!

But NuttX doesn’t actually need a Device Tree!

Yeah but because the Flat Image Tree needs a Device Tree, we do it anyway.

We created all this code from scratch?

Actually most of the code came from NuttX for QEMU RISC-V! (In Kernel Mode)

It’s amazing that we reused so much code from NuttX QEMU. And ported everything to Star64 within 2 months!

What’s the catch?

We have some Size Limitations on the Initial RAM Disk, NuttX Apps and NuttX Stacks. Here are the workarounds…

Porting Linux / Unix / POSIX Apps to NuttX might need extra work, check out this example…

Pull Request for NuttX Board

§5 Add the NuttX Arch and Board

How did we add Star64 JH7110 to NuttX?

When we add a new board to NuttX, we do it in 4 steps…

  1. Patch the NuttX Dependencies

    (Like this)

  2. Add the NuttX Arch (JH7110 SoC)

    (Like this)

  3. Add the NuttX Board (Star64 SBC)

    (Like this)

  4. Update the NuttX Documentation

    (Also here)

This is how we did it for Star64 SBC (with JH7110 SoC) in 3 Pull Requests

§5.1 Patch the NuttX Dependencies

First we patch any NuttX Dependencies needed by Star64 JH7110.

We discovered that JH7110 triggers too many spurious UART interrupts

JH7110 uses a Synopsys DesignWare 8250 UART that has a peculiar problem with the Line Control Register (LCR)… If we write to LCR while the UART is busy, it will trigger spurious UART Interrupts.

The fix is to wait for the UART to be not busy before writing to LCR. We submitted this Pull Request to fix the NuttX 16550 UART Driver

Wait for the Pull Request to be merged. Then we add the NuttX Arch…

(How to submit a Pull Request for NuttX)

§5.2 Add the NuttX Arch

Next we submit the Pull Request that implements the support for JH7110 SoC as a NuttX Arch

We insert JH7110 SoC into the Kconfig for the RISC-V SoCs: arch/risc-v/Kconfig

config ARCH_CHIP_JH7110
	bool "StarFive JH7110"
	select ARCH_RV64
	select ARCH_RV_ISA_M
	select ARCH_RV_ISA_A
	select ARCH_RV_ISA_C
	select ARCH_HAVE_FPU
	select ARCH_HAVE_DPFPU
	select ARCH_HAVE_MULTICPU
	select ARCH_HAVE_MPU
	select ARCH_HAVE_MMU
	select ARCH_MMU_TYPE_SV39
	select ARCH_HAVE_ADDRENV
	select ARCH_NEED_ADDRENV_MAPPING
	select ARCH_HAVE_S_MODE
	select ONESHOT
	select ALARM_ARCH
	---help---
		StarFive JH7110 SoC.
...
config ARCH_CHIP
	...
	default "jh7110"	if ARCH_CHIP_JH7110
...
if ARCH_CHIP_JH7110
source "arch/risc-v/src/jh7110/Kconfig"
endif

(Remember to indent with Tabs, not Spaces!)

And we create a Kconfig for JH7110 SoC: arch/risc-v/src/jh7110/Kconfig

Then we add the NuttX Arch Source Files for JH7110 SoC at…

§5.3 Add the NuttX Board

Finally we create the Pull Request that implements the support for Star64 SBC as a NuttX Board

We insert Star64 SBC into the Kconfig for NuttX Boards: nuttx/boards/Kconfig

config ARCH_BOARD_JH7110_STAR64
	bool "PINE64 Star64"
	depends on ARCH_CHIP_JH7110
	---help---
		This options selects support for NuttX on PINE64 Star64 based
		on StarFive JH7110 SoC.
...
config ARCH_BOARD
	...
	default "star64"                    if ARCH_BOARD_JH7110_STAR64
...
if ARCH_BOARD_JH7110_STAR64
source "boards/risc-v/jh7110/star64/Kconfig"
endif

(Remember to indent with Tabs, not Spaces!)

We create a Kconfig for Star64 SBC: nuttx/boards/risc-v/jh7110/star64/Kconfig

And we add the NuttX Board Source Files for Star64 SBC at…

But don’t submit the Pull Request yet! We’ll add the NuttX Documentation in the next section.

We’re good for RISC-V. What about Arm?

This is how we add a NuttX Arch and Board for Arm64

Though we probably should have split it into multiple Pull Requests like for RISC-V.

Seems we need to copy a bunch of source files across branches?

No sweat! Suppose we created a staging Pull Request in our own repo…

This command produces a list of Modified Files in our Pull Request

## TODO: Change this to your PR
pr=https://github.com/lupyuen2/wip-nuttx/pull/40
curl -L $pr.diff \
  | grep "diff --git" \
  | sort \
  | cut -d" " -f3 \
  | cut -c3-

Like this…

boards/risc-v/jh7110/star64/include/board.h
boards/risc-v/jh7110/star64/include/board_memorymap.h
boards/risc-v/jh7110/star64/scripts/Make.defs
boards/risc-v/jh7110/star64/scripts/ld.script

That we can copy to another branch in a simple script…

b=$HOME/new_branch
mkdir -p $b/boards/risc-v/jh7110/star64/include
mkdir -p $b/boards/risc-v/jh7110/star64/scripts
a=boards/risc-v/jh7110/star64/include/board.h
cp $a $b/$a
a=boards/risc-v/jh7110/star64/include/board_memorymap.h
cp $a $b/$a
a=boards/risc-v/jh7110/star64/scripts/Make.defs
cp $a $b/$a
a=boards/risc-v/jh7110/star64/scripts/ld.script
cp $a $b/$a

How did we generate the NuttX Build Configuration?

The NuttX Build Configuration for Star64 is at…

We generated the defconfig with this command…

make menuconfig \
  && make savedefconfig \
  && grep -v CONFIG_HOST defconfig \
  >boards/risc-v/jh7110/star64/configs/nsh/defconfig

(How we computed the UART Clock)

During development, we should enable additional Debug Options

CONFIG_DEBUG_ASSERTIONS=y
CONFIG_DEBUG_ASSERTIONS_EXPRESSION=y
CONFIG_DEBUG_BINFMT=y
CONFIG_DEBUG_BINFMT_ERROR=y
CONFIG_DEBUG_BINFMT_WARN=y
CONFIG_DEBUG_ERROR=y
CONFIG_DEBUG_FEATURES=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_FS_ERROR=y
CONFIG_DEBUG_FS_WARN=y
CONFIG_DEBUG_FULLOPT=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_MM=y
CONFIG_DEBUG_MM_ERROR=y
CONFIG_DEBUG_MM_WARN=y
CONFIG_DEBUG_SCHED=y
CONFIG_DEBUG_SCHED_ERROR=y
CONFIG_DEBUG_SCHED_INFO=y
CONFIG_DEBUG_SCHED_WARN=y
CONFIG_DEBUG_SYMBOLS=y
CONFIG_DEBUG_WARN=y

(Source)

Before merging with NuttX Mainline, remember to remove the Debug Options for BINFMT, FS, MM and SCHED.

JH7110 NuttX Arch

§6 Update the NuttX Docs

Earlier we created a Pull Request to add a new NuttX Board.

In the same Pull Request, we update the NuttX Docs like so…

Create a page for the JH7110 NuttX Arch (pic above)…

Documentation/platforms/risc-v/ jh7110/index.rst

===============
StarFive JH7110
===============

`StarFive JH7110 <https://doc-en.rvspace.org/Doc_Center/jh7110.html>`_ is a 64-bit RISC-V SoC that features:

- **CPU:** SiFive RISC-V U74 Application Cores (4 cores, RV64GCB) and SiFive RISC-V S7 Monitor Core (single core, RV64IMACB)
...

(Which goes under “RISC-V” because it’s a RISC-V SoC)

(Tips for Restructured Text)

Star64 NuttX Board

Under JH7110, create a page for the Star64 NuttX Board (pic above)…

Documentation/platforms/risc-v/ jh7110/boards/star64/index.rst

=============
PINE64 Star64
=============

`Star64 <https://wiki.pine64.org/wiki/STAR64>`_ is a 64-bit RISC-V based
Single Board Computer powered by StarFive JH7110 Quad-Core SiFive U74 64-Bit CPU,
Imagination Technology BX-4-32 GPU and supports up to 8GB 1866MHz LPDDR4 memory.
...

On that page, remember to document the steps to Build and Boot NuttX

To update and preview the NuttX Docs, follow the instructions here…

And now we’re ready to submit the Pull Request. That’s how we add a NuttX Arch and NuttX Board!

Testing Apache NuttX RTOS on Star64 JH7110 SBC

§7 Upcoming Features

How will we create the missing drivers for Star64 JH7110?

We welcome your contribution to NuttX!

Based on the official docs…

We have started working on the HDMI Support for NuttX on Star64 JH7110…

Here are the relevant docs for the other JH7110 Peripherals…

GPIO:

I2C:

RTC, SPI, UART, DMA, I2S, PWM:

USB, Ethernet:

Image Sensor Processor:

Display:

We hope to test NuttX soon on the PineTab-V RISC-V Tablet. Stay tuned for updates!

TODO: Fix the System Timer by calling OpenSBI, similar to Ox64 BL808

Apache NuttX RTOS boots OK on StarFive VisionFive2 SBC

§8 What’s Next

Today we finally have NuttX running on a Single-Board Computer: Star64 JH7110 SBC! (And StarFive VisionFive2, pic above)

Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/release.md

§9 Appendix: Missing Math.h

Why did the NuttX Build fail with missing math.h?

$ sudo apt install \
  gcc-riscv64-unknown-elf \
  picolibc-riscv64-unknown-elf

$ make
./stdio/lib_dtoa_engine.c:40:10:
  fatal error: math.h: No such file or directory
  #include <math.h>

If the NuttX Build fails due to missing math.h, install the xPack GNU RISC-V Embedded GCC Toolchain

Is there another solution?

Here’s a quick hack: Edit the file nuttx/.config and add…

NEED_MATH_H=y
CONFIG_LIBM=y

This fixes the NuttX Build to use the NuttX Version of math.h. (Instead of the System Version)

NuttX Kernel will boot OK if we don’t actually use any Math Functions. But NuttX Apps will fail to load if they call Math Functions. (Like floor)

(See this)

(More about CONFIG_LIBM)

(Thanks to Ken Dickey for the tip!)

§10 Appendix: StarFive VisionFive2 Software Release

The StarFive VisionFive2 Software Release was helpful for creating the Bootable microSD for NuttX.

The Software Release includes an SD Card Image that boots OK on Star64…

(We reused the SD Card Image for NuttX)

Based on the files above, we figured out how to generate the Flat Image Tree for NuttX…

Also we see the script that generates the SD Card Image: genimage.sh

genimage \
  --rootpath "${ROOTPATH_TMP}"     \
  --tmppath "${GENIMAGE_TMP}"    \
  --inputpath "${INPUT_DIR}"  \
  --outputpath "${OUTPUT_DIR}" \
  --config genimage-vf2.cfg

The SD Card Partitions are defined here: genimage-vf2.cfg

image sdcard.img {
  hdimage {
    gpt = true
  }

  partition spl {
    image = "work/u-boot-spl.bin.normal.out"
    partition-type-uuid = 2E54B353-1271-4842-806F-E436D6AF6985
    offset = 2M
    size = 2M
  }

  partition uboot {
    image = "work/visionfive2_fw_payload.img"
    partition-type-uuid = 5B193300-FC78-40CD-8002-E86C45580B47
    offset = 4M
    size = 4M
  }

  partition image {
    # partition-type = 0xC
    partition-type-uuid = EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
    image = "work/starfive-visionfive2-vfat.part"
    offset = 8M
    size = 292M
  }

  partition root {
    # partition-type = 0x83
    partition-type-uuid = 0FC63DAF-8483-4772-8E79-3D69D8477DE4
    image = "work/buildroot_rootfs/images/rootfs.ext4"
    offset = 300M
    bootable = true
  }
}

Useful for creating our own SD Card Partitions!

(We won’t need the spl, uboot and root partitions for NuttX)

§11 Appendix: UART Clock for JH7110

How did we figure out the UART Clock for JH7110?

CONFIG_16550_UART0_CLOCK=23040000

(Source)

We logged the values of DLM and DLL in the UART Driver during startup…

uint32_t dlm = u16550_serialin(priv, UART_DLM_OFFSET);
uint32_t dll = u16550_serialin(priv, UART_DLL_OFFSET);

(We capture DLM and DLL only when DLAB=1)

(Be careful to print only when DLAB=0)

According to our log, DLM is 0 and DLL is 13. Which means..

dlm =  0 = (div >> 8)
dll = 13 = (div & 0xff)

Which gives div=13. Now since baud=115200 at startup…

div = (uartclk + (baud << 3)) / (baud << 4)
13  = (uartclk + 921600) / 1843200
uartclk = (13 * 1843200) - 921600
        = 23040000

Thus uartclk=23040000. And that’s why we set…

CONFIG_16550_UART0_CLOCK=23040000

(Source)

Shouldn’t we get the UART Clock from the SoC Datasheet?

Yep absolutely! For JH7110 there isn’t a complete Datasheet, the docs only point to the Linux Device Tree. That’s why we derived the UART Clock ourselves.

Sometimes we work backwards when porting NuttX to SBCs

  1. Assume that the UART Driver is already configured by U-Boot

  2. Get the simple UART Driver working, without configuring the UART

  3. Figure out the right values of DLL and DLM, so that UART Driver will configure the UART correctly

  4. DLL and DLM will be reused by other UARTs: UART 1, 2, 3, …

For Ox64 BL808 SBC: We skipped the UART Configuration completely. Which is OK because we won’t use the other UART Ports anyway…

So if we’re building the UART Driver ourselves and it’s incomplete, it’s OK to upstream it first and complete it later. That’s how we did it for PinePhone on Allwinner A64