Star64 JH7110 RISC-V SBC: Experiments with OpenSBI (Supervisor Binary Interface)

đź“ť 29 Oct 2023

OpenSBI on Star64 JH7110 RISC-V SBC

Bare Metal Programming on a RISC-V SBC (Single-Board Computer) sounds difficult… Thankfully we can get help from the OpenSBI Supervisor Binary Interface!

(A little like BIOS, but for RISC-V)

In this article, we call OpenSBI to…

We’ll do this on the Star64 JH7110 RISC-V SBC. (Pic below)

(The same steps will work OK on StarFive VisionFive2, Milk-V Mars and other SBCs based on the StarFive JH7110 SoC)

We’re running Bare Metal Code on our SBC?

Not quite, but close to the Metal!

We’re running our code with Apache NuttX Real-Time Operating System (RTOS). NuttX lets us inject our Test Code into its tiny Kernel and boot it easily on our SBC.

(Without messing around with the Linux Kernel)

Why are we doing this?

Right now we’re porting NuttX RTOS to the Star64 SBC.

The experiments that we run today will be super helpful as we integrate NuttX with OpenSBI for System Timers, CPU Scheduling and other System Functions.

Pine64 Star64 RISC-V SBC

1 OpenSBI Supervisor Binary Interface

What’s this OpenSBI?

OpenSBI (Open Source Supervisor Binary Interface) is the first thing that boots on our JH7110 RISC-V SBC…

U-Boot SPL 2021.10 (Jan 19 2023 - 04:09:41 +0800)
DDR version: dc2e84f0.
Trying to boot from SPI
OpenSBI v1.2
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|
Platform Name             : StarFive VisionFive V2
Platform Features         : medeleg
Platform HART Count       : 5
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 4000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : jh7110-hsm
Platform PMU Device       : ---
Platform Reboot Device    : pm-reset
Platform Shutdown Device  : pm-reset
Firmware Base             : 0x40000000
Firmware Size             : 288 KB
Runtime SBI Version       : 1.0

(Source)

OpenSBI provides Secure Access to the Low-Level System Functions (controlling CPUs, Timers, Interrupts) for the JH7110 SoC, as described in the SBI Spec…

Can we access the Low-Level System Features without OpenSBI?

Our code runs in RISC-V Supervisor Mode, which doesn’t allow direct access to Low-Level System Features, like for starting a CPU. (Pic below)

(NuttX Kernel, Linux Kernel and U-Boot Bootloader all run in Supervisor Mode)

OpenSBI runs in RISC-V Machine Mode, which has complete access to Low-Level System Features. That’s why we call OpenSBI from our code.

OpenSBI runs in RISC-V Machine Mode

2 Call OpenSBI from NuttX

How to call OpenSBI from our code?

Suppose we’re calling OpenSBI to print something to the Serial Console like so: jh7110_appinit.c

// After NuttX Kernel boots on JH7110...
void board_late_initialize(void) {
  ...
  // Call OpenSBI to print something
  test_opensbi();
}

// Call OpenSBI to print something. Based on
// https://github.com/riscv-software-src/opensbi/blob/master/firmware/payloads/test_main.c
// https://www.thegoodpenguin.co.uk/blog/an-overview-of-opensbi/
int test_opensbi(void) {

  // Print `123` with (Legacy) Console Putchar.
  // Call sbi_console_putchar: Extension ID 1, Function ID 0
  // https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/src/ext-legacy.adoc
  sbi_ecall(
    SBI_EXT_0_1_CONSOLE_PUTCHAR,  // Extension ID: 1
    0,    // Function ID: 0
    '1',  // Character to be printed
    0, 0, 0, 0, 0  // Other Parameters (unused)
  );

  // Do the same, but print `2` and `3`
  sbi_ecall(SBI_EXT_0_1_CONSOLE_PUTCHAR, 0, '2', 0, 0, 0, 0, 0);
  sbi_ecall(SBI_EXT_0_1_CONSOLE_PUTCHAR, 0, '3', 0, 0, 0, 0, 0);

This calls the (Legacy) Console Putchar Function from the SBI Spec…

(There’s a newer version of this, we’ll soon see)

What’s this ecall to SBI?

Remember that OpenSBI runs in (super-privileged) RISC-V Machine Mode. And our code runs in (less-privileged) RISC-V Supervisor Mode.

To jump from Supervisor Mode to Machine Mode, we execute the ecall RISC-V Instruction like this: jh7110_appinit.c

// Make an `ecall` to OpenSBI. Based on
// https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/supervisor/riscv_sbi.c#L52-L77
// https://github.com/riscv-software-src/opensbi/blob/master/firmware/payloads/test_main.c
static struct sbiret sbi_ecall(
  unsigned int extid,  // Extension ID
  unsigned int fid,    // Function ID
  uintptr_t parm0, uintptr_t parm1,  // Parameters 0 and 1
  uintptr_t parm2, uintptr_t parm3,  // Parameters 2 and 3
  uintptr_t parm4, uintptr_t parm5   // Parameters 4 and 5
) {
  // Pass the Extension ID, Function ID and Parameters
  // in RISC-V Registers A0 to A7
  register long r0 asm("a0") = (long)(parm0);
  register long r1 asm("a1") = (long)(parm1);
  register long r2 asm("a2") = (long)(parm2);
  register long r3 asm("a3") = (long)(parm3);
  register long r4 asm("a4") = (long)(parm4);
  register long r5 asm("a5") = (long)(parm5);
  register long r6 asm("a6") = (long)(fid);
  register long r7 asm("a7") = (long)(extid);

  // Execute the `ecall` RISC-V Instruction
  // Input+Output Registers: A0 and A1
  // Input-Only Registers: A2 to A7
  // Clobbers the Memory
  asm volatile (
    "ecall"
    : "+r"(r0), "+r"(r1)
    : "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r6), "r"(r7)
    : "memory"
  );

  // Return the OpenSBI Error and Value
  struct sbiret ret;
  ret.error = r0;
  ret.value = r1;
  return ret;
}

(sbiret is defined here)

(See the RISC-V Disassembly)

Now we run this on our SBC…

NuttX calls OpenSBI on Star64 JH7110 RISC-V SBC

3 Run NuttX with OpenSBI

Will our Test Code print correctly to the Serial Console?

Let’s find out!

  1. Follow these steps to download Apache NuttX RTOS and compile the NuttX Kernel and Apps…

    “Build NuttX for Star64”

  2. Locate this NuttX Source File…

    nuttx/boards/risc-v/jh7110/star64/src/jh7110_appinit.c
    

    Replace the contents of that file by this Test Code…

    jh7110_appinit.c: OpenSBI Test Code

  3. Rebuild the NuttX Kernel…

    $ make
    $ riscv64-unknown-elf-objcopy -O binary nuttx nuttx.bin
    
  4. Copy the NuttX Kernel and NuttX Apps to a microSD Card…

    “NuttX in a Bootable microSD”

  5. Insert the microSD Card into our SBC and power up…

    “Boot NuttX on Star64”

    (Or boot our SBC over the Network with TFTP)

When we boot the Modified NuttX Kernel on our SBC, we see “123” printed on the Serial Console (pic above)…

Starting kernel ...
123
NuttShell (NSH) NuttX-12.0.3
nsh> 

(Source)

Our OpenSBI Experiment works OK yay!

NuttX calls OpenSBI Debug Console on Star64 JH7110 RISC-V SBC

4 OpenSBI Debug Console

But that’s calling the Legacy Console Putchar Function…

What about the newer Debug Console Functions?

Yeah we called the Legacy Console Putchar Function, which is expected to be deprecated.

Let’s call the newer Debug Console Functions in OpenSBI. This function prints a string to the Debug Console…

And this function prints a single character…

This is how we print to the Debug Console: jh7110_appinit.c

// Print `456` to Debug Console as a String.
// Call sbi_debug_console_write: EID 0x4442434E "DBCN", FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/src/ext-debug-console.adoc#function-console-write-fid-0
const char *str = "456";
struct sbiret sret = sbi_ecall(
  SBI_EXT_DBCN,  // Extension ID: 0x4442434E "DBCN"
  SBI_EXT_DBCN_CONSOLE_WRITE,  // Function ID: 0
  strlen(str),         // Number of bytes
  (unsigned long)str,  // Address Low
  0,                   // Address High
  0, 0, 0              // Other Parameters (unused)
);
_info("debug_console_write: value=%d, error=%d\n", sret.value, sret.error);

// Print `789` to Debug Console, byte by byte.
// Call sbi_debug_console_write_byte: EID 0x4442434E "DBCN", FID 2
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/src/ext-debug-console.adoc#function-console-write-byte-fid-2
sret = sbi_ecall(
  SBI_EXT_DBCN,  // Extension ID: 0x4442434E "DBCN"
  SBI_EXT_DBCN_CONSOLE_WRITE_BYTE,  // Function ID: 2
  '7',           // Character to be printed
  0, 0, 0, 0, 0  // Other Parameters (unused)
);
_info("debug_console_write_byte: value=%d, error=%d\n", sret.value, sret.error);
// Omitted: Do the same, but print `8` and `9`

But our Test Code fails with error NOT_SUPPORTED (pic above)…

debug_console_write:
  value=0, error=-2

debug_console_write_byte:
  value=0, error=-2

(Source)

Why? Let’s find out…

5 Read the SBI Version

We tried printing to Debug Console but failed…

Maybe OpenSBI in our SBC doesn’t support Debug Console?

Debug Console was introduced in SBI Spec Version 2.0.

To get the SBI Spec Version supported by our SBC, we call this SBI Function…

Like this: jh7110_appinit.c

// Get SBI Spec Version
// Call sbi_get_spec_version: EID 0x10, FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#41-function-get-sbi-specification-version-fid-0
sret = sbi_ecall(
  SBI_EXT_BASE,     // Extension ID: 0x10 
  SBI_EXT_BASE_GET_SPEC_VERSION,  // Function ID: 0
  0, 0, 0, 0, 0, 0  // Parameters (unused)
);
_info("get_spec_version: value=0x%x, error=%d\n", sret.value, sret.error);

Which tells us…

get_spec_version:
  value=0x1000000
  error=0

(Source)

0x100 0000 says that the SBI Spec Version is…

Thus our SBC supports SBI Spec Version 1.0.

Aha! Our SBC doesn’t support Debug Console, because this feature was introduced in Version 2.0!

Mystery solved! Actually if we’re super observant, SBI Version 1.0 also appears when our SBC boots OpenSBI (pic below)…

Runtime SBI Version: 1.0

(Source)

Is our SBC stuck forever with SBI Version 1.0?

Actually we can upgrade OpenSBI by reflashing the Onboard SPI Flash.

But let’s stick with SBI Version 1.0 for now.

(Mainline OpenSBI now supports SBI 2.0 and Debug Console)

SBI v1.0 appears in the JH7110 OpenSBI Log

6 Probe the SBI Extensions

Bummer our SBC doesn’t support Debug Console…

How to check if our SBC supports ANY specific feature?

SBI lets us Probe its Extensions to discover the Supported SBI Extensions (like Debug Console)…

Like this: jh7110_appinit.c

// Probe SBI Extension: Base Extension
// Call sbi_probe_extension: EID 0x10, FID 3
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#44-function-probe-sbi-extension-fid-3
struct sbiret sret = sbi_ecall(
  SBI_EXT_BASE,   // Extension ID: 0x10 
  SBI_EXT_BASE_PROBE_EXT,  // Function ID: 3
  SBI_EXT_BASE,  // Probe for "Base Extension": 0x10
  0, 0, 0, 0, 0  // Other Parameters (unused)
);
_info("probe_extension[0x10]: value=0x%x, error=%d\n", sret.value, sret.error);

// Probe SBI Extension: Debug Console Extension.
// Same as above, but we change the parameter to
// "Debug Console" 0x4442434E.
sret = sbi_ecall(SBI_EXT_BASE, SBI_EXT_BASE_PROBE_EXT, SBI_EXT_DBCN, 0, 0, 0, 0, 0);
_info("probe_extension[0x4442434E]: value=0x%x, error=%d\n", sret.value, sret.error);

Which will show…

probe_extension[0x10]:
  value=0x1, error=0

probe_extension[0x4442434E]:
  value=0x0, error=0

(Source)

Hence we learn that…

Thus we always Probe the Extensions before calling them!

NuttX calls OpenSBI Hart State Management on Star64 JH7110 RISC-V SBC

7 Query the RISC-V CPUs

OK so OpenSBI can do trivial things…

What about controlling the CPUs?

Now we experiment with the RISC-V CPU Cores (“Hart” / Hardware Thread) in our SBC.

We call Hart State Management (HSM) to query the Hart Status…

(Not to be confused with Hardware Security Module)

Here’s how: jh7110_appinit.c

// For each Hart ID from 0 to 5...
for (uintptr_t hart = 0; hart < 6; hart++) {

  // HART Get Status
  // Call sbi_hart_get_status: EID 0x48534D "HSM", FID 2
  // https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#93-function-hart-get-status-fid-2
  struct sbiret sret = sbi_ecall(
    SBI_EXT_HSM,    // Extension ID: 0x48534D "HSM"
    SBI_EXT_HSM_HART_GET_STATUS,   // Function ID: 2
    hart,          // Parameter 0: Hart ID
    0, 0, 0, 0, 0  // Other Parameters (unused)
  );
  _info("hart_get_status[%d]: value=0x%x, error=%d\n", hart, sret.value, sret.error);
}

Our SBC says (pic above)…

hart_get_status[0]: value=0x1, error=0
hart_get_status[1]: value=0x0, error=0
hart_get_status[2]: value=0x1, error=0
hart_get_status[3]: value=0x1, error=0
hart_get_status[4]: value=0x1, error=0
hart_get_status[5]: value=0x0, error=-3

(Source)

When we decode the values, we learn that…

Huh? Why is Hart 0 stopped while Hart 1 is running?

According to the SiFive U74 Manual (Page 96), there are 5 RISC-V Cores in JH7110 (pic below)…

OpenSBI and NuttX will boot on the First Application Core. That’s why Hart 1 is running. (And not Hart 0)

How do we start a Hart?

(With a Defibrillator heh heh)

Check out these SBI Functions…

In future we’ll call these SBI Functions to start NuttX on Multiple CPUs.

(More about Hart States)

5 RISC-V Cores in JH7110 SoC

8 Shutdown and Reboot the SBC

OpenSBI looks mighty powerful. Can it control our ENTIRE SBC?

Yep! OpenSBI supports System Reset for…

Which we call like so: jh7110_appinit.c

// System Reset: Shutdown
// Call sbi_system_reset: EID 0x53525354 "SRST", FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#101-function-system-reset-fid-0
struct sbiret sret = sbi_ecall(
  SBI_EXT_SRST, SBI_EXT_SRST_RESET,  // System Reset
  SBI_SRST_RESET_TYPE_SHUTDOWN,      // Shutdown
  SBI_SRST_RESET_REASON_NONE, 0, 0, 0, 0);

// System Reset: Cold Reboot
sret = sbi_ecall(
  SBI_EXT_SRST, SBI_EXT_SRST_RESET,  // System Reset
  SBI_SRST_RESET_TYPE_COLD_REBOOT,   // Cold Reboot
  SBI_SRST_RESET_REASON_NONE, 0, 0, 0, 0);

// System Reset: Warm Reboot
sret = sbi_ecall(
  SBI_EXT_SRST, SBI_EXT_SRST_RESET,  // System Reset
  SBI_SRST_RESET_TYPE_WARM_REBOOT,   // Warm Reboot
  SBI_SRST_RESET_REASON_NONE, 0, 0, 0, 0);

What happens when we run this?

How do we know that VisionFive2 has a newer build of OpenSBI?

When Star64 boots, this is the first thing that we see…

U-Boot SPL 2021.10 (Jan 19 2023)

(Source)

Which says that Star64 ships with a Secondary Program Loader + OpenSBI + U-Bootloader that was built on 19 Jan 2023.

On VisionFive2, we see a newer date: 21 Jun 2023…

U-Boot SPL 2021.10 (Jun 21 2023)

Thus VisionFive2 ships with a newer build of OpenSBI.

OpenSBI Shutdown on Star64 JH7110 RISC-V SBC

9 Set a System Timer

NuttX / Linux Kernel runs in RISC-V Supervisor Mode (not Machine Mode)…

How will it control the System Timer?

That’s why OpenSBI provides the Set Timer function: jh7110_appinit.c

// Set Timer
// Call sbi_set_timer: EID 0x54494D45 "TIME", FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#61-function-set-timer-fid-0
sret = sbi_ecall(
  SBI_EXT_TIME,  // Extension ID: 0x54494D45 "TIME"
  SBI_EXT_TIME_SET_TIMER,  // Function ID: 0
  0,  // TODO: Absolute Time for Timer Expiry
  0, 0, 0, 0, 0);

It doesn’t seem to do anything…

set_timer:
  value=0x0
  error=0

(Source)

But that’s because our SBC will trigger an interrupt when the System Timer expires.

Someday NuttX will call this function to set the System Timer.

NuttX fetches OpenSBI System Info on Star64 JH7110 RISC-V SBC

10 Fetch the System Info

Earlier we called OpenSBI to fetch the SBI Spec Version…

What else can we fetch from OpenSBI?

We can snoop a whole bunch of System Info like this: jh7110_appinit.c

// Get SBI Implementation ID: EID 0x10, FID 1
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#42-function-get-sbi-implementation-id-fid-1
struct sbiret sret = sbi_ecall(
  SBI_EXT_BASE, SBI_EXT_BASE_GET_IMP_ID, 0, 0, 0, 0, 0, 0);

// Get SBI Implementation Version: EID 0x10, FID 2
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#43-function-get-sbi-implementation-version-fid-2
struct sbiret sret = sbi_ecall(
  SBI_EXT_BASE, SBI_EXT_BASE_GET_IMP_VERSION, 0, 0, 0, 0, 0, 0);

// Get Machine Vendor ID: EID 0x10, FID 4
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#45-function-get-machine-vendor-id-fid-4
sret = sbi_ecall(
  SBI_EXT_BASE, SBI_EXT_BASE_GET_MVENDORID, 0, 0, 0, 0, 0, 0);

// Get Machine Architecture ID: EID 0x10, FID 5
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#46-function-get-machine-architecture-id-fid-5
sret = sbi_ecall(
  SBI_EXT_BASE, SBI_EXT_BASE_GET_MARCHID, 0, 0, 0, 0, 0, 0);

// Get Machine Implementation ID: EID 0x10, FID 6
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#47-function-get-machine-implementation-id-fid-6
sret = sbi_ecall(
  SBI_EXT_BASE, SBI_EXT_BASE_GET_MIMPID, 0, 0, 0, 0, 0, 0);

// Omitted: Print `sret.value` and `sret.error`

Our SBC will print (pic above)…

// OpenSBI Implementation ID is 1
get_impl_id: 0x1

// OpenSBI Version is 1.2
get_impl_version: 0x10002

// RISC-V Vendor is SiFive
get_mvendorid: 0x489

// RISC-V Machine Architecture is SiFive U7 Series
get_marchid: 0x7

// RISC-V Machine Implementation is 0x4210427 (?)
get_mimpid: 0x4210427

(Source)

The last 3 values are documented in the SiFive U74 Manual. (Pages 136 to 137)

11 Integrate OpenSBI with NuttX

Phew that’s plenty of OpenSBI Functions…

How will NuttX use them?

As we port Apache NuttX RTOS to Star64 JH7110 SBC, we shall call…

And we’ll Probe the SBI Extensions before calling them.

We’re not calling the SBI Debug Console?

We’ve already implemented the NuttX UART Driver for JH7110. So we won’t call OpenSBI for Console Input / Output.

But when we port NuttX to a new SBC, we should consider SBI Debug Console for simple debug logging.

(Like for Ox64 BL808)

Can NuttX Apps call OpenSBI?

Nope, only the NuttX Kernel is allowed to call OpenSBI.

That’s because NuttX Apps run in RISC-V User Mode. When NuttX Apps execute the ecall Instruction, they will jump from User Mode to Supervisor Mode to execute NuttX Kernel Functions. (Not OpenSBI Functions)

Thus NuttX Apps are prevented from calling OpenSBI to meddle with CPUs, Timers and Interrupts. (Which should be meddled by the NuttX Kernel anyway)

12 What’s Next

I hope this article has been helpful for learning about OpenSBI and how it works with Apache NuttX RTOS (and Linux)…

  1. We printed to the Serial Console

  2. Set a System Timer

  3. Queried the RISC-V CPUs

  4. Fetched the System Information

  5. And Shutdown / Rebooted our SBC (somewhat)

Stay tuned for more integration with NuttX and OpenSBI!

Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/sbi.md