đź“ť 29 Oct 2023
Bare Metal Programming on a RISC-V SBC (Single-Board Computer) sounds difficult… Thankfully we can get help from the OpenSBI Supervisor Binary Interface!
(A little like BIOS, but for RISC-V)
In this article, we call OpenSBI to…
Print to the Serial Console
Set a System Timer
Query the RISC-V CPUs
Fetch the System Information
And Shutdown / Reboot our SBC
We’ll do this on the Star64 JH7110 RISC-V SBC. (Pic below)
(The same steps will work OK on StarFive VisionFive2, Milk-V Mars and other SBCs based on the StarFive JH7110 SoC)
We’re running Bare Metal Code on our SBC?
Not quite, but close to the Metal!
We’re running our code with Apache NuttX Real-Time Operating System (RTOS). NuttX lets us inject our Test Code into its tiny Kernel and boot it easily on our SBC.
(Without messing around with the Linux Kernel)
Why are we doing this?
Right now we’re porting NuttX RTOS to the Star64 SBC.
The experiments that we run today will be super helpful as we integrate NuttX with OpenSBI for System Timers, CPU Scheduling and other System Functions.
What’s this OpenSBI?
OpenSBI (Open Source Supervisor Binary Interface) is the first thing that boots on our JH7110 RISC-V SBC…
U-Boot SPL 2021.10 (Jan 19 2023 - 04:09:41 +0800)
DDR version: dc2e84f0.
Trying to boot from SPI
OpenSBI v1.2
____ _____ ____ _____
/ __ \ / ____| _ \_ _|
| | | |_ __ ___ _ __ | (___ | |_) || |
| | | | '_ \ / _ \ '_ \ \___ \| _ < | |
| |__| | |_) | __/ | | |____) | |_) || |_
\____/| .__/ \___|_| |_|_____/|____/_____|
| |
|_|
Platform Name : StarFive VisionFive V2
Platform Features : medeleg
Platform HART Count : 5
Platform IPI Device : aclint-mswi
Platform Timer Device : aclint-mtimer @ 4000000Hz
Platform Console Device : uart8250
Platform HSM Device : jh7110-hsm
Platform PMU Device : ---
Platform Reboot Device : pm-reset
Platform Shutdown Device : pm-reset
Firmware Base : 0x40000000
Firmware Size : 288 KB
Runtime SBI Version : 1.0
OpenSBI provides Secure Access to the Low-Level System Functions (controlling CPUs, Timers, Interrupts) for the JH7110 SoC, as described in the SBI Spec…
Can we access the Low-Level System Features without OpenSBI?
Our code runs in RISC-V Supervisor Mode, which doesn’t allow direct access to Low-Level System Features, like for starting a CPU. (Pic below)
(NuttX Kernel, Linux Kernel and U-Boot Bootloader all run in Supervisor Mode)
OpenSBI runs in RISC-V Machine Mode, which has complete access to Low-Level System Features. That’s why we call OpenSBI from our code.
How to call OpenSBI from our code?
Suppose we’re calling OpenSBI to print something to the Serial Console like so: jh7110_appinit.c
// After NuttX Kernel boots on JH7110...
void board_late_initialize(void) {
...
// Call OpenSBI to print something
test_opensbi();
}
// Call OpenSBI to print something. Based on
// https://github.com/riscv-software-src/opensbi/blob/master/firmware/payloads/test_main.c
// https://www.thegoodpenguin.co.uk/blog/an-overview-of-opensbi/
int test_opensbi(void) {
// Print `123` with (Legacy) Console Putchar.
// Call sbi_console_putchar: Extension ID 1, Function ID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/src/ext-legacy.adoc
sbi_ecall(
SBI_EXT_0_1_CONSOLE_PUTCHAR, // Extension ID: 1
0, // Function ID: 0
'1', // Character to be printed
0, 0, 0, 0, 0 // Other Parameters (unused)
);
// Do the same, but print `2` and `3`
sbi_ecall(SBI_EXT_0_1_CONSOLE_PUTCHAR, 0, '2', 0, 0, 0, 0, 0);
sbi_ecall(SBI_EXT_0_1_CONSOLE_PUTCHAR, 0, '3', 0, 0, 0, 0, 0);
This calls the (Legacy) Console Putchar Function from the SBI Spec…
Extension ID: 1 (Console Putchar)
Function ID: 0
Parameter: Character to be printed
(There’s a newer version of this, we’ll soon see)
What’s this ecall to SBI?
Remember that OpenSBI runs in (super-privileged) RISC-V Machine Mode. And our code runs in (less-privileged) RISC-V Supervisor Mode.
To jump from Supervisor Mode to Machine Mode, we execute the ecall RISC-V Instruction like this: jh7110_appinit.c
// Make an `ecall` to OpenSBI. Based on
// https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/supervisor/riscv_sbi.c#L52-L77
// https://github.com/riscv-software-src/opensbi/blob/master/firmware/payloads/test_main.c
static struct sbiret sbi_ecall(
unsigned int extid, // Extension ID
unsigned int fid, // Function ID
uintptr_t parm0, uintptr_t parm1, // Parameters 0 and 1
uintptr_t parm2, uintptr_t parm3, // Parameters 2 and 3
uintptr_t parm4, uintptr_t parm5 // Parameters 4 and 5
) {
// Pass the Extension ID, Function ID and Parameters
// in RISC-V Registers A0 to A7
register long r0 asm("a0") = (long)(parm0);
register long r1 asm("a1") = (long)(parm1);
register long r2 asm("a2") = (long)(parm2);
register long r3 asm("a3") = (long)(parm3);
register long r4 asm("a4") = (long)(parm4);
register long r5 asm("a5") = (long)(parm5);
register long r6 asm("a6") = (long)(fid);
register long r7 asm("a7") = (long)(extid);
// Execute the `ecall` RISC-V Instruction
// Input+Output Registers: A0 and A1
// Input-Only Registers: A2 to A7
// Clobbers the Memory
asm volatile (
"ecall"
: "+r"(r0), "+r"(r1)
: "r"(r2), "r"(r3), "r"(r4), "r"(r5), "r"(r6), "r"(r7)
: "memory"
);
// Return the OpenSBI Error and Value
struct sbiret ret;
ret.error = r0;
ret.value = r1;
return ret;
}
Now we run this on our SBC…
Will our Test Code print correctly to the Serial Console?
Let’s find out!
Follow these steps to download Apache NuttX RTOS and compile the NuttX Kernel and Apps…
Locate this NuttX Source File…
nuttx/boards/risc-v/jh7110/star64/src/jh7110_appinit.c
Replace the contents of that file by this Test Code…
Rebuild the NuttX Kernel…
$ make
$ riscv64-unknown-elf-objcopy -O binary nuttx nuttx.bin
Copy the NuttX Kernel and NuttX Apps to a microSD Card…
Insert the microSD Card into our SBC and power up…
When we boot the Modified NuttX Kernel on our SBC, we see “123” printed on the Serial Console (pic above)…
Starting kernel ...
123
NuttShell (NSH) NuttX-12.0.3
nsh>
Our OpenSBI Experiment works OK yay!
But that’s calling the Legacy Console Putchar Function…
What about the newer Debug Console Functions?
Yeah we called the Legacy Console Putchar Function, which is expected to be deprecated.
Let’s call the newer Debug Console Functions in OpenSBI. This function prints a string to the Debug Console…
Extension ID: 0x4442 434E “DBCN”
Function ID: 0 (Console Write)
Parameter 0: String Length
Parameter 1: Low Address of String
Parameter 2: High Address of String
And this function prints a single character…
Extension ID: 0x4442 434E “DBCN”
Function ID: 2 (Console Write Byte)
Parameter 0: Character to be printed
This is how we print to the Debug Console: jh7110_appinit.c
// Print `456` to Debug Console as a String.
// Call sbi_debug_console_write: EID 0x4442434E "DBCN", FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/src/ext-debug-console.adoc#function-console-write-fid-0
const char *str = "456";
struct sbiret sret = sbi_ecall(
SBI_EXT_DBCN, // Extension ID: 0x4442434E "DBCN"
SBI_EXT_DBCN_CONSOLE_WRITE, // Function ID: 0
strlen(str), // Number of bytes
(unsigned long)str, // Address Low
0, // Address High
0, 0, 0 // Other Parameters (unused)
);
_info("debug_console_write: value=%d, error=%d\n", sret.value, sret.error);
// Print `789` to Debug Console, byte by byte.
// Call sbi_debug_console_write_byte: EID 0x4442434E "DBCN", FID 2
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/src/ext-debug-console.adoc#function-console-write-byte-fid-2
sret = sbi_ecall(
SBI_EXT_DBCN, // Extension ID: 0x4442434E "DBCN"
SBI_EXT_DBCN_CONSOLE_WRITE_BYTE, // Function ID: 2
'7', // Character to be printed
0, 0, 0, 0, 0 // Other Parameters (unused)
);
_info("debug_console_write_byte: value=%d, error=%d\n", sret.value, sret.error);
// Omitted: Do the same, but print `8` and `9`
But our Test Code fails with error NOT_SUPPORTED (pic above)…
debug_console_write:
value=0, error=-2
debug_console_write_byte:
value=0, error=-2
Why? Let’s find out…
We tried printing to Debug Console but failed…
Maybe OpenSBI in our SBC doesn’t support Debug Console?
Debug Console was introduced in SBI Spec Version 2.0.
To get the SBI Spec Version supported by our SBC, we call this SBI Function…
Extension ID: 0x10 (Base Extension)
Function ID: 0 (SBI Spec Version)
Like this: jh7110_appinit.c
// Get SBI Spec Version
// Call sbi_get_spec_version: EID 0x10, FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#41-function-get-sbi-specification-version-fid-0
sret = sbi_ecall(
SBI_EXT_BASE, // Extension ID: 0x10
SBI_EXT_BASE_GET_SPEC_VERSION, // Function ID: 0
0, 0, 0, 0, 0, 0 // Parameters (unused)
);
_info("get_spec_version: value=0x%x, error=%d\n", sret.value, sret.error);
Which tells us…
get_spec_version:
value=0x1000000
error=0
0x100 0000 says that the SBI Spec Version is…
Major Version: 1 (Bits 24 to 30)
Minor Version: 0 (Bits 0 to 23)
Thus our SBC supports SBI Spec Version 1.0.
Aha! Our SBC doesn’t support Debug Console, because this feature was introduced in Version 2.0!
Mystery solved! Actually if we’re super observant, SBI Version 1.0 also appears when our SBC boots OpenSBI (pic below)…
Runtime SBI Version: 1.0
Is our SBC stuck forever with SBI Version 1.0?
Actually we can upgrade OpenSBI by reflashing the Onboard SPI Flash.
But let’s stick with SBI Version 1.0 for now.
(Mainline OpenSBI now supports SBI 2.0 and Debug Console)
Bummer our SBC doesn’t support Debug Console…
How to check if our SBC supports ANY specific feature?
SBI lets us Probe its Extensions to discover the Supported SBI Extensions (like Debug Console)…
Extension ID: 0x10 (Base Extension)
Function ID: 3 (Probe SBI Extension)
Parameter 0: Extension ID to be probed
Like this: jh7110_appinit.c
// Probe SBI Extension: Base Extension
// Call sbi_probe_extension: EID 0x10, FID 3
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#44-function-probe-sbi-extension-fid-3
struct sbiret sret = sbi_ecall(
SBI_EXT_BASE, // Extension ID: 0x10
SBI_EXT_BASE_PROBE_EXT, // Function ID: 3
SBI_EXT_BASE, // Probe for "Base Extension": 0x10
0, 0, 0, 0, 0 // Other Parameters (unused)
);
_info("probe_extension[0x10]: value=0x%x, error=%d\n", sret.value, sret.error);
// Probe SBI Extension: Debug Console Extension.
// Same as above, but we change the parameter to
// "Debug Console" 0x4442434E.
sret = sbi_ecall(SBI_EXT_BASE, SBI_EXT_BASE_PROBE_EXT, SBI_EXT_DBCN, 0, 0, 0, 0, 0);
_info("probe_extension[0x4442434E]: value=0x%x, error=%d\n", sret.value, sret.error);
Which will show…
probe_extension[0x10]:
value=0x1, error=0
probe_extension[0x4442434E]:
value=0x0, error=0
Hence we learn that…
Base Extension (0x10) is supported
Debug Console Extension (0x4442 434E) is NOT supported
Thus we always Probe the Extensions before calling them!
OK so OpenSBI can do trivial things…
What about controlling the CPUs?
Now we experiment with the RISC-V CPU Cores (“Hart” / Hardware Thread) in our SBC.
We call Hart State Management (HSM) to query the Hart Status…
Extension ID: 0x48 534D “HSM”
Function ID: 2 (Get Hart Status)
Parameter 0: Hart ID (CPU Core ID)
(Not to be confused with Hardware Security Module)
Here’s how: jh7110_appinit.c
// For each Hart ID from 0 to 5...
for (uintptr_t hart = 0; hart < 6; hart++) {
// HART Get Status
// Call sbi_hart_get_status: EID 0x48534D "HSM", FID 2
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#93-function-hart-get-status-fid-2
struct sbiret sret = sbi_ecall(
SBI_EXT_HSM, // Extension ID: 0x48534D "HSM"
SBI_EXT_HSM_HART_GET_STATUS, // Function ID: 2
hart, // Parameter 0: Hart ID
0, 0, 0, 0, 0 // Other Parameters (unused)
);
_info("hart_get_status[%d]: value=0x%x, error=%d\n", hart, sret.value, sret.error);
}
Our SBC says (pic above)…
hart_get_status[0]: value=0x1, error=0
hart_get_status[1]: value=0x0, error=0
hart_get_status[2]: value=0x1, error=0
hart_get_status[3]: value=0x1, error=0
hart_get_status[4]: value=0x1, error=0
hart_get_status[5]: value=0x0, error=-3
When we decode the values, we learn that…
Hart 1 is Running
Other Harts are Stopped
Hart 5 doesn’t exist, because our SBC has only 5 CPU Cores (0 to 4)
Huh? Why is Hart 0 stopped while Hart 1 is running?
According to the SiFive U74 Manual (Page 96), there are 5 RISC-V Cores in JH7110 (pic below)…
Hart 0: S7 Monitor Core (RV64IMACB)
Harts 1 to 4: U74 Application Cores (RV64GCB)
OpenSBI and NuttX will boot on the First Application Core. That’s why Hart 1 is running. (And not Hart 0)
How do we start a Hart?
(With a Defibrillator heh heh)
Check out these SBI Functions…
In future we’ll call these SBI Functions to start NuttX on Multiple CPUs.
OpenSBI looks mighty powerful. Can it control our ENTIRE SBC?
Yep! OpenSBI supports System Reset for…
Shutdown: “physical power down of the entire system”
Cold Reboot: “physical power cycle of the entire system”
Warm Reboot: “power cycle of main processor and parts of the system but not the entire system”
Which we call like so: jh7110_appinit.c
// System Reset: Shutdown
// Call sbi_system_reset: EID 0x53525354 "SRST", FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#101-function-system-reset-fid-0
struct sbiret sret = sbi_ecall(
SBI_EXT_SRST, SBI_EXT_SRST_RESET, // System Reset
SBI_SRST_RESET_TYPE_SHUTDOWN, // Shutdown
SBI_SRST_RESET_REASON_NONE, 0, 0, 0, 0);
// System Reset: Cold Reboot
sret = sbi_ecall(
SBI_EXT_SRST, SBI_EXT_SRST_RESET, // System Reset
SBI_SRST_RESET_TYPE_COLD_REBOOT, // Cold Reboot
SBI_SRST_RESET_REASON_NONE, 0, 0, 0, 0);
// System Reset: Warm Reboot
sret = sbi_ecall(
SBI_EXT_SRST, SBI_EXT_SRST_RESET, // System Reset
SBI_SRST_RESET_TYPE_WARM_REBOOT, // Warm Reboot
SBI_SRST_RESET_REASON_NONE, 0, 0, 0, 0);
What happens when we run this?
Shutdown: Our SBC prints this and halts (without catching fire, pic below)…
i2c read: write daddr 36 to
cannot read pmic power register
Cold Reboot: Same behaviour as Shutdown. (Pic below)
(Not yet implemented on JH7110?)
Warm Reboot: Not supported on our Star64 SBC…
system_reset[warm_reboot]:
value=0x0
error=-2
Warm Reboot works OK on VisionFive2 SBC because it has a newer build of OpenSBI.
How do we know that VisionFive2 has a newer build of OpenSBI?
When Star64 boots, this is the first thing that we see…
U-Boot SPL 2021.10 (Jan 19 2023)
Which says that Star64 ships with a Secondary Program Loader + OpenSBI + U-Bootloader that was built on 19 Jan 2023.
On VisionFive2, we see a newer date: 21 Jun 2023…
U-Boot SPL 2021.10 (Jun 21 2023)
Thus VisionFive2 ships with a newer build of OpenSBI.
NuttX / Linux Kernel runs in RISC-V Supervisor Mode (not Machine Mode)…
How will it control the System Timer?
That’s why OpenSBI provides the Set Timer function: jh7110_appinit.c
// Set Timer
// Call sbi_set_timer: EID 0x54494D45 "TIME", FID 0
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#61-function-set-timer-fid-0
sret = sbi_ecall(
SBI_EXT_TIME, // Extension ID: 0x54494D45 "TIME"
SBI_EXT_TIME_SET_TIMER, // Function ID: 0
0, // TODO: Absolute Time for Timer Expiry
0, 0, 0, 0, 0);
It doesn’t seem to do anything…
set_timer:
value=0x0
error=0
But that’s because our SBC will trigger an interrupt when the System Timer expires.
Someday NuttX will call this function to set the System Timer.
Earlier we called OpenSBI to fetch the SBI Spec Version…
What else can we fetch from OpenSBI?
We can snoop a whole bunch of System Info like this: jh7110_appinit.c
// Get SBI Implementation ID: EID 0x10, FID 1
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#42-function-get-sbi-implementation-id-fid-1
struct sbiret sret = sbi_ecall(
SBI_EXT_BASE, SBI_EXT_BASE_GET_IMP_ID, 0, 0, 0, 0, 0, 0);
// Get SBI Implementation Version: EID 0x10, FID 2
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#43-function-get-sbi-implementation-version-fid-2
struct sbiret sret = sbi_ecall(
SBI_EXT_BASE, SBI_EXT_BASE_GET_IMP_VERSION, 0, 0, 0, 0, 0, 0);
// Get Machine Vendor ID: EID 0x10, FID 4
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#45-function-get-machine-vendor-id-fid-4
sret = sbi_ecall(
SBI_EXT_BASE, SBI_EXT_BASE_GET_MVENDORID, 0, 0, 0, 0, 0, 0);
// Get Machine Architecture ID: EID 0x10, FID 5
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#46-function-get-machine-architecture-id-fid-5
sret = sbi_ecall(
SBI_EXT_BASE, SBI_EXT_BASE_GET_MARCHID, 0, 0, 0, 0, 0, 0);
// Get Machine Implementation ID: EID 0x10, FID 6
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#47-function-get-machine-implementation-id-fid-6
sret = sbi_ecall(
SBI_EXT_BASE, SBI_EXT_BASE_GET_MIMPID, 0, 0, 0, 0, 0, 0);
// Omitted: Print `sret.value` and `sret.error`
Our SBC will print (pic above)…
// OpenSBI Implementation ID is 1
get_impl_id: 0x1
// OpenSBI Version is 1.2
get_impl_version: 0x10002
// RISC-V Vendor is SiFive
get_mvendorid: 0x489
// RISC-V Machine Architecture is SiFive U7 Series
get_marchid: 0x7
// RISC-V Machine Implementation is 0x4210427 (?)
get_mimpid: 0x4210427
The last 3 values are documented in the SiFive U74 Manual. (Pages 136 to 137)
Phew that’s plenty of OpenSBI Functions…
How will NuttX use them?
As we port Apache NuttX RTOS to Star64 JH7110 SBC, we shall call…
SBI Hart State Management to start NuttX on Multiple CPUs
(Including the RV64IMACB Monitor Core)
SBI Inter-Processor Interrupts to communicate across CPUs
SBI Timer to set the System Timer
SBI RFENCE to flush Device I/O and Memory Accesses
Performance Monitoring might be helpful for NuttX
Shutdown and Reboot because what goes up, must come down
And we’ll Probe the SBI Extensions before calling them.
We’re not calling the SBI Debug Console?
We’ve already implemented the NuttX UART Driver for JH7110. So we won’t call OpenSBI for Console Input / Output.
But when we port NuttX to a new SBC, we should consider SBI Debug Console for simple debug logging.
Can NuttX Apps call OpenSBI?
Nope, only the NuttX Kernel is allowed to call OpenSBI.
That’s because NuttX Apps run in RISC-V User Mode. When NuttX Apps execute the ecall Instruction, they will jump from User Mode to Supervisor Mode to execute NuttX Kernel Functions. (Not OpenSBI Functions)
Thus NuttX Apps are prevented from calling OpenSBI to meddle with CPUs, Timers and Interrupts. (Which should be meddled by the NuttX Kernel anyway)
I hope this article has been helpful for learning about OpenSBI and how it works with Apache NuttX RTOS (and Linux)…
We printed to the Serial Console
Set a System Timer
Queried the RISC-V CPUs
Fetched the System Information
And Shutdown / Rebooted our SBC (somewhat)
Stay tuned for more integration with NuttX and OpenSBI!
Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…