[onaffaa]

Problem Description

Currently the behavior of Zephyr is to compile its ELF as position dependent, which results in the generated ELF needing to be loaded at the physical address the firmware was compiled against.

By adding support to compile position independent flags using the -fPIC and -pie flags several benefits are introduced:

1. A PIE executable can be loaded from a different physical address since a fixed physical address is not assumed
2. Improved support for relocatable images by allowing self-relocation to take place, which is beneficial as it removes reliance on statically compiling an ELF for a different address or requiring the bootloader to maintain logic to perform the relocation of the image

Proposed Change (Summary)

The high-level scope of changes to enable this functionality are:

1. Add compilation / linking support by introducing a new feature flag, CONFIG_PIC_OPTIONS, that determines which compiler / linker flags are to be included.
2. Update the linker script for RISC-V targets to address newly generated sections specific to PIE execution
3. Perform self-relocation early in the boot-up process (i.e. just after reset)

Proposed Change (Detailed)

Adding PIE Compilation / Linking Support

A new feature flag CONFIG_PIC_OPTIONS would be introduced to support building PIE if enabled. If disabled (default), the current functionality that prevents using position independent compilation / linking remains.

• If disabled the property no_position_independent is used, which adds the compiler properties -fno-pic and -fno-pie and --no-pie as a linker property

With CONFIG_PIC_OPTIONS=y new a property position_independent_elf that has the following effects:

• Sets -fPIC as a property for compilation
• Sets -Wl, -pie -Wl, --no-dynamic-linker as a property for linking

Note: The --no-dynamic-linker linker option prevents generation of the PT_INTERP program header and other metadata required for dynamic loading. As a result, the relocatable PIE image remains a static, self-contained ELF binary that performs any required relocations internally during startup.

Modify Linker Scripts to Handle PIE-specific Sections

At link-time several sections are generated for PIE images:

• .dynsym: Dynamic symbol table containing symbols that may need to be referenced or resolved at load/runtime.
• .dynstr: String table containing the names associated with entries in .dynsym and other dynamic-linking metadata.
• .hash: Hash table used by the dynamic linker to efficiently locate symbols in .dynsym.
• .plt: Procedure Linkage Table containing stubs used to call dynamically linked functions.
• .got: Global Offset Table containing addresses used by position-independent code to access global data and external symbols.

Rather than resolve symbols dynamically at run-time early boot-up code will be added that resolves all symbols present in relocation sections. To identify this the location of the Global Offset Table (GOT) and relocation sections are determined by the symbols:

1. __got_start, __got_end for the GOT section range
2. __rela_dyn_start, __rela_dyn_end for the range that defines relocation sections

Performing Self Relocation

Relocation Process (Example)

With a PIE binary self relocation is done for global data that would otherwise reference a stale address if the ELF is executed from a different physical address than what was compiled against. PC-relative addressing is used to get the address of the GOT entry for some global symbol, which is then used to get the linking address of the global symbol. The image below shows a typical entry within the rela.dyn section.

The example below uses a GOT entry, but this is not the only r_offset type in rela.dyn; one other example would be const string pointers.

• The address representing the GOT entry (computed via auipc) is marked in red
• The address representing the link-time address of the global symbol is marked in yellow

Procedure

Adjusting Entries in GOT

To determine the offset where the ELF is loaded from relative to its physical address logic will be added within the __start routine within vector.S. The logic below introduces a label - reloc_pc - whose runtime and link-time address are used to determine the offset used to adjust the entries in the GOT.

reloc_pc:
auipc t0, 0 /* Get the runtime address of this instruction */
la t1, reloc_pc /* Get the linker time address of this instruction */
sub t2, t0, t1

Once this offset is computed iterate through each entry stored in __rel_dyn_start to __rel_dyn_end, where each entry is 12 bytes composed of 3 4-byte values: r_offset, r_info, and r_addend, that have an r_info value of type R_RISCV_RELATIVE.

• r_offset - the link-time address of the memory location that needs to be patched
• r_info - packs the symbol index and relocation type
• r_addend - the constant value used to compute the relocated address; adjusted by the load offset to get the actual runtime value that gets stored

Some rela.dyn entries aren't guaranteed to be 4-byte aligned (e.g. packed elements within a struct), which requires additional handling when writing the patched values (e.g. r_offset + t2) to perform half-word or single-byte stores to avoid a fault due to misalignment.

Interrupt Relocation

An additional update would be needed for gen_isr_defines.py, which parses and patches the .intList section to resolve addresses in _sw_isr_table. This is done in 2 steps:

1. Update function pointer addresses representing the ISR to call - func member of _isr_list
2. Update function parameter for non-direct IRQs - param member of _isr_list

Care must be taken when updating function parameters (point 2), as param values may represent a literal rather than an address. The example below illustrates this nuance:

• Entry 4 is a function pointer to isr_handler, which has a parameter 0x4 that is converted to an integer using POINTER_TO_INT
• Entry 11 is a function pointer to plic_irq_handler, whose parameter is a pointer to a device struct that needs to be relocated

The symbol table (.symtab) is used to determine what interrupt relocations should occur at runtime and populates a list of interrupts to relocate. This table is consumed by a newly added z_adjust_isr_tables function that is invoked in z_cstart.

• Written into build/zephyr/isr_tables.c at compile-time
• {11, 0} indicates that the parameter of ISR 11 was identified as needing to be relocated
• {-1, -1} denotes the end of the list of interrupts to relocate

The second index (e.g. 0 in {11, 0}) is used when CONFIG_SHARED_INTERRUPTS are enabled to determine which client index requires it's arg to be relocated. If CONFIG_SHARED_INTERRUPTS the value is a "don't care".

Dependencies

GCC does not support the necessary -pie linking flag, which requires that this feature is compiled with LLVM.
Changes made were verified downstream using:

• CPULLVM
• Embedded LD (eld)

Additionally, CONFIG_QEMU_DEVICE_LOADER must be enabled if testing on QEMU as the dynamic sections are placed prior to rom_start.

Concerns and Unresolved Questions

Interactions with userspace, as kernel objects are added to a lookup table at compile time. Follow up work would be to address this similar to how interrupts are relocated, but for now CONFIG_PIC_OPTIONS would depend on userspace being disabled.

CONFIG_GEN_SW_ISR_TABLE_SWITCH is not supported with this feature, as interrupt relocation requires patching the pointers to each ISR at runtime. This feature generates C-code with literal addresses that can't be patched, unlike CONFIG_GEN_SW_ISR_TABLE_ARRAY

Alternatives Considered

1. Mapping the physical address where the ELF resides to the desired virtual address. Infeasible because no MMU on hardware
2. Perform relocation via bootloader, which requires additional loader logic / is less secure

[onaffaa]

For whatever reason moving __rel_dyn_end inside the .rela.dyn section results in errors.

In our downstream it was left commented out:

[onaffaa]

Modification of the following from zephyr/kernel/config/soc/sifive/common/vector.S made with the intention that 4k alignment is not strictly required. If the value is offset by some value less than 4k the lui t1, 0xFFFFF instruction masks the offset entirely (e.g. if offsetting by 5k).

The code below relies on the address of __start being close to the runtime address where the calculation is done, whereas defining reloc_pc allows the calculation to be done anywhere in this routine.

    /* Calculate the delta between linker time address and run time address */
    /* Get the runtime address */
    auipc t0, 0
    lui t1, 0xFFFFF     /* load 0xFFFFF000 into t1 */
    and t0, t0, t1
    /* Get the linker time address */
    la t1, __start
    /* get the delta between t1 and t0 */
    sub t2, t0, t1

[onaffaa]

This goes in hand with relaxing the requirement that the offset must be a multiple of 4K. This leads to an infinite loop to diagnose the fault instead of silently masking the offset if it doesn't meet alignment criteria.

It may be useful to make a variant of this ( andi t5, t2, 0xFFF ) instead of using lui t1, 0xFFFFF above even if 4k alignment is required if we want to fault on a non-4k alignment instead of silently rounding it down.

[onaffaa]

This should be cleaned up to reference LLVM rather than CPULLVM.

[onaffaa]

Feedback appreciated on this idea for testing in upstream. It requires manually shifting the LMA so that the ELF is interpreted as relocated (in downstream we use T32 to load the ELF at an offset).