nagai/kernel
3
0
Fork 0
forked from mirrors/linux
Code Pull requests Activity

RISC-V Patches for the 6.16 Merge Window, Part 1

Browse source 
* Support for the FWFT SBI extension, which is part of SBI 3.0 and a
   dependency for many new SBI and ISA extensions.
 * Support for getrandom() in the VDSO.
 * Support for mseal.
 * Optimized routines for raid6 syndrome and recovery calculations.
 * kexec_file() supports loading Image-formatted kernel binaries.
 * Improvements to the instruction patching framework to allow for atomic
   instruction patching, along with rules as to how systems need to
   behave in order to function correctly.
 * Support for a handful of new ISA extensions: Svinval, Zicbop, Zabha,
   some SiFive vendor extensions.
 * Various fixes and cleanups, including: misaligned access handling, perf
   symbol mangling, module loading, PUD THPs, and improved uaccess
   routines.
 -----BEGIN PGP SIGNATURE-----
 
 iQJNBAABCAA3FiEEKzw3R0RoQ7JKlDp6LhMZ81+7GIkFAmhDLP8ZHHBhbG1lcmRh
 YmJlbHRAZ29vZ2xlLmNvbQAKCRAuExnzX7sYiZhFD/4+Zikkld812VjFb9dTF+Wj
 n/x9h86zDwAEFgf2BMIpUQhHru6vtdkO2l/Ky6mQblTPMWLafF4eK85yCsf84sQ0
 +RX4sOMLZ0+qvqxKX+aOFe9JXOWB0QIQuPvgBfDDOV4UTm60sglIxwqOpKcsBEHs
 2nplXXjiv0ckaMFLos8xlwu1uy4A/jMfT3Y9FDcABxYCqBoKOZ1frcL9ezJZbHbv
 BoOKLDH8ZypFxIG/eQ511lIXXtrnLas0l4jHWjrfsWu6pmXTgJasKtbGuH3LoLnM
 G/4qvHufR6lpVUOIL5L0V6PpsmYwDi/ciFIFlc8NH2oOZil3qiVaGSEbJIkWGFu9
 8lWTXQWnbinZbfg2oYbWp8GlwI70vKomtDyYNyB9q9Cq9jyiTChMklRNODr4764j
 ZiEnzc/l4KyvaxUg8RLKCT595lKECiUDnMytbIbunJu05HBqRCoGpBtMVzlQsyUd
 ybkRt3BA7eOR8/xFA7ZZQeJofmiu2yxkBs5ggMo8UnSragw27hmv/OA0mWMXEuaD
 aaWc4ZKpKqf7qLchLHOvEl5ORUhsisyIJgZwOqdme5rQoWorVtr51faA4AKwFAN4
 vcKgc5qJjK8vnpW+rl3LNJF9LtH+h4TgmUI853vUlukPoH2oqRkeKVGSkxG0iAze
 eQy2VjP1fJz6ciRtJZn9aw==
 =cZGy
 -----END PGP SIGNATURE-----

Merge tag 'riscv-for-linus-6.16-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux

Pull RISC-V updates from Palmer Dabbelt:

 - Support for the FWFT SBI extension, which is part of SBI 3.0 and a
   dependency for many new SBI and ISA extensions

 - Support for getrandom() in the VDSO

 - Support for mseal

 - Optimized routines for raid6 syndrome and recovery calculations

 - kexec_file() supports loading Image-formatted kernel binaries

 - Improvements to the instruction patching framework to allow for
   atomic instruction patching, along with rules as to how systems need
   to behave in order to function correctly

 - Support for a handful of new ISA extensions: Svinval, Zicbop, Zabha,
   some SiFive vendor extensions

 - Various fixes and cleanups, including: misaligned access handling,
   perf symbol mangling, module loading, PUD THPs, and improved uaccess
   routines

* tag 'riscv-for-linus-6.16-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (69 commits)
  riscv: uaccess: Only restore the CSR_STATUS SUM bit
  RISC-V: vDSO: Wire up getrandom() vDSO implementation
  riscv: enable mseal sysmap for RV64
  raid6: Add RISC-V SIMD syndrome and recovery calculations
  riscv: mm: Add support for Svinval extension
  RISC-V: Documentation: Add enough title underlines to CMODX
  riscv: Improve Kconfig help for RISCV_ISA_V_PREEMPTIVE
  MAINTAINERS: Update Atish's email address
  riscv: uaccess: do not do misaligned accesses in get/put_user()
  riscv: process: use unsigned int instead of unsigned long for put_user()
  riscv: make unsafe user copy routines use existing assembly routines
  riscv: hwprobe: export Zabha extension
  riscv: Make regs_irqs_disabled() more clear
  perf symbols: Ignore mapping symbols on riscv
  RISC-V: Kconfig: Fix help text of CMDLINE_EXTEND
  riscv: module: Optimize PLT/GOT entry counting
  riscv: Add support for PUD THP
  riscv: xchg: Prefetch the destination word for sc.w
  riscv: Add ARCH_HAS_PREFETCH[W] support with Zicbop
  riscv: Add support for Zicbop
  ...
This commit is contained in:
Linus Torvalds 2025-06-06 18:05:18 -07:00
commit 119b1e61a7
74 changed files with 3793 additions and 861 deletions
repo.diff.show_diff_stats Download patch file Download diff file Expand all files Collapse all files

3
.mailmap
View file

@ -107,7 +107,8 @@ Asahi Lina <lina+kernel@asahilina.net> <lina@asahilina.net>
Ashok Raj Nagarajan <quic_arnagara@quicinc.com> <arnagara@codeaurora.org>
Ashwin Chaugule <quic_ashwinc@quicinc.com> <ashwinc@codeaurora.org>
Asutosh Das <quic_asutoshd@quicinc.com> <asutoshd@codeaurora.org>
Atish Patra <atishp@atishpatra.org> <atish.patra@wdc.com>
Atish Patra <atish.patra@linux.dev> <atishp@atishpatra.org>
Atish Patra <atish.patra@linux.dev> <atish.patra@wdc.com>
Avaneesh Kumar Dwivedi <quic_akdwived@quicinc.com> <akdwived@codeaurora.org>
Axel Dyks <xl@xlsigned.net>
Axel Lin <axel.lin@gmail.com>

46
Documentation/arch/riscv/cmodx.rst
View file

@ -10,13 +10,45 @@ modified by the program itself. Instruction storage and the instruction cache
program must enforce its own synchronization with the unprivileged fence.i
instruction.
However, the default Linux ABI prohibits the use of fence.i in userspace
applications. At any point the scheduler may migrate a task onto a new hart. If
migration occurs after the userspace synchronized the icache and instruction
storage with fence.i, the icache on the new hart will no longer be clean. This
is due to the behavior of fence.i only affecting the hart that it is called on.
Thus, the hart that the task has been migrated to may not have synchronized
instruction storage and icache.
CMODX in the Kernel Space
-------------------------
Dynamic ftrace
---------------------
Essentially, dynamic ftrace directs the control flow by inserting a function
call at each patchable function entry, and patches it dynamically at runtime to
enable or disable the redirection. In the case of RISC-V, 2 instructions,
AUIPC + JALR, are required to compose a function call. However, it is impossible
to patch 2 instructions and expect that a concurrent read-side executes them
without a race condition. This series makes atmoic code patching possible in
RISC-V ftrace. Kernel preemption makes things even worse as it allows the old
state to persist across the patching process with stop_machine().
In order to get rid of stop_machine() and run dynamic ftrace with full kernel
preemption, we partially initialize each patchable function entry at boot-time,
setting the first instruction to AUIPC, and the second to NOP. Now, atmoic
patching is possible because the kernel only has to update one instruction.
According to Ziccif, as long as an instruction is naturally aligned, the ISA
guarantee an atomic update.
By fixing down the first instruction, AUIPC, the range of the ftrace trampoline
is limited to +-2K from the predetermined target, ftrace_caller, due to the lack
of immediate encoding space in RISC-V. To address the issue, we introduce
CALL_OPS, where an 8B naturally align metadata is added in front of each
pacthable function. The metadata is resolved at the first trampoline, then the
execution can be derect to another custom trampoline.
CMODX in the User Space
-----------------------
Though fence.i is an unprivileged instruction, the default Linux ABI prohibits
the use of fence.i in userspace applications. At any point the scheduler may
migrate a task onto a new hart. If migration occurs after the userspace
synchronized the icache and instruction storage with fence.i, the icache on the
new hart will no longer be clean. This is due to the behavior of fence.i only
affecting the hart that it is called on. Thus, the hart that the task has been
migrated to may not have synchronized instruction storage and icache.
There are two ways to solve this problem: use the riscv_flush_icache() syscall,
or use the ``PR_RISCV_SET_ICACHE_FLUSH_CTX`` prctl() and emit fence.i in

26
Documentation/arch/riscv/hwprobe.rst
View file

@ -271,6 +271,10 @@ The following keys are defined:
* :c:macro:`RISCV_HWPROBE_EXT_ZICBOM`: The Zicbom extension is supported, as
ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
* :c:macro:`RISCV_HWPROBE_EXT_ZABHA`: The Zabha extension is supported as
ratified in commit 49f49c842ff9 ("Update to Rafified state") of
riscv-zabha.
* :c:macro:`RISCV_HWPROBE_KEY_CPUPERF_0`: Deprecated. Returns similar values to
:c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF`, but the key was
mistakenly classified as a bitmask rather than a value.
@ -335,3 +339,25 @@ The following keys are defined:
* :c:macro:`RISCV_HWPROBE_KEY_ZICBOM_BLOCK_SIZE`: An unsigned int which
represents the size of the Zicbom block in bytes.
* :c:macro:`RISCV_HWPROBE_KEY_VENDOR_EXT_SIFIVE_0`: A bitmask containing the
sifive vendor extensions that are compatible with the
:c:macro:`RISCV_HWPROBE_BASE_BEHAVIOR_IMA`: base system behavior.
* SIFIVE
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVQMACCDOD`: The Xsfqmaccdod vendor
extension is supported in version 1.1 of SiFive Int8 Matrix Multiplication
Extensions Specification.
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVQMACCQOQ`: The Xsfqmaccqoq vendor
extension is supported in version 1.1 of SiFive Int8 Matrix Multiplication
Instruction Extensions Specification.
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVFNRCLIPXFQF`: The Xsfvfnrclipxfqf
vendor extension is supported in version 1.0 of SiFive FP32-to-int8 Ranged
Clip Instructions Extensions Specification.
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVFWMACCQQQ`: The Xsfvfwmaccqqq
vendor extension is supported in version 1.0 of Matrix Multiply Accumulate
Instruction Extensions Specification.

25
Documentation/devicetree/bindings/riscv/extensions.yaml
View file

@ -662,6 +662,31 @@ properties:
Registers in the AX45MP datasheet.
https://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf
# SiFive
- const: xsfvqmaccdod
description:
SiFive Int8 Matrix Multiplication Extensions Specification.
See more details in
https://www.sifive.com/document-file/sifive-int8-matrix-multiplication-extensions-specification
- const: xsfvqmaccqoq
description:
SiFive Int8 Matrix Multiplication Extensions Specification.
See more details in
https://www.sifive.com/document-file/sifive-int8-matrix-multiplication-extensions-specification
- const: xsfvfnrclipxfqf
description:
SiFive FP32-to-int8 Ranged Clip Instructions Extensions Specification.
See more details in
https://www.sifive.com/document-file/fp32-to-int8-ranged-clip-instructions
- const: xsfvfwmaccqqq
description:
SiFive Matrix Multiply Accumulate Instruction Extensions Specification.
See more details in
https://www.sifive.com/document-file/matrix-multiply-accumulate-instruction
# T-HEAD
- const: xtheadvector
description:

4
MAINTAINERS
View file

@ -13270,7 +13270,7 @@ F: arch/powerpc/kvm/
KERNEL VIRTUAL MACHINE FOR RISC-V (KVM/riscv)
M: Anup Patel <anup@brainfault.org>
R: Atish Patra <atishp@atishpatra.org>
R: Atish Patra <atish.patra@linux.dev>
L: kvm@vger.kernel.org
L: kvm-riscv@lists.infradead.org
L: linux-riscv@lists.infradead.org
@ -21332,7 +21332,7 @@ F: arch/riscv/boot/dts/sifive/
F: arch/riscv/boot/dts/starfive/
RISC-V PMU DRIVERS
M: Atish Patra <atishp@atishpatra.org>
M: Atish Patra <atish.patra@linux.dev>
R: Anup Patel <anup@brainfault.org>
L: linux-riscv@lists.infradead.org
S: Supported

38
arch/riscv/Kconfig
View file

@ -70,6 +70,7 @@ config RISCV
# LLD >= 14: https://github.com/llvm/llvm-project/issues/50505
select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS if 64BIT && MMU
select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU
select ARCH_SUPPORTS_PER_VMA_LOCK if MMU
select ARCH_SUPPORTS_RT
@ -99,6 +100,7 @@ config RISCV
select EDAC_SUPPORT
select FRAME_POINTER if PERF_EVENTS || (FUNCTION_TRACER && !DYNAMIC_FTRACE)
select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY if DYNAMIC_FTRACE
select FUNCTION_ALIGNMENT_8B if DYNAMIC_FTRACE_WITH_CALL_OPS
select GENERIC_ARCH_TOPOLOGY
select GENERIC_ATOMIC64 if !64BIT
select GENERIC_CLOCKEVENTS_BROADCAST if SMP
@ -143,6 +145,7 @@ config RISCV
select HAVE_ARCH_THREAD_STRUCT_WHITELIST
select HAVE_ARCH_TRACEHOOK
select HAVE_ARCH_TRANSPARENT_HUGEPAGE if 64BIT && MMU
select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if 64BIT && MMU
select HAVE_ARCH_USERFAULTFD_MINOR if 64BIT && USERFAULTFD
select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
select HAVE_ASM_MODVERSIONS
@ -150,13 +153,15 @@ config RISCV
select HAVE_DEBUG_KMEMLEAK
select HAVE_DMA_CONTIGUOUS if MMU
select HAVE_DYNAMIC_FTRACE if !XIP_KERNEL && MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE)
select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
select FUNCTION_ALIGNMENT_4B if HAVE_DYNAMIC_FTRACE && RISCV_ISA_C
select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS if HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS
select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG)
select HAVE_DYNAMIC_FTRACE_WITH_ARGS if HAVE_DYNAMIC_FTRACE
select HAVE_FTRACE_GRAPH_FUNC
select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL
select HAVE_FUNCTION_GRAPH_TRACER if HAVE_DYNAMIC_FTRACE_WITH_ARGS
select HAVE_FUNCTION_GRAPH_FREGS
select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION
select HAVE_FUNCTION_TRACER if !XIP_KERNEL
select HAVE_EBPF_JIT if MMU
select HAVE_GUP_FAST if MMU
select HAVE_FUNCTION_ARG_ACCESS_API
@ -218,6 +223,7 @@ config RISCV
select THREAD_INFO_IN_TASK
select TRACE_IRQFLAGS_SUPPORT
select UACCESS_MEMCPY if !MMU
select VDSO_GETRANDOM if HAVE_GENERIC_VDSO
select USER_STACKTRACE_SUPPORT
select ZONE_DMA32 if 64BIT
@ -236,6 +242,7 @@ config CLANG_SUPPORTS_DYNAMIC_FTRACE
config GCC_SUPPORTS_DYNAMIC_FTRACE
def_bool CC_IS_GCC
depends on $(cc-option,-fpatchable-function-entry=8)
depends on CC_HAS_MIN_FUNCTION_ALIGNMENT || !RISCV_ISA_C
config HAVE_SHADOW_CALL_STACK
def_bool $(cc-option,-fsanitize=shadow-call-stack)
@ -664,12 +671,12 @@ config RISCV_ISA_V_PREEMPTIVE
default y
help
Usually, in-kernel SIMD routines are run with preemption disabled.
Functions which envoke long running SIMD thus must yield core's
Functions which invoke long running SIMD thus must yield the core's
vector unit to prevent blocking other tasks for too long.
This config allows kernel to run SIMD without explicitly disable
preemption. Enabling this config will result in higher memory
consumption due to the allocation of per-task's kernel Vector context.
This config allows the kernel to run SIMD without explicitly disabling
preemption. Enabling this config will result in higher memory consumption
due to the allocation of per-task's kernel Vector context.
config RISCV_ISA_ZAWRS
bool "Zawrs extension support for more efficient busy waiting"
@ -842,6 +849,21 @@ config RISCV_ISA_ZICBOZ
If you don't know what to do here, say Y.
config RISCV_ISA_ZICBOP
bool "Zicbop extension support for cache block prefetch"
depends on MMU
depends on RISCV_ALTERNATIVE
default y
help
Adds support to dynamically detect the presence of the ZICBOP
extension (Cache Block Prefetch Operations) and enable its
usage.
The Zicbop extension can be used to prefetch cache blocks for
read/write fetch.
If you don't know what to do here, say Y.
config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
def_bool y
# https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
@ -1171,8 +1193,8 @@ config CMDLINE_FALLBACK
config CMDLINE_EXTEND
bool "Extend bootloader kernel arguments"
help
The command-line arguments provided during boot will be
appended to the built-in command line. This is useful in
The built-in command line will be appended to the command-
line arguments provided during boot. This is useful in
cases where the provided arguments are insufficient and
you don't want to or cannot modify them.

13
arch/riscv/Kconfig.vendor
View file

@ -16,6 +16,19 @@ config RISCV_ISA_VENDOR_EXT_ANDES
If you don't know what to do here, say Y.
endmenu
menu "SiFive"
config RISCV_ISA_VENDOR_EXT_SIFIVE
bool "SiFive vendor extension support"
select RISCV_ISA_VENDOR_EXT
default y
help
Say N here if you want to disable all SiFive vendor extension
support. This will cause any SiFive vendor extensions that are
requested by hardware probing to be ignored.
If you don't know what to do here, say Y.
endmenu
menu "T-Head"
config RISCV_ISA_VENDOR_EXT_THEAD
bool "T-Head vendor extension support"

4
arch/riscv/Makefile
View file

@ -15,9 +15,9 @@ ifeq ($(CONFIG_DYNAMIC_FTRACE),y)
LDFLAGS_vmlinux += --no-relax
KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY
ifeq ($(CONFIG_RISCV_ISA_C),y)
CC_FLAGS_FTRACE := -fpatchable-function-entry=4
CC_FLAGS_FTRACE := -fpatchable-function-entry=8,4
else
CC_FLAGS_FTRACE := -fpatchable-function-entry=2
CC_FLAGS_FTRACE := -fpatchable-function-entry=4,2
endif
endif

24
arch/riscv/configs/defconfig
View file

@ -18,12 +18,9 @@ CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
# CONFIG_SYSFS_SYSCALL is not set
CONFIG_PROFILING=y
CONFIG_ARCH_MICROCHIP=y
CONFIG_ARCH_SIFIVE=y
@ -182,6 +179,7 @@ CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_REGULATOR_AXP20X=y
CONFIG_REGULATOR_GPIO=y
CONFIG_MEDIA_SUPPORT=m
CONFIG_MEDIA_PLATFORM_SUPPORT=y
CONFIG_VIDEO_CADENCE_CSI2RX=m
CONFIG_DRM=m
CONFIG_DRM_RADEON=m
@ -297,25 +295,7 @@ CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_DEV_VIRTIO=y
CONFIG_PRINTK_TIME=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_PGFLAGS=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_WQ_WATCHDOG=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_RWSEMS=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_EQS_DEBUG=y
# CONFIG_FTRACE is not set
# CONFIG_RUNTIME_TESTING_MENU is not set
CONFIG_MEMTEST=y

2
arch/riscv/include/asm/asm-prototypes.h
View file

@ -12,7 +12,7 @@ long long __ashlti3(long long a, int b);
#ifdef CONFIG_RISCV_ISA_V
#ifdef CONFIG_MMU
asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n);
asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n, bool enable_sum);
#endif /* CONFIG_MMU */
void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1,

5
arch/riscv/include/asm/barrier.h
View file

@ -14,11 +14,6 @@
#include <asm/cmpxchg.h>
#include <asm/fence.h>
#define nop() __asm__ __volatile__ ("nop")
#define __nops(n) ".rept " #n "\nnop\n.endr\n"
#define nops(n) __asm__ __volatile__ (__nops(n))
/* These barriers need to enforce ordering on both devices or memory. */
#define __mb() RISCV_FENCE(iorw, iorw)
#define __rmb() RISCV_FENCE(ir, ir)

1
arch/riscv/include/asm/cacheflush.h
View file

@ -85,6 +85,7 @@ static inline void flush_icache_range(unsigned long start, unsigned long end)
extern unsigned int riscv_cbom_block_size;
extern unsigned int riscv_cboz_block_size;
extern unsigned int riscv_cbop_block_size;
void riscv_init_cbo_blocksizes(void);
#ifdef CONFIG_RISCV_DMA_NONCOHERENT

4
arch/riscv/include/asm/cmpxchg.h
View file

@ -13,6 +13,7 @@
#include <asm/hwcap.h>
#include <asm/insn-def.h>
#include <asm/cpufeature-macros.h>
#include <asm/processor.h>
#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \
swap_append, r, p, n) \
@ -37,6 +38,7 @@
\
__asm__ __volatile__ ( \
prepend \
PREFETCHW_ASM(%5) \
"0: lr.w %0, %2\n" \
" and %1, %0, %z4\n" \
" or %1, %1, %z3\n" \
@ -44,7 +46,7 @@
" bnez %1, 0b\n" \
sc_append \
: "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
: "rJ" (__newx), "rJ" (~__mask) \
: "rJ" (__newx), "rJ" (~__mask), "rJ" (__ptr32b) \
: "memory"); \
\
r = (__typeof__(*(p)))((__retx & __mask) >> __s); \

14
arch/riscv/include/asm/cpufeature.h
View file

@ -67,11 +67,11 @@ void __init riscv_user_isa_enable(void);
_RISCV_ISA_EXT_DATA(_name, _id, _sub_exts, ARRAY_SIZE(_sub_exts), _validate)
bool __init check_unaligned_access_emulated_all_cpus(void);
void unaligned_access_init(void);
int cpu_online_unaligned_access_init(unsigned int cpu);
#if defined(CONFIG_RISCV_SCALAR_MISALIGNED)
void check_unaligned_access_emulated(struct work_struct *work __always_unused);
void unaligned_emulation_finish(void);
bool unaligned_ctl_available(void);
DECLARE_PER_CPU(long, misaligned_access_speed);
#else
static inline bool unaligned_ctl_available(void)
{
@ -79,6 +79,16 @@ static inline bool unaligned_ctl_available(void)
}
#endif
#if defined(CONFIG_RISCV_MISALIGNED)
DECLARE_PER_CPU(long, misaligned_access_speed);
bool misaligned_traps_can_delegate(void);
#else
static inline bool misaligned_traps_can_delegate(void)
{
return false;
}
#endif
bool __init check_vector_unaligned_access_emulated_all_cpus(void);
#if defined(CONFIG_RISCV_VECTOR_MISALIGNED)
void check_vector_unaligned_access_emulated(struct work_struct *work __always_unused);

62
arch/riscv/include/asm/ftrace.h
View file

@ -20,10 +20,9 @@ extern void *return_address(unsigned int level);
#define ftrace_return_address(n) return_address(n)
void _mcount(void);
static inline unsigned long ftrace_call_adjust(unsigned long addr)
{
return addr;
}
unsigned long ftrace_call_adjust(unsigned long addr);
unsigned long arch_ftrace_get_symaddr(unsigned long fentry_ip);
#define ftrace_get_symaddr(fentry_ip) arch_ftrace_get_symaddr(fentry_ip)
/*
* Let's do like x86/arm64 and ignore the compat syscalls.
@ -57,12 +56,21 @@ struct dyn_arch_ftrace {
* 2) jalr: setting low-12 offset to ra, jump to ra, and set ra to
* return address (original pc + 4)
*
* The first 2 instructions for each tracable function is compiled to 2 nop
* instructions. Then, the kernel initializes the first instruction to auipc at
* boot time (<ftrace disable>). The second instruction is patched to jalr to
* start the trace.
*
*<Image>:
* 0: nop
* 4: nop
*
*<ftrace enable>:
* 0: auipc t0/ra, 0x?
* 4: jalr t0/ra, ?(t0/ra)
* 0: auipc t0, 0x?
* 4: jalr t0, ?(t0)
*
*<ftrace disable>:
* 0: nop
* 0: auipc t0, 0x?
* 4: nop
*
* Dynamic ftrace generates probes to call sites, so we must deal with
@ -75,10 +83,9 @@ struct dyn_arch_ftrace {
#define AUIPC_OFFSET_MASK (0xfffff000)
#define AUIPC_PAD (0x00001000)
#define JALR_SHIFT 20
#define JALR_RA (0x000080e7)
#define AUIPC_RA (0x00000097)
#define JALR_T0 (0x000282e7)
#define AUIPC_T0 (0x00000297)
#define JALR_RANGE (JALR_SIGN_MASK - 1)
#define to_jalr_t0(offset) \
(((offset & JALR_OFFSET_MASK) << JALR_SHIFT) | JALR_T0)
@ -96,26 +103,14 @@ do { \
call[1] = to_jalr_t0(offset); \
} while (0)
#define to_jalr_ra(offset) \
(((offset & JALR_OFFSET_MASK) << JALR_SHIFT) | JALR_RA)
#define to_auipc_ra(offset) \
((offset & JALR_SIGN_MASK) ? \
(((offset & AUIPC_OFFSET_MASK) + AUIPC_PAD) | AUIPC_RA) : \
((offset & AUIPC_OFFSET_MASK) | AUIPC_RA))
#define make_call_ra(caller, callee, call) \
do { \
unsigned int offset = \
(unsigned long) (callee) - (unsigned long) (caller); \
call[0] = to_auipc_ra(offset); \
call[1] = to_jalr_ra(offset); \
} while (0)
/*
* Let auipc+jalr be the basic *mcount unit*, so we make it 8 bytes here.
* Only the jalr insn in the auipc+jalr is patched, so we make it 4
* bytes here.
*/
#define MCOUNT_INSN_SIZE 8
#define MCOUNT_INSN_SIZE 4
#define MCOUNT_AUIPC_SIZE 4
#define MCOUNT_JALR_SIZE 4
#define MCOUNT_NOP4_SIZE 4
#ifndef __ASSEMBLY__
struct dyn_ftrace;
@ -135,6 +130,9 @@ struct __arch_ftrace_regs {
unsigned long sp;
unsigned long s0;
unsigned long t1;
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
unsigned long direct_tramp;
#endif
union {
unsigned long args[8];
struct {
@ -146,6 +144,13 @@ struct __arch_ftrace_regs {
unsigned long a5;
unsigned long a6;
unsigned long a7;
#ifdef CONFIG_CC_IS_CLANG
unsigned long t2;
unsigned long t3;
unsigned long t4;
unsigned long t5;
unsigned long t6;
#endif
};
};
};
@ -221,10 +226,13 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
struct ftrace_ops *op, struct ftrace_regs *fregs);
#define ftrace_graph_func ftrace_graph_func
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, unsigned long addr)
{
arch_ftrace_regs(fregs)->t1 = addr;
}
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
#endif /* __ASSEMBLY__ */

1
arch/riscv/include/asm/hwcap.h
View file

@ -105,6 +105,7 @@
#define RISCV_ISA_EXT_ZVFBFWMA 96
#define RISCV_ISA_EXT_ZAAMO 97
#define RISCV_ISA_EXT_ZALRSC 98
#define RISCV_ISA_EXT_ZICBOP 99
#define RISCV_ISA_EXT_XLINUXENVCFG 127

3
arch/riscv/include/asm/hwprobe.h
View file

@ -8,7 +8,7 @@
#include <uapi/asm/hwprobe.h>
#define RISCV_HWPROBE_MAX_KEY 12
#define RISCV_HWPROBE_MAX_KEY 13
static inline bool riscv_hwprobe_key_is_valid(__s64 key)
{
@ -22,6 +22,7 @@ static inline bool hwprobe_key_is_bitmask(__s64 key)
case RISCV_HWPROBE_KEY_IMA_EXT_0:
case RISCV_HWPROBE_KEY_CPUPERF_0:
case RISCV_HWPROBE_KEY_VENDOR_EXT_THEAD_0:
case RISCV_HWPROBE_KEY_VENDOR_EXT_SIFIVE_0:
return true;
}

2
arch/riscv/include/asm/image.h
View file

@ -30,6 +30,8 @@
RISCV_HEADER_VERSION_MINOR)
#ifndef __ASSEMBLY__
#define riscv_image_flag_field(flags, field)\
(((flags) >> field##_SHIFT) & field##_MASK)
/**
* struct riscv_image_header - riscv kernel image header
* @code0: Executable code

66
arch/riscv/include/asm/insn-def.h
View file

@ -18,6 +18,13 @@
#define INSN_I_RD_SHIFT 7
#define INSN_I_OPCODE_SHIFT 0
#define INSN_S_SIMM7_SHIFT 25
#define INSN_S_RS2_SHIFT 20
#define INSN_S_RS1_SHIFT 15
#define INSN_S_FUNC3_SHIFT 12
#define INSN_S_SIMM5_SHIFT 7
#define INSN_S_OPCODE_SHIFT 0
#ifdef __ASSEMBLY__
#ifdef CONFIG_AS_HAS_INSN
@ -30,6 +37,10 @@
.insn i \opcode, \func3, \rd, \rs1, \simm12
.endm
.macro insn_s, opcode, func3, rs2, simm12, rs1
.insn s \opcode, \func3, \rs2, \simm12(\rs1)
.endm
#else
#include <asm/gpr-num.h>
@ -51,10 +62,20 @@
(\simm12 << INSN_I_SIMM12_SHIFT))
.endm
.macro insn_s, opcode, func3, rs2, simm12, rs1
.4byte ((\opcode << INSN_S_OPCODE_SHIFT) | \
(\func3 << INSN_S_FUNC3_SHIFT) | \
(.L__gpr_num_\rs2 << INSN_S_RS2_SHIFT) | \
(.L__gpr_num_\rs1 << INSN_S_RS1_SHIFT) | \
((\simm12 & 0x1f) << INSN_S_SIMM5_SHIFT) | \
(((\simm12 >> 5) & 0x7f) << INSN_S_SIMM7_SHIFT))
.endm
#endif
#define __INSN_R(...) insn_r __VA_ARGS__
#define __INSN_I(...) insn_i __VA_ARGS__
#define __INSN_S(...) insn_s __VA_ARGS__
#else /* ! __ASSEMBLY__ */
@ -66,6 +87,9 @@
#define __INSN_I(opcode, func3, rd, rs1, simm12) \
".insn i " opcode ", " func3 ", " rd ", " rs1 ", " simm12 "\n"
#define __INSN_S(opcode, func3, rs2, simm12, rs1) \
".insn s " opcode ", " func3 ", " rs2 ", " simm12 "(" rs1 ")\n"
#else
#include <linux/stringify.h>
@ -92,12 +116,26 @@
" (\\simm12 << " __stringify(INSN_I_SIMM12_SHIFT) "))\n" \
" .endm\n"
#define DEFINE_INSN_S \
__DEFINE_ASM_GPR_NUMS \
" .macro insn_s, opcode, func3, rs2, simm12, rs1\n" \
" .4byte ((\\opcode << " __stringify(INSN_S_OPCODE_SHIFT) ") |" \
" (\\func3 << " __stringify(INSN_S_FUNC3_SHIFT) ") |" \
" (.L__gpr_num_\\rs2 << " __stringify(INSN_S_RS2_SHIFT) ") |" \
" (.L__gpr_num_\\rs1 << " __stringify(INSN_S_RS1_SHIFT) ") |" \
" ((\\simm12 & 0x1f) << " __stringify(INSN_S_SIMM5_SHIFT) ") |" \
" (((\\simm12 >> 5) & 0x7f) << " __stringify(INSN_S_SIMM7_SHIFT) "))\n" \
" .endm\n"
#define UNDEFINE_INSN_R \
" .purgem insn_r\n"
#define UNDEFINE_INSN_I \
" .purgem insn_i\n"
#define UNDEFINE_INSN_S \
" .purgem insn_s\n"
#define __INSN_R(opcode, func3, func7, rd, rs1, rs2) \
DEFINE_INSN_R \
"insn_r " opcode ", " func3 ", " func7 ", " rd ", " rs1 ", " rs2 "\n" \
@ -108,6 +146,11 @@
"insn_i " opcode ", " func3 ", " rd ", " rs1 ", " simm12 "\n" \
UNDEFINE_INSN_I
#define __INSN_S(opcode, func3, rs2, simm12, rs1) \
DEFINE_INSN_S \
"insn_s " opcode ", " func3 ", " rs2 ", " simm12 ", " rs1 "\n" \
UNDEFINE_INSN_S
#endif
#endif /* ! __ASSEMBLY__ */
@ -120,6 +163,10 @@
__INSN_I(RV_##opcode, RV_##func3, RV_##rd, \
RV_##rs1, RV_##simm12)
#define INSN_S(opcode, func3, rs2, simm12, rs1) \
__INSN_S(RV_##opcode, RV_##func3, RV_##rs2, \
RV_##simm12, RV_##rs1)
#define RV_OPCODE(v) __ASM_STR(v)
#define RV_FUNC3(v) __ASM_STR(v)
#define RV_FUNC7(v) __ASM_STR(v)
@ -133,6 +180,7 @@
#define RV___RS2(v) __RV_REG(v)
#define RV_OPCODE_MISC_MEM RV_OPCODE(15)
#define RV_OPCODE_OP_IMM RV_OPCODE(19)
#define RV_OPCODE_SYSTEM RV_OPCODE(115)
#define HFENCE_VVMA(vaddr, asid) \
@ -196,6 +244,18 @@
INSN_I(OPCODE_MISC_MEM, FUNC3(2), __RD(0), \
RS1(base), SIMM12(4))
#define PREFETCH_I(base, offset) \
INSN_S(OPCODE_OP_IMM, FUNC3(6), __RS2(0), \
SIMM12((offset) & 0xfe0), RS1(base))
#define PREFETCH_R(base, offset) \
INSN_S(OPCODE_OP_IMM, FUNC3(6), __RS2(1), \
SIMM12((offset) & 0xfe0), RS1(base))
#define PREFETCH_W(base, offset) \
INSN_S(OPCODE_OP_IMM, FUNC3(6), __RS2(3), \
SIMM12((offset) & 0xfe0), RS1(base))
#define RISCV_PAUSE ".4byte 0x100000f"
#define ZAWRS_WRS_NTO ".4byte 0x00d00073"
#define ZAWRS_WRS_STO ".4byte 0x01d00073"
@ -203,4 +263,10 @@
#define RISCV_INSN_NOP4 _AC(0x00000013, U)
#ifndef __ASSEMBLY__
#define nop() __asm__ __volatile__ ("nop")
#define __nops(n) ".rept " #n "\nnop\n.endr\n"
#define nops(n) __asm__ __volatile__ (__nops(n))
#endif
#endif /* __ASM_INSN_DEF_H */

6
arch/riscv/include/asm/kexec.h
View file

@ -56,6 +56,7 @@ extern riscv_kexec_method riscv_kexec_norelocate;
#ifdef CONFIG_KEXEC_FILE
extern const struct kexec_file_ops elf_kexec_ops;
extern const struct kexec_file_ops image_kexec_ops;
struct purgatory_info;
int arch_kexec_apply_relocations_add(struct purgatory_info *pi,
@ -67,6 +68,11 @@ int arch_kexec_apply_relocations_add(struct purgatory_info *pi,
struct kimage;
int arch_kimage_file_post_load_cleanup(struct kimage *image);
#define arch_kimage_file_post_load_cleanup arch_kimage_file_post_load_cleanup
int load_extra_segments(struct kimage *image, unsigned long kernel_start,
unsigned long kernel_len, char *initrd,
unsigned long initrd_len, char *cmdline,
unsigned long cmdline_len);
#endif
#endif

5
arch/riscv/include/asm/pgtable-64.h
View file

@ -184,7 +184,7 @@ static inline int pud_none(pud_t pud)
static inline int pud_bad(pud_t pud)
{
return !pud_present(pud);
return !pud_present(pud) || (pud_val(pud) & _PAGE_LEAF);
}
#define pud_leaf pud_leaf
@ -399,6 +399,7 @@ p4d_t *p4d_offset(pgd_t *pgd, unsigned long address);
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
static inline int pte_devmap(pte_t pte);
static inline pte_t pmd_pte(pmd_t pmd);
static inline pte_t pud_pte(pud_t pud);
static inline int pmd_devmap(pmd_t pmd)
{
@ -407,7 +408,7 @@ static inline int pmd_devmap(pmd_t pmd)
static inline int pud_devmap(pud_t pud)
{
return 0;
return pte_devmap(pud_pte(pud));
}
static inline int pgd_devmap(pgd_t pgd)

97
arch/riscv/include/asm/pgtable.h
View file

@ -900,6 +900,103 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
#define pmdp_collapse_flush pmdp_collapse_flush
extern pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmdp);
static inline pud_t pud_wrprotect(pud_t pud)
{
return pte_pud(pte_wrprotect(pud_pte(pud)));
}
static inline int pud_trans_huge(pud_t pud)
{
return pud_leaf(pud);
}
static inline int pud_dirty(pud_t pud)
{
return pte_dirty(pud_pte(pud));
}
static inline pud_t pud_mkyoung(pud_t pud)
{
return pte_pud(pte_mkyoung(pud_pte(pud)));
}
static inline pud_t pud_mkold(pud_t pud)
{
return pte_pud(pte_mkold(pud_pte(pud)));
}
static inline pud_t pud_mkdirty(pud_t pud)
{
return pte_pud(pte_mkdirty(pud_pte(pud)));
}
static inline pud_t pud_mkclean(pud_t pud)
{
return pte_pud(pte_mkclean(pud_pte(pud)));
}
static inline pud_t pud_mkwrite(pud_t pud)
{
return pte_pud(pte_mkwrite_novma(pud_pte(pud)));
}
static inline pud_t pud_mkhuge(pud_t pud)
{
return pud;
}
static inline pud_t pud_mkdevmap(pud_t pud)
{
return pte_pud(pte_mkdevmap(pud_pte(pud)));
}
static inline int pudp_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pud_t *pudp,
pud_t entry, int dirty)
{
return ptep_set_access_flags(vma, address, (pte_t *)pudp, pud_pte(entry), dirty);
}
static inline int pudp_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address, pud_t *pudp)
{
return ptep_test_and_clear_young(vma, address, (pte_t *)pudp);
}
static inline int pud_young(pud_t pud)
{
return pte_young(pud_pte(pud));
}
static inline void update_mmu_cache_pud(struct vm_area_struct *vma,
unsigned long address, pud_t *pudp)
{
pte_t *ptep = (pte_t *)pudp;
update_mmu_cache(vma, address, ptep);
}
static inline pud_t pudp_establish(struct vm_area_struct *vma,
unsigned long address, pud_t *pudp, pud_t pud)
{
page_table_check_pud_set(vma->vm_mm, pudp, pud);
return __pud(atomic_long_xchg((atomic_long_t *)pudp, pud_val(pud)));
}
static inline pud_t pud_mkinvalid(pud_t pud)
{
return __pud(pud_val(pud) & ~(_PAGE_PRESENT | _PAGE_PROT_NONE));
}
extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address,
pud_t *pudp);
static inline pud_t pud_modify(pud_t pud, pgprot_t newprot)
{
return pte_pud(pte_modify(pud_pte(pud), newprot));
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
/*

31
arch/riscv/include/asm/processor.h
View file

@ -13,6 +13,9 @@
#include <vdso/processor.h>
#include <asm/ptrace.h>
#include <asm/insn-def.h>
#include <asm/alternative-macros.h>
#include <asm/hwcap.h>
#define arch_get_mmap_end(addr, len, flags) \
({ \
@ -52,7 +55,6 @@
#endif
#ifndef __ASSEMBLY__
#include <linux/cpumask.h>
struct task_struct;
struct pt_regs;
@ -79,6 +81,10 @@ struct pt_regs;
* Thus, the task does not own preempt_v. Any use of Vector will have to
* save preempt_v, if dirty, and fallback to non-preemptible kernel-mode
* Vector.
* - bit 29: The thread voluntarily calls schedule() while holding an active
* preempt_v. All preempt_v context should be dropped in such case because
* V-regs are caller-saved. Only sstatus.VS=ON is persisted across a
* schedule() call.
* - bit 30: The in-kernel preempt_v context is saved, and requries to be
* restored when returning to the context that owns the preempt_v.
* - bit 31: The in-kernel preempt_v context is dirty, as signaled by the
@ -93,6 +99,7 @@ struct pt_regs;
#define RISCV_PREEMPT_V 0x00000100
#define RISCV_PREEMPT_V_DIRTY 0x80000000
#define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000
#define RISCV_PREEMPT_V_IN_SCHEDULE 0x20000000
/* CPU-specific state of a task */
struct thread_struct {
@ -103,6 +110,7 @@ struct thread_struct {
struct __riscv_d_ext_state fstate;
unsigned long bad_cause;
unsigned long envcfg;
unsigned long sum;
u32 riscv_v_flags;
u32 vstate_ctrl;
struct __riscv_v_ext_state vstate;
@ -136,6 +144,27 @@ static inline void arch_thread_struct_whitelist(unsigned long *offset,
#define KSTK_EIP(tsk) (task_pt_regs(tsk)->epc)
#define KSTK_ESP(tsk) (task_pt_regs(tsk)->sp)
#define PREFETCH_ASM(x) \
ALTERNATIVE(__nops(1), PREFETCH_R(x, 0), 0, \
RISCV_ISA_EXT_ZICBOP, CONFIG_RISCV_ISA_ZICBOP)
#define PREFETCHW_ASM(x) \
ALTERNATIVE(__nops(1), PREFETCH_W(x, 0), 0, \
RISCV_ISA_EXT_ZICBOP, CONFIG_RISCV_ISA_ZICBOP)
#ifdef CONFIG_RISCV_ISA_ZICBOP
#define ARCH_HAS_PREFETCH
static inline void prefetch(const void *x)
{
__asm__ __volatile__(PREFETCH_ASM(%0) : : "r" (x) : "memory");
}
#define ARCH_HAS_PREFETCHW
static inline void prefetchw(const void *x)
{
__asm__ __volatile__(PREFETCHW_ASM(%0) : : "r" (x) : "memory");
}
#endif /* CONFIG_RISCV_ISA_ZICBOP */
/* Do necessary setup to start up a newly executed thread. */
extern void start_thread(struct pt_regs *regs,

2
arch/riscv/include/asm/ptrace.h
View file

@ -175,7 +175,7 @@ static inline unsigned long regs_get_kernel_argument(struct pt_regs *regs,
return 0;
}
static inline int regs_irqs_disabled(struct pt_regs *regs)
static __always_inline bool regs_irqs_disabled(struct pt_regs *regs)
{
return !(regs->status & SR_PIE);
}

60
arch/riscv/include/asm/sbi.h
View file

@ -35,6 +35,7 @@ enum sbi_ext_id {
SBI_EXT_DBCN = 0x4442434E,
SBI_EXT_STA = 0x535441,
SBI_EXT_NACL = 0x4E41434C,
SBI_EXT_FWFT = 0x46574654,
/* Experimentals extensions must lie within this range */
SBI_EXT_EXPERIMENTAL_START = 0x08000000,
@ -402,6 +403,33 @@ enum sbi_ext_nacl_feature {
#define SBI_NACL_SHMEM_SRET_X(__i) ((__riscv_xlen / 8) * (__i))
#define SBI_NACL_SHMEM_SRET_X_LAST 31
/* SBI function IDs for FW feature extension */
#define SBI_EXT_FWFT_SET 0x0
#define SBI_EXT_FWFT_GET 0x1
enum sbi_fwft_feature_t {
SBI_FWFT_MISALIGNED_EXC_DELEG = 0x0,
SBI_FWFT_LANDING_PAD = 0x1,
SBI_FWFT_SHADOW_STACK = 0x2,
SBI_FWFT_DOUBLE_TRAP = 0x3,
SBI_FWFT_PTE_AD_HW_UPDATING = 0x4,
SBI_FWFT_POINTER_MASKING_PMLEN = 0x5,
SBI_FWFT_LOCAL_RESERVED_START = 0x6,
SBI_FWFT_LOCAL_RESERVED_END = 0x3fffffff,
SBI_FWFT_LOCAL_PLATFORM_START = 0x40000000,
SBI_FWFT_LOCAL_PLATFORM_END = 0x7fffffff,
SBI_FWFT_GLOBAL_RESERVED_START = 0x80000000,
SBI_FWFT_GLOBAL_RESERVED_END = 0xbfffffff,
SBI_FWFT_GLOBAL_PLATFORM_START = 0xc0000000,
SBI_FWFT_GLOBAL_PLATFORM_END = 0xffffffff,
};
#define SBI_FWFT_PLATFORM_FEATURE_BIT BIT(30)
#define SBI_FWFT_GLOBAL_FEATURE_BIT BIT(31)
#define SBI_FWFT_SET_FLAG_LOCK BIT(0)
/* SBI spec version fields */
#define SBI_SPEC_VERSION_DEFAULT 0x1
#define SBI_SPEC_VERSION_MAJOR_SHIFT 24
@ -419,6 +447,11 @@ enum sbi_ext_nacl_feature {
#define SBI_ERR_ALREADY_STARTED -7
#define SBI_ERR_ALREADY_STOPPED -8
#define SBI_ERR_NO_SHMEM -9
#define SBI_ERR_INVALID_STATE -10
#define SBI_ERR_BAD_RANGE -11
#define SBI_ERR_TIMEOUT -12
#define SBI_ERR_IO -13
#define SBI_ERR_DENIED_LOCKED -14
extern unsigned long sbi_spec_version;
struct sbiret {
@ -470,6 +503,23 @@ int sbi_remote_hfence_vvma_asid(const struct cpumask *cpu_mask,
unsigned long asid);
long sbi_probe_extension(int ext);
int sbi_fwft_set(u32 feature, unsigned long value, unsigned long flags);
int sbi_fwft_set_cpumask(const cpumask_t *mask, u32 feature,
unsigned long value, unsigned long flags);
/**
* sbi_fwft_set_online_cpus() - Set a feature on all online cpus
* @feature: The feature to be set
* @value: The feature value to be set
* @flags: FWFT feature set flags
*
* Return: 0 on success, appropriate linux error code otherwise.
*/
static inline int sbi_fwft_set_online_cpus(u32 feature, unsigned long value,
unsigned long flags)
{
return sbi_fwft_set_cpumask(cpu_online_mask, feature, value, flags);
}
/* Check if current SBI specification version is 0.1 or not */
static inline int sbi_spec_is_0_1(void)
{
@ -503,11 +553,21 @@ static inline int sbi_err_map_linux_errno(int err)
case SBI_SUCCESS:
return 0;
case SBI_ERR_DENIED:
case SBI_ERR_DENIED_LOCKED:
return -EPERM;
case SBI_ERR_INVALID_PARAM:
case SBI_ERR_INVALID_STATE:
return -EINVAL;
case SBI_ERR_BAD_RANGE:
return -ERANGE;
case SBI_ERR_INVALID_ADDRESS:
return -EFAULT;
case SBI_ERR_NO_SHMEM:
return -ENOMEM;
case SBI_ERR_TIMEOUT:
return -ETIMEDOUT;
case SBI_ERR_IO:
return -EIO;
case SBI_ERR_NOT_SUPPORTED:
case SBI_ERR_FAILURE:
default:

2
arch/riscv/include/asm/tlbflush.h
View file

@ -56,6 +56,8 @@ void local_flush_tlb_kernel_range(unsigned long start, unsigned long end);
#define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE
void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
unsigned long end);
void flush_pud_tlb_range(struct vm_area_struct *vma, unsigned long start,
unsigned long end);
#endif
bool arch_tlbbatch_should_defer(struct mm_struct *mm);

216
arch/riscv/include/asm/uaccess.h
View file

@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne
#define __disable_user_access() \
__asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory")
/*
* This is the smallest unsigned integer type that can fit a value
* (up to 'long long')
*/
#define __inttype(x) __typeof__( \
__typefits(x, char, \
__typefits(x, short, \
__typefits(x, int, \
__typefits(x, long, 0ULL)))))
#define __typefits(x, type, not) \
__builtin_choose_expr(sizeof(x) <= sizeof(type), (unsigned type)0, not)
/*
* The exception table consists of pairs of addresses: the first is the
* address of an instruction that is allowed to fault, and the second is
@ -83,27 +96,58 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne
* call.
*/
#define __get_user_asm(insn, x, ptr, err) \
#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT
#define __get_user_asm(insn, x, ptr, label) \
asm_goto_output( \
"1:\n" \
" " insn " %0, %1\n" \
_ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \
: "=&r" (x) \
: "m" (*(ptr)) : : label)
#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */
#define __get_user_asm(insn, x, ptr, label) \
do { \
__typeof__(x) __x; \
long __gua_err = 0; \
__asm__ __volatile__ ( \
"1:\n" \
" " insn " %1, %2\n" \
"2:\n" \
_ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \
: "+r" (err), "=&r" (__x) \
: "+r" (__gua_err), "=&r" (x) \
: "m" (*(ptr))); \
(x) = __x; \
if (__gua_err) \
goto label; \
} while (0)
#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */
#ifdef CONFIG_64BIT
#define __get_user_8(x, ptr, err) \
__get_user_asm("ld", x, ptr, err)
#define __get_user_8(x, ptr, label) \
__get_user_asm("ld", x, ptr, label)
#else /* !CONFIG_64BIT */
#define __get_user_8(x, ptr, err) \
#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT
#define __get_user_8(x, ptr, label) \
u32 __user *__ptr = (u32 __user *)(ptr); \
u32 __lo, __hi; \
asm_goto_output( \
"1:\n" \
" lw %0, %2\n" \
"2:\n" \
" lw %1, %3\n" \
_ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \
_ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \
: "=&r" (__lo), "=r" (__hi) \
: "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \
: : label); \
(x) = (__typeof__(x))((__typeof__((x) - (x)))( \
(((u64)__hi << 32) | __lo))); \
#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */
#define __get_user_8(x, ptr, label) \
do { \
u32 __user *__ptr = (u32 __user *)(ptr); \
u32 __lo, __hi; \
long __gu8_err = 0; \
__asm__ __volatile__ ( \
"1:\n" \
" lw %1, %3\n" \
@ -112,35 +156,62 @@ do { \
"3:\n" \
_ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \
_ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \
: "+r" (err), "=&r" (__lo), "=r" (__hi) \
: "+r" (__gu8_err), "=&r" (__lo), "=r" (__hi) \
: "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \
if (err) \
if (__gu8_err) { \
__hi = 0; \
goto label; \
} \
(x) = (__typeof__(x))((__typeof__((x) - (x)))( \
(((u64)__hi << 32) | __lo))); \
} while (0)
#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */
#endif /* CONFIG_64BIT */
#define __get_user_nocheck(x, __gu_ptr, __gu_err) \
unsigned long __must_check __asm_copy_to_user_sum_enabled(void __user *to,
const void *from, unsigned long n);
unsigned long __must_check __asm_copy_from_user_sum_enabled(void *to,
const void __user *from, unsigned long n);
#define __get_user_nocheck(x, __gu_ptr, label) \
do { \
if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && \
!IS_ALIGNED((uintptr_t)__gu_ptr, sizeof(*__gu_ptr))) { \
if (__asm_copy_from_user_sum_enabled(&(x), __gu_ptr, sizeof(*__gu_ptr))) \
goto label; \
break; \
} \
switch (sizeof(*__gu_ptr)) { \
case 1: \
__get_user_asm("lb", (x), __gu_ptr, __gu_err); \
__get_user_asm("lb", (x), __gu_ptr, label); \
break; \
case 2: \
__get_user_asm("lh", (x), __gu_ptr, __gu_err); \
__get_user_asm("lh", (x), __gu_ptr, label); \
break; \
case 4: \
__get_user_asm("lw", (x), __gu_ptr, __gu_err); \
__get_user_asm("lw", (x), __gu_ptr, label); \
break; \
case 8: \
__get_user_8((x), __gu_ptr, __gu_err); \
__get_user_8((x), __gu_ptr, label); \
break; \
default: \
BUILD_BUG(); \
} \
} while (0)
#define __get_user_error(x, ptr, err) \
do { \
__label__ __gu_failed; \
\
__get_user_nocheck(x, ptr, __gu_failed); \
err = 0; \
break; \
__gu_failed: \
x = 0; \
err = -EFAULT; \
} while (0)
/**
* __get_user: - Get a simple variable from user space, with less checking.
* @x: Variable to store result.
@ -165,13 +236,16 @@ do { \
({ \
const __typeof__(*(ptr)) __user *__gu_ptr = untagged_addr(ptr); \
long __gu_err = 0; \
__typeof__(x) __gu_val; \
\
__chk_user_ptr(__gu_ptr); \
\
__enable_user_access(); \
__get_user_nocheck(x, __gu_ptr, __gu_err); \
__get_user_error(__gu_val, __gu_ptr, __gu_err); \
__disable_user_access(); \
\
(x) = __gu_val; \
\
__gu_err; \
})
@ -201,61 +275,73 @@ do { \
((x) = (__force __typeof__(x))0, -EFAULT); \
})
#define __put_user_asm(insn, x, ptr, err) \
#define __put_user_asm(insn, x, ptr, label) \
do { \
__typeof__(*(ptr)) __x = x; \
__asm__ __volatile__ ( \
asm goto( \
"1:\n" \
" " insn " %z2, %1\n" \
"2:\n" \
_ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \
: "+r" (err), "=m" (*(ptr)) \
: "rJ" (__x)); \
" " insn " %z0, %1\n" \
_ASM_EXTABLE(1b, %l2) \
: : "rJ" (__x), "m"(*(ptr)) : : label); \
} while (0)
#ifdef CONFIG_64BIT
#define __put_user_8(x, ptr, err) \
__put_user_asm("sd", x, ptr, err)
#define __put_user_8(x, ptr, label) \
__put_user_asm("sd", x, ptr, label)
#else /* !CONFIG_64BIT */
#define __put_user_8(x, ptr, err) \
#define __put_user_8(x, ptr, label) \
do { \
u32 __user *__ptr = (u32 __user *)(ptr); \
u64 __x = (__typeof__((x)-(x)))(x); \
__asm__ __volatile__ ( \
asm goto( \
"1:\n" \
" sw %z3, %1\n" \
" sw %z0, %2\n" \
"2:\n" \
" sw %z4, %2\n" \
"3:\n" \
_ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \
_ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \
: "+r" (err), \
"=m" (__ptr[__LSW]), \
"=m" (__ptr[__MSW]) \
: "rJ" (__x), "rJ" (__x >> 32)); \
" sw %z1, %3\n" \
_ASM_EXTABLE(1b, %l4) \
_ASM_EXTABLE(2b, %l4) \
: : "rJ" (__x), "rJ" (__x >> 32), \
"m" (__ptr[__LSW]), \
"m" (__ptr[__MSW]) : : label); \
} while (0)
#endif /* CONFIG_64BIT */
#define __put_user_nocheck(x, __gu_ptr, __pu_err) \
#define __put_user_nocheck(x, __gu_ptr, label) \
do { \
if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && \
!IS_ALIGNED((uintptr_t)__gu_ptr, sizeof(*__gu_ptr))) { \
__inttype(x) val = (__inttype(x))x; \
if (__asm_copy_to_user_sum_enabled(__gu_ptr, &(val), sizeof(*__gu_ptr))) \
goto label; \
break; \
} \
switch (sizeof(*__gu_ptr)) { \
case 1: \
__put_user_asm("sb", (x), __gu_ptr, __pu_err); \
__put_user_asm("sb", (x), __gu_ptr, label); \
break; \
case 2: \
__put_user_asm("sh", (x), __gu_ptr, __pu_err); \
__put_user_asm("sh", (x), __gu_ptr, label); \
break; \
case 4: \
__put_user_asm("sw", (x), __gu_ptr, __pu_err); \
__put_user_asm("sw", (x), __gu_ptr, label); \
break; \
case 8: \
__put_user_8((x), __gu_ptr, __pu_err); \
__put_user_8((x), __gu_ptr, label); \
break; \
default: \
BUILD_BUG(); \
} \
} while (0)
#define __put_user_error(x, ptr, err) \
do { \
__label__ err_label; \
__put_user_nocheck(x, ptr, err_label); \
break; \
err_label: \
(err) = -EFAULT; \
} while (0)
/**
* __put_user: - Write a simple value into user space, with less checking.
* @x: Value to copy to user space.
@ -286,7 +372,7 @@ do { \
__chk_user_ptr(__gu_ptr); \
\
__enable_user_access(); \
__put_user_nocheck(__val, __gu_ptr, __pu_err); \
__put_user_error(__val, __gu_ptr, __pu_err); \
__disable_user_access(); \
\
__pu_err; \
@ -351,23 +437,45 @@ unsigned long __must_check clear_user(void __user *to, unsigned long n)
}
#define __get_kernel_nofault(dst, src, type, err_label) \
do { \
long __kr_err = 0; \
\
__get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \
if (unlikely(__kr_err)) \
goto err_label; \
} while (0)
__get_user_nocheck(*((type *)(dst)), (type *)(src), err_label)
#define __put_kernel_nofault(dst, src, type, err_label) \
do { \
long __kr_err = 0; \
\
__put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \
if (unlikely(__kr_err)) \
goto err_label; \
__put_user_nocheck(*((type *)(src)), (type *)(dst), err_label)
static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len)
{
if (unlikely(!access_ok(ptr, len)))
return 0;
__enable_user_access();
return 1;
}
#define user_access_begin user_access_begin
#define user_access_end __disable_user_access
static inline unsigned long user_access_save(void) { return 0UL; }
static inline void user_access_restore(unsigned long enabled) { }
/*
* We want the unsafe accessors to always be inlined and use
* the error labels - thus the macro games.
*/
#define unsafe_put_user(x, ptr, label) \
__put_user_nocheck(x, (ptr), label)
#define unsafe_get_user(x, ptr, label) do { \
__inttype(*(ptr)) __gu_val; \
__get_user_nocheck(__gu_val, (ptr), label); \
(x) = (__force __typeof__(*(ptr)))__gu_val; \
} while (0)
#define unsafe_copy_to_user(_dst, _src, _len, label) \
if (__asm_copy_to_user_sum_enabled(_dst, _src, _len)) \
goto label;
#define unsafe_copy_from_user(_dst, _src, _len, label) \
if (__asm_copy_from_user_sum_enabled(_dst, _src, _len)) \
goto label;
#else /* CONFIG_MMU */
#include <asm-generic/uaccess.h>
#endif /* CONFIG_MMU */

30
arch/riscv/include/asm/vdso/getrandom.h Normal file
View file

@ -0,0 +1,30 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (C) 2025 Xi Ruoyao <xry111@xry111.site>. All Rights Reserved.
*/
#ifndef __ASM_VDSO_GETRANDOM_H
#define __ASM_VDSO_GETRANDOM_H
#ifndef __ASSEMBLY__
#include <asm/unistd.h>
static __always_inline ssize_t getrandom_syscall(void *_buffer, size_t _len, unsigned int _flags)
{
register long ret asm("a0");
register long nr asm("a7") = __NR_getrandom;
register void *buffer asm("a0") = _buffer;
register size_t len asm("a1") = _len;
register unsigned int flags asm("a2") = _flags;
asm volatile ("ecall\n"
: "+r" (ret)
: "r" (nr), "r" (buffer), "r" (len), "r" (flags)
: "memory");
return ret;
}
#endif /* !__ASSEMBLY__ */
#endif /* __ASM_VDSO_GETRANDOM_H */

20
arch/riscv/include/asm/vector.h
View file

@ -120,6 +120,11 @@ static __always_inline void riscv_v_disable(void)
csr_clear(CSR_SSTATUS, SR_VS);
}
static __always_inline bool riscv_v_is_on(void)
{
return !!(csr_read(CSR_SSTATUS) & SR_VS);
}
static __always_inline void __vstate_csr_save(struct __riscv_v_ext_state *dest)
{
asm volatile (
@ -366,6 +371,11 @@ static inline void __switch_to_vector(struct task_struct *prev,
struct pt_regs *regs;
if (riscv_preempt_v_started(prev)) {
if (riscv_v_is_on()) {
WARN_ON(prev->thread.riscv_v_flags & RISCV_V_CTX_DEPTH_MASK);
riscv_v_disable();
prev->thread.riscv_v_flags |= RISCV_PREEMPT_V_IN_SCHEDULE;
}
if (riscv_preempt_v_dirty(prev)) {
__riscv_v_vstate_save(&prev->thread.kernel_vstate,
prev->thread.kernel_vstate.datap);
@ -376,11 +386,17 @@ static inline void __switch_to_vector(struct task_struct *prev,
riscv_v_vstate_save(&prev->thread.vstate, regs);
}
if (riscv_preempt_v_started(next))
if (riscv_preempt_v_started(next)) {
if (next->thread.riscv_v_flags & RISCV_PREEMPT_V_IN_SCHEDULE) {
next->thread.riscv_v_flags &= ~RISCV_PREEMPT_V_IN_SCHEDULE;
riscv_v_enable();
} else {
riscv_preempt_v_set_restore(next);
else
}
} else {
riscv_v_vstate_set_restore(next, task_pt_regs(next));
}
}
void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
bool riscv_v_vstate_ctrl_user_allowed(void);

16
arch/riscv/include/asm/vendor_extensions/sifive.h Normal file
View file

@ -0,0 +1,16 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _ASM_RISCV_VENDOR_EXTENSIONS_SIFIVE_H
#define _ASM_RISCV_VENDOR_EXTENSIONS_SIFIVE_H
#include <asm/vendor_extensions.h>
#include <linux/types.h>
#define RISCV_ISA_VENDOR_EXT_XSFVQMACCDOD 0
#define RISCV_ISA_VENDOR_EXT_XSFVQMACCQOQ 1
#define RISCV_ISA_VENDOR_EXT_XSFVFNRCLIPXFQF 2
#define RISCV_ISA_VENDOR_EXT_XSFVFWMACCQQQ 3
extern struct riscv_isa_vendor_ext_data_list riscv_isa_vendor_ext_list_sifive;
#endif

19
arch/riscv/include/asm/vendor_extensions/sifive_hwprobe.h Normal file
View file

@ -0,0 +1,19 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _ASM_RISCV_VENDOR_EXTENSIONS_SIFIVE_HWPROBE_H
#define _ASM_RISCV_VENDOR_EXTENSIONS_SIFIVE_HWPROBE_H
#include <linux/cpumask.h>
#include <uapi/asm/hwprobe.h>
#ifdef CONFIG_RISCV_ISA_VENDOR_EXT_SIFIVE
void hwprobe_isa_vendor_ext_sifive_0(struct riscv_hwprobe *pair, const struct cpumask *cpus);
#else
static inline void hwprobe_isa_vendor_ext_sifive_0(struct riscv_hwprobe *pair,
const struct cpumask *cpus)
{
pair->value = 0;
}
#endif
#endif

2
arch/riscv/include/uapi/asm/hwprobe.h
View file

@ -81,6 +81,7 @@ struct riscv_hwprobe {
#define RISCV_HWPROBE_EXT_ZICBOM (1ULL << 55)
#define RISCV_HWPROBE_EXT_ZAAMO (1ULL << 56)
#define RISCV_HWPROBE_EXT_ZALRSC (1ULL << 57)
#define RISCV_HWPROBE_EXT_ZABHA (1ULL << 58)
#define RISCV_HWPROBE_KEY_CPUPERF_0 5
#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0)
#define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0)
@ -104,6 +105,7 @@ struct riscv_hwprobe {
#define RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED 4
#define RISCV_HWPROBE_KEY_VENDOR_EXT_THEAD_0 11
#define RISCV_HWPROBE_KEY_ZICBOM_BLOCK_SIZE 12
#define RISCV_HWPROBE_KEY_VENDOR_EXT_SIFIVE_0 13
/* Increase RISCV_HWPROBE_MAX_KEY when adding items. */
/* Flags */

6
arch/riscv/include/uapi/asm/vendor/sifive.h vendored Normal file
View file

@ -0,0 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
#define RISCV_HWPROBE_VENDOR_EXT_XSFVQMACCDOD (1 << 0)
#define RISCV_HWPROBE_VENDOR_EXT_XSFVQMACCQOQ (1 << 1)
#define RISCV_HWPROBE_VENDOR_EXT_XSFVFNRCLIPXFQF (1 << 2)
#define RISCV_HWPROBE_VENDOR_EXT_XSFVFWMACCQQQ (1 << 3)

2
arch/riscv/kernel/Makefile
View file

@ -107,7 +107,7 @@ obj-$(CONFIG_HOTPLUG_CPU) += cpu-hotplug.o
obj-$(CONFIG_PARAVIRT) += paravirt.o
obj-$(CONFIG_KGDB) += kgdb.o
obj-$(CONFIG_KEXEC_CORE) += kexec_relocate.o crash_save_regs.o machine_kexec.o
obj-$(CONFIG_KEXEC_FILE) += elf_kexec.o machine_kexec_file.o
obj-$(CONFIG_KEXEC_FILE) += kexec_elf.o kexec_image.o machine_kexec_file.o
obj-$(CONFIG_CRASH_DUMP) += crash_dump.o
obj-$(CONFIG_VMCORE_INFO) += vmcore_info.o

18
arch/riscv/kernel/asm-offsets.c
View file

@ -34,6 +34,7 @@ void asm_offsets(void)
OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]);
OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]);
OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]);
OFFSET(TASK_THREAD_SUM, task_struct, thread.sum);
OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu);
OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count);
@ -346,6 +347,10 @@ void asm_offsets(void)
offsetof(struct task_struct, thread.s[11])
- offsetof(struct task_struct, thread.ra)
);
DEFINE(TASK_THREAD_SUM_RA,
offsetof(struct task_struct, thread.sum)
- offsetof(struct task_struct, thread.ra)
);
DEFINE(TASK_THREAD_F0_F0,
offsetof(struct task_struct, thread.fstate.f[0])
@ -493,6 +498,12 @@ void asm_offsets(void)
DEFINE(STACKFRAME_SIZE_ON_STACK, ALIGN(sizeof(struct stackframe), STACK_ALIGN));
OFFSET(STACKFRAME_FP, stackframe, fp);
OFFSET(STACKFRAME_RA, stackframe, ra);
#ifdef CONFIG_FUNCTION_TRACER
DEFINE(FTRACE_OPS_FUNC, offsetof(struct ftrace_ops, func));
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
DEFINE(FTRACE_OPS_DIRECT_CALL, offsetof(struct ftrace_ops, direct_call));
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
#endif
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
DEFINE(FREGS_SIZE_ON_STACK, ALIGN(sizeof(struct __arch_ftrace_regs), STACK_ALIGN));
@ -501,6 +512,13 @@ void asm_offsets(void)
DEFINE(FREGS_SP, offsetof(struct __arch_ftrace_regs, sp));
DEFINE(FREGS_S0, offsetof(struct __arch_ftrace_regs, s0));
DEFINE(FREGS_T1, offsetof(struct __arch_ftrace_regs, t1));
#ifdef CONFIG_CC_IS_CLANG
DEFINE(FREGS_T2, offsetof(struct __arch_ftrace_regs, t2));
DEFINE(FREGS_T3, offsetof(struct __arch_ftrace_regs, t3));
DEFINE(FREGS_T4, offsetof(struct __arch_ftrace_regs, t4));
DEFINE(FREGS_T5, offsetof(struct __arch_ftrace_regs, t5));
DEFINE(FREGS_T6, offsetof(struct __arch_ftrace_regs, t6));
#endif
DEFINE(FREGS_A0, offsetof(struct __arch_ftrace_regs, a0));
DEFINE(FREGS_A1, offsetof(struct __arch_ftrace_regs, a1));
DEFINE(FREGS_A2, offsetof(struct __arch_ftrace_regs, a2));

21
arch/riscv/kernel/cpufeature.c
View file

@ -32,6 +32,7 @@
#define NUM_ALPHA_EXTS ('z' - 'a' + 1)
static bool any_cpu_has_zicboz;
static bool any_cpu_has_zicbop;
static bool any_cpu_has_zicbom;
unsigned long elf_hwcap __read_mostly;
@ -119,6 +120,21 @@ static int riscv_ext_zicboz_validate(const struct riscv_isa_ext_data *data,
return 0;
}
static int riscv_ext_zicbop_validate(const struct riscv_isa_ext_data *data,
const unsigned long *isa_bitmap)
{
if (!riscv_cbop_block_size) {
pr_err("Zicbop detected in ISA string, disabling as no cbop-block-size found\n");
return -EINVAL;
}
if (!is_power_of_2(riscv_cbop_block_size)) {
pr_err("Zicbop disabled as cbop-block-size present, but is not a power-of-2\n");
return -EINVAL;
}
any_cpu_has_zicbop = true;
return 0;
}
static int riscv_ext_f_validate(const struct riscv_isa_ext_data *data,
const unsigned long *isa_bitmap)
{
@ -442,6 +458,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = {
__RISCV_ISA_EXT_SUPERSET_VALIDATE(v, RISCV_ISA_EXT_v, riscv_v_exts, riscv_ext_vector_float_validate),
__RISCV_ISA_EXT_DATA(h, RISCV_ISA_EXT_h),
__RISCV_ISA_EXT_SUPERSET_VALIDATE(zicbom, RISCV_ISA_EXT_ZICBOM, riscv_xlinuxenvcfg_exts, riscv_ext_zicbom_validate),
__RISCV_ISA_EXT_DATA_VALIDATE(zicbop, RISCV_ISA_EXT_ZICBOP, riscv_ext_zicbop_validate),
__RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts, riscv_ext_zicboz_validate),
__RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE),
__RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR),
@ -1112,6 +1129,10 @@ void __init riscv_user_isa_enable(void)
current->thread.envcfg |= ENVCFG_CBCFE;
else if (any_cpu_has_zicbom)
pr_warn("Zicbom disabled as it is unavailable on some harts\n");
if (!riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICBOP) &&
any_cpu_has_zicbop)
pr_warn("Zicbop disabled as it is unavailable on some harts\n");
}
#ifdef CONFIG_RISCV_ALTERNATIVE

485
arch/riscv/kernel/elf_kexec.c
View file

@ -1,485 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Load ELF vmlinux file for the kexec_file_load syscall.
*
* Copyright (C) 2021 Huawei Technologies Co, Ltd.
*
* Author: Liao Chang (liaochang1@huawei.com)
*
* Based on kexec-tools' kexec-elf-riscv.c, heavily modified
* for kernel.
*/
#define pr_fmt(fmt) "kexec_image: " fmt
#include <linux/elf.h>
#include <linux/kexec.h>
#include <linux/slab.h>
#include <linux/of.h>
#include <linux/libfdt.h>
#include <linux/types.h>
#include <linux/memblock.h>
#include <linux/vmalloc.h>
#include <asm/setup.h>
int arch_kimage_file_post_load_cleanup(struct kimage *image)
{
kvfree(image->arch.fdt);
image->arch.fdt = NULL;
vfree(image->elf_headers);
image->elf_headers = NULL;
image->elf_headers_sz = 0;
return kexec_image_post_load_cleanup_default(image);
}
static int riscv_kexec_elf_load(struct kimage *image, struct elfhdr *ehdr,
struct kexec_elf_info *elf_info, unsigned long old_pbase,
unsigned long new_pbase)
{
int i;
int ret = 0;
size_t size;
struct kexec_buf kbuf;
const struct elf_phdr *phdr;
kbuf.image = image;
for (i = 0; i < ehdr->e_phnum; i++) {
phdr = &elf_info->proghdrs[i];
if (phdr->p_type != PT_LOAD)
continue;
size = phdr->p_filesz;
if (size > phdr->p_memsz)
size = phdr->p_memsz;
kbuf.buffer = (void *) elf_info->buffer + phdr->p_offset;
kbuf.bufsz = size;
kbuf.buf_align = phdr->p_align;
kbuf.mem = phdr->p_paddr - old_pbase + new_pbase;
kbuf.memsz = phdr->p_memsz;
kbuf.top_down = false;
ret = kexec_add_buffer(&kbuf);
if (ret)
break;
}
return ret;
}
/*
* Go through the available phsyical memory regions and find one that hold
* an image of the specified size.
*/
static int elf_find_pbase(struct kimage *image, unsigned long kernel_len,
struct elfhdr *ehdr, struct kexec_elf_info *elf_info,
unsigned long *old_pbase, unsigned long *new_pbase)
{
int i;
int ret;
struct kexec_buf kbuf;
const struct elf_phdr *phdr;
unsigned long lowest_paddr = ULONG_MAX;
unsigned long lowest_vaddr = ULONG_MAX;
for (i = 0; i < ehdr->e_phnum; i++) {
phdr = &elf_info->proghdrs[i];
if (phdr->p_type != PT_LOAD)
continue;
if (lowest_paddr > phdr->p_paddr)
lowest_paddr = phdr->p_paddr;
if (lowest_vaddr > phdr->p_vaddr)
lowest_vaddr = phdr->p_vaddr;
}
kbuf.image = image;
kbuf.buf_min = lowest_paddr;
kbuf.buf_max = ULONG_MAX;
/*
* Current riscv boot protocol requires 2MB alignment for
* RV64 and 4MB alignment for RV32
*
*/
kbuf.buf_align = PMD_SIZE;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
kbuf.memsz = ALIGN(kernel_len, PAGE_SIZE);
kbuf.top_down = false;
ret = arch_kexec_locate_mem_hole(&kbuf);
if (!ret) {
*old_pbase = lowest_paddr;
*new_pbase = kbuf.mem;
image->start = ehdr->e_entry - lowest_vaddr + kbuf.mem;
}
return ret;
}
#ifdef CONFIG_CRASH_DUMP
static int get_nr_ram_ranges_callback(struct resource *res, void *arg)
{
unsigned int *nr_ranges = arg;
(*nr_ranges)++;
return 0;
}
static int prepare_elf64_ram_headers_callback(struct resource *res, void *arg)
{
struct crash_mem *cmem = arg;
cmem->ranges[cmem->nr_ranges].start = res->start;
cmem->ranges[cmem->nr_ranges].end = res->end;
cmem->nr_ranges++;
return 0;
}
static int prepare_elf_headers(void **addr, unsigned long *sz)
{
struct crash_mem *cmem;
unsigned int nr_ranges;
int ret;
nr_ranges = 1; /* For exclusion of crashkernel region */
walk_system_ram_res(0, -1, &nr_ranges, get_nr_ram_ranges_callback);
cmem = kmalloc(struct_size(cmem, ranges, nr_ranges), GFP_KERNEL);
if (!cmem)
return -ENOMEM;
cmem->max_nr_ranges = nr_ranges;
cmem->nr_ranges = 0;
ret = walk_system_ram_res(0, -1, cmem, prepare_elf64_ram_headers_callback);
if (ret)
goto out;
/* Exclude crashkernel region */
ret = crash_exclude_mem_range(cmem, crashk_res.start, crashk_res.end);
if (!ret)
ret = crash_prepare_elf64_headers(cmem, true, addr, sz);
out:
kfree(cmem);
return ret;
}
static char *setup_kdump_cmdline(struct kimage *image, char *cmdline,
unsigned long cmdline_len)
{
int elfcorehdr_strlen;
char *cmdline_ptr;
cmdline_ptr = kzalloc(COMMAND_LINE_SIZE, GFP_KERNEL);
if (!cmdline_ptr)
return NULL;
elfcorehdr_strlen = sprintf(cmdline_ptr, "elfcorehdr=0x%lx ",
image->elf_load_addr);
if (elfcorehdr_strlen + cmdline_len > COMMAND_LINE_SIZE) {
pr_err("Appending elfcorehdr=<addr> exceeds cmdline size\n");
kfree(cmdline_ptr);
return NULL;
}
memcpy(cmdline_ptr + elfcorehdr_strlen, cmdline, cmdline_len);
/* Ensure it's nul terminated */
cmdline_ptr[COMMAND_LINE_SIZE - 1] = '\0';
return cmdline_ptr;
}
#endif
static void *elf_kexec_load(struct kimage *image, char *kernel_buf,
unsigned long kernel_len, char *initrd,
unsigned long initrd_len, char *cmdline,
unsigned long cmdline_len)
{
int ret;
void *fdt;
unsigned long old_kernel_pbase = ULONG_MAX;
unsigned long new_kernel_pbase = 0UL;
unsigned long initrd_pbase = 0UL;
unsigned long kernel_start;
struct elfhdr ehdr;
struct kexec_buf kbuf;
struct kexec_elf_info elf_info;
char *modified_cmdline = NULL;
ret = kexec_build_elf_info(kernel_buf, kernel_len, &ehdr, &elf_info);
if (ret)
return ERR_PTR(ret);
ret = elf_find_pbase(image, kernel_len, &ehdr, &elf_info,
&old_kernel_pbase, &new_kernel_pbase);
if (ret)
goto out;
kernel_start = image->start;
/* Add the kernel binary to the image */
ret = riscv_kexec_elf_load(image, &ehdr, &elf_info,
old_kernel_pbase, new_kernel_pbase);
if (ret)
goto out;
kbuf.image = image;
kbuf.buf_min = new_kernel_pbase + kernel_len;
kbuf.buf_max = ULONG_MAX;
#ifdef CONFIG_CRASH_DUMP
/* Add elfcorehdr */
if (image->type == KEXEC_TYPE_CRASH) {
void *headers;
unsigned long headers_sz;
ret = prepare_elf_headers(&headers, &headers_sz);
if (ret) {
pr_err("Preparing elf core header failed\n");
goto out;
}
kbuf.buffer = headers;
kbuf.bufsz = headers_sz;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
kbuf.memsz = headers_sz;
kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
kbuf.top_down = true;
ret = kexec_add_buffer(&kbuf);
if (ret) {
vfree(headers);
goto out;
}
image->elf_headers = headers;
image->elf_load_addr = kbuf.mem;
image->elf_headers_sz = headers_sz;
kexec_dprintk("Loaded elf core header at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
image->elf_load_addr, kbuf.bufsz, kbuf.memsz);
/* Setup cmdline for kdump kernel case */
modified_cmdline = setup_kdump_cmdline(image, cmdline,
cmdline_len);
if (!modified_cmdline) {
pr_err("Setting up cmdline for kdump kernel failed\n");
ret = -EINVAL;
goto out;
}
cmdline = modified_cmdline;
}
#endif
#ifdef CONFIG_ARCH_SUPPORTS_KEXEC_PURGATORY
/* Add purgatory to the image */
kbuf.top_down = true;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
ret = kexec_load_purgatory(image, &kbuf);
if (ret) {
pr_err("Error loading purgatory ret=%d\n", ret);
goto out;
}
kexec_dprintk("Loaded purgatory at 0x%lx\n", kbuf.mem);
ret = kexec_purgatory_get_set_symbol(image, "riscv_kernel_entry",
&kernel_start,
sizeof(kernel_start), 0);
if (ret)
pr_err("Error update purgatory ret=%d\n", ret);
#endif /* CONFIG_ARCH_SUPPORTS_KEXEC_PURGATORY */
/* Add the initrd to the image */
if (initrd != NULL) {
kbuf.buffer = initrd;
kbuf.bufsz = kbuf.memsz = initrd_len;
kbuf.buf_align = PAGE_SIZE;
kbuf.top_down = true;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
ret = kexec_add_buffer(&kbuf);
if (ret)
goto out;
initrd_pbase = kbuf.mem;
kexec_dprintk("Loaded initrd at 0x%lx\n", initrd_pbase);
}
/* Add the DTB to the image */
fdt = of_kexec_alloc_and_setup_fdt(image, initrd_pbase,
initrd_len, cmdline, 0);
if (!fdt) {
pr_err("Error setting up the new device tree.\n");
ret = -EINVAL;
goto out;
}
fdt_pack(fdt);
kbuf.buffer = fdt;
kbuf.bufsz = kbuf.memsz = fdt_totalsize(fdt);
kbuf.buf_align = PAGE_SIZE;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
kbuf.top_down = true;
ret = kexec_add_buffer(&kbuf);
if (ret) {
pr_err("Error add DTB kbuf ret=%d\n", ret);
goto out_free_fdt;
}
/* Cache the fdt buffer address for memory cleanup */
image->arch.fdt = fdt;
kexec_dprintk("Loaded device tree at 0x%lx\n", kbuf.mem);
goto out;
out_free_fdt:
kvfree(fdt);
out:
kfree(modified_cmdline);
kexec_free_elf_info(&elf_info);
return ret ? ERR_PTR(ret) : NULL;
}
#define RV_X(x, s, n) (((x) >> (s)) & ((1 << (n)) - 1))
#define RISCV_IMM_BITS 12
#define RISCV_IMM_REACH (1LL << RISCV_IMM_BITS)
#define RISCV_CONST_HIGH_PART(x) \
(((x) + (RISCV_IMM_REACH >> 1)) & ~(RISCV_IMM_REACH - 1))
#define RISCV_CONST_LOW_PART(x) ((x) - RISCV_CONST_HIGH_PART(x))
#define ENCODE_ITYPE_IMM(x) \
(RV_X(x, 0, 12) << 20)
#define ENCODE_BTYPE_IMM(x) \
((RV_X(x, 1, 4) << 8) | (RV_X(x, 5, 6) << 25) | \
(RV_X(x, 11, 1) << 7) | (RV_X(x, 12, 1) << 31))
#define ENCODE_UTYPE_IMM(x) \
(RV_X(x, 12, 20) << 12)
#define ENCODE_JTYPE_IMM(x) \
((RV_X(x, 1, 10) << 21) | (RV_X(x, 11, 1) << 20) | \
(RV_X(x, 12, 8) << 12) | (RV_X(x, 20, 1) << 31))
#define ENCODE_CBTYPE_IMM(x) \
((RV_X(x, 1, 2) << 3) | (RV_X(x, 3, 2) << 10) | (RV_X(x, 5, 1) << 2) | \
(RV_X(x, 6, 2) << 5) | (RV_X(x, 8, 1) << 12))
#define ENCODE_CJTYPE_IMM(x) \
((RV_X(x, 1, 3) << 3) | (RV_X(x, 4, 1) << 11) | (RV_X(x, 5, 1) << 2) | \
(RV_X(x, 6, 1) << 7) | (RV_X(x, 7, 1) << 6) | (RV_X(x, 8, 2) << 9) | \
(RV_X(x, 10, 1) << 8) | (RV_X(x, 11, 1) << 12))
#define ENCODE_UJTYPE_IMM(x) \
(ENCODE_UTYPE_IMM(RISCV_CONST_HIGH_PART(x)) | \
(ENCODE_ITYPE_IMM(RISCV_CONST_LOW_PART(x)) << 32))
#define ENCODE_UITYPE_IMM(x) \
(ENCODE_UTYPE_IMM(x) | (ENCODE_ITYPE_IMM(x) << 32))
#define CLEAN_IMM(type, x) \
((~ENCODE_##type##_IMM((uint64_t)(-1))) & (x))
int arch_kexec_apply_relocations_add(struct purgatory_info *pi,
Elf_Shdr *section,
const Elf_Shdr *relsec,
const Elf_Shdr *symtab)
{
const char *strtab, *name, *shstrtab;
const Elf_Shdr *sechdrs;
Elf64_Rela *relas;
int i, r_type;
/* String & section header string table */
sechdrs = (void *)pi->ehdr + pi->ehdr->e_shoff;
strtab = (char *)pi->ehdr + sechdrs[symtab->sh_link].sh_offset;
shstrtab = (char *)pi->ehdr + sechdrs[pi->ehdr->e_shstrndx].sh_offset;
relas = (void *)pi->ehdr + relsec->sh_offset;
for (i = 0; i < relsec->sh_size / sizeof(*relas); i++) {
const Elf_Sym *sym; /* symbol to relocate */
unsigned long addr; /* final location after relocation */
unsigned long val; /* relocated symbol value */
unsigned long sec_base; /* relocated symbol value */
void *loc; /* tmp location to modify */
sym = (void *)pi->ehdr + symtab->sh_offset;
sym += ELF64_R_SYM(relas[i].r_info);
if (sym->st_name)
name = strtab + sym->st_name;
else
name = shstrtab + sechdrs[sym->st_shndx].sh_name;
loc = pi->purgatory_buf;
loc += section->sh_offset;
loc += relas[i].r_offset;
if (sym->st_shndx == SHN_ABS)
sec_base = 0;
else if (sym->st_shndx >= pi->ehdr->e_shnum) {
pr_err("Invalid section %d for symbol %s\n",
sym->st_shndx, name);
return -ENOEXEC;
} else
sec_base = pi->sechdrs[sym->st_shndx].sh_addr;
val = sym->st_value;
val += sec_base;
val += relas[i].r_addend;
addr = section->sh_addr + relas[i].r_offset;
r_type = ELF64_R_TYPE(relas[i].r_info);
switch (r_type) {
case R_RISCV_BRANCH:
*(u32 *)loc = CLEAN_IMM(BTYPE, *(u32 *)loc) |
ENCODE_BTYPE_IMM(val - addr);
break;
case R_RISCV_JAL:
*(u32 *)loc = CLEAN_IMM(JTYPE, *(u32 *)loc) |
ENCODE_JTYPE_IMM(val - addr);
break;
/*
* With no R_RISCV_PCREL_LO12_S, R_RISCV_PCREL_LO12_I
* sym is expected to be next to R_RISCV_PCREL_HI20
* in purgatory relsec. Handle it like R_RISCV_CALL
* sym, instead of searching the whole relsec.
*/
case R_RISCV_PCREL_HI20:
case R_RISCV_CALL_PLT:
case R_RISCV_CALL:
*(u64 *)loc = CLEAN_IMM(UITYPE, *(u64 *)loc) |
ENCODE_UJTYPE_IMM(val - addr);
break;
case R_RISCV_RVC_BRANCH:
*(u32 *)loc = CLEAN_IMM(CBTYPE, *(u32 *)loc) |
ENCODE_CBTYPE_IMM(val - addr);
break;
case R_RISCV_RVC_JUMP:
*(u32 *)loc = CLEAN_IMM(CJTYPE, *(u32 *)loc) |
ENCODE_CJTYPE_IMM(val - addr);
break;
case R_RISCV_ADD16:
*(u16 *)loc += val;
break;
case R_RISCV_SUB16:
*(u16 *)loc -= val;
break;
case R_RISCV_ADD32:
*(u32 *)loc += val;
break;
case R_RISCV_SUB32:
*(u32 *)loc -= val;
break;
/* It has been applied by R_RISCV_PCREL_HI20 sym */
case R_RISCV_PCREL_LO12_I:
case R_RISCV_ALIGN:
case R_RISCV_RELAX:
break;
case R_RISCV_64:
*(u64 *)loc = val;
break;
default:
pr_err("Unknown rela relocation: %d\n", r_type);
return -ENOEXEC;
}
}
return 0;
}
const struct kexec_file_ops elf_kexec_ops = {
.probe = kexec_elf_probe,
.load = elf_kexec_load,
};

9
arch/riscv/kernel/entry.S
View file

@ -401,9 +401,18 @@ SYM_FUNC_START(__switch_to)
REG_S s9, TASK_THREAD_S9_RA(a3)
REG_S s10, TASK_THREAD_S10_RA(a3)
REG_S s11, TASK_THREAD_S11_RA(a3)
/* save the user space access flag */
csrr s0, CSR_STATUS
REG_S s0, TASK_THREAD_SUM_RA(a3)
/* Save the kernel shadow call stack pointer */
scs_save_current
/* Restore context from next->thread */
REG_L s0, TASK_THREAD_SUM_RA(a4)
li s1, SR_SUM
and s0, s0, s1
csrs CSR_STATUS, s0
REG_L ra, TASK_THREAD_RA_RA(a4)
REG_L sp, TASK_THREAD_SP_RA(a4)
REG_L s0, TASK_THREAD_S0_RA(a4)

236
arch/riscv/kernel/ftrace.c
View file

@ -8,98 +8,129 @@
#include <linux/ftrace.h>
#include <linux/uaccess.h>
#include <linux/memory.h>
#include <linux/irqflags.h>
#include <linux/stop_machine.h>
#include <asm/cacheflush.h>
#include <asm/text-patching.h>
#ifdef CONFIG_DYNAMIC_FTRACE
void ftrace_arch_code_modify_prepare(void) __acquires(&text_mutex)
unsigned long ftrace_call_adjust(unsigned long addr)
{
if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS))
return addr + 8 + MCOUNT_AUIPC_SIZE;
return addr + MCOUNT_AUIPC_SIZE;
}
unsigned long arch_ftrace_get_symaddr(unsigned long fentry_ip)
{
return fentry_ip - MCOUNT_AUIPC_SIZE;
}
void arch_ftrace_update_code(int command)
{
mutex_lock(&text_mutex);
/*
* The code sequences we use for ftrace can't be patched while the
* kernel is running, so we need to use stop_machine() to modify them
* for now. This doesn't play nice with text_mutex, we use this flag
* to elide the check.
*/
riscv_patch_in_stop_machine = true;
}
void ftrace_arch_code_modify_post_process(void) __releases(&text_mutex)
{
riscv_patch_in_stop_machine = false;
command |= FTRACE_MAY_SLEEP;
ftrace_modify_all_code(command);
mutex_unlock(&text_mutex);
flush_icache_all();
}
static int ftrace_check_current_call(unsigned long hook_pos,
unsigned int *expected)
static int __ftrace_modify_call(unsigned long source, unsigned long target, bool validate)
{
unsigned int call[2], offset;
unsigned int replaced[2];
unsigned int nops[2] = {RISCV_INSN_NOP4, RISCV_INSN_NOP4};
/* we expect nops at the hook position */
if (!expected)
expected = nops;
offset = target - source;
call[1] = to_jalr_t0(offset);
if (validate) {
call[0] = to_auipc_t0(offset);
/*
* Read the text we want to modify;
* return must be -EFAULT on read error
*/
if (copy_from_kernel_nofault(replaced, (void *)hook_pos,
MCOUNT_INSN_SIZE))
if (copy_from_kernel_nofault(replaced, (void *)source, 2 * MCOUNT_INSN_SIZE))
return -EFAULT;
/*
* Make sure it is what we expect it to be;
* return must be -EINVAL on failed comparison
*/
if (memcmp(expected, replaced, sizeof(replaced))) {
pr_err("%p: expected (%08x %08x) but got (%08x %08x)\n",
(void *)hook_pos, expected[0], expected[1], replaced[0],
replaced[1]);
if (replaced[0] != call[0]) {
pr_err("%p: expected (%08x) but got (%08x)\n",
(void *)source, call[0], replaced[0]);
return -EINVAL;
}
return 0;
}
static int __ftrace_modify_call(unsigned long hook_pos, unsigned long target,
bool enable, bool ra)
{
unsigned int call[2];
unsigned int nops[2] = {RISCV_INSN_NOP4, RISCV_INSN_NOP4};
if (ra)
make_call_ra(hook_pos, target, call);
else
make_call_t0(hook_pos, target, call);
/* Replace the auipc-jalr pair at once. Return -EPERM on write error. */
if (patch_insn_write((void *)hook_pos, enable ? call : nops, MCOUNT_INSN_SIZE))
/* Replace the jalr at once. Return -EPERM on write error. */
if (patch_insn_write((void *)(source + MCOUNT_AUIPC_SIZE), call + 1, MCOUNT_JALR_SIZE))
return -EPERM;
return 0;
}
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
static const struct ftrace_ops *riscv64_rec_get_ops(struct dyn_ftrace *rec)
{
const struct ftrace_ops *ops = NULL;
if (rec->flags & FTRACE_FL_CALL_OPS_EN) {
ops = ftrace_find_unique_ops(rec);
WARN_ON_ONCE(!ops);
}
if (!ops)
ops = &ftrace_list_ops;
return ops;
}
static int ftrace_rec_set_ops(const struct dyn_ftrace *rec, const struct ftrace_ops *ops)
{
unsigned long literal = ALIGN_DOWN(rec->ip - 12, 8);
return patch_text_nosync((void *)literal, &ops, sizeof(ops));
}
static int ftrace_rec_set_nop_ops(struct dyn_ftrace *rec)
{
return ftrace_rec_set_ops(rec, &ftrace_nop_ops);
}
static int ftrace_rec_update_ops(struct dyn_ftrace *rec)
{
return ftrace_rec_set_ops(rec, riscv64_rec_get_ops(rec));
}
#else
static int ftrace_rec_set_nop_ops(struct dyn_ftrace *rec) { return 0; }
static int ftrace_rec_update_ops(struct dyn_ftrace *rec) { return 0; }
#endif
int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
{
unsigned int call[2];
unsigned long distance, orig_addr, pc = rec->ip - MCOUNT_AUIPC_SIZE;
int ret;
make_call_t0(rec->ip, addr, call);
ret = ftrace_rec_update_ops(rec);
if (ret)
return ret;
if (patch_insn_write((void *)rec->ip, call, MCOUNT_INSN_SIZE))
return -EPERM;
orig_addr = (unsigned long)&ftrace_caller;
distance = addr > orig_addr ? addr - orig_addr : orig_addr - addr;
if (distance > JALR_RANGE)
addr = FTRACE_ADDR;
return 0;
return __ftrace_modify_call(pc, addr, false);
}
int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec,
unsigned long addr)
int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr)
{
unsigned int nops[2] = {RISCV_INSN_NOP4, RISCV_INSN_NOP4};
u32 nop4 = RISCV_INSN_NOP4;
int ret;
if (patch_insn_write((void *)rec->ip, nops, MCOUNT_INSN_SIZE))
ret = ftrace_rec_set_nop_ops(rec);
if (ret)
return ret;
if (patch_insn_write((void *)rec->ip, &nop4, MCOUNT_NOP4_SIZE))
return -EPERM;
return 0;
@ -114,75 +145,71 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec,
*/
int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec)
{
int out;
unsigned long pc = rec->ip - MCOUNT_AUIPC_SIZE;
unsigned int nops[2], offset;
int ret;
ret = ftrace_rec_set_nop_ops(rec);
if (ret)
return ret;
offset = (unsigned long) &ftrace_caller - pc;
nops[0] = to_auipc_t0(offset);
nops[1] = RISCV_INSN_NOP4;
mutex_lock(&text_mutex);
out = ftrace_make_nop(mod, rec, MCOUNT_ADDR);
ret = patch_insn_write((void *)pc, nops, 2 * MCOUNT_INSN_SIZE);
mutex_unlock(&text_mutex);
return out;
}
int ftrace_update_ftrace_func(ftrace_func_t func)
{
int ret = __ftrace_modify_call((unsigned long)&ftrace_call,
(unsigned long)func, true, true);
return ret;
}
struct ftrace_modify_param {
int command;
atomic_t cpu_count;
};
static int __ftrace_modify_code(void *data)
ftrace_func_t ftrace_call_dest = ftrace_stub;
int ftrace_update_ftrace_func(ftrace_func_t func)
{
struct ftrace_modify_param *param = data;
if (atomic_inc_return(&param->cpu_count) == num_online_cpus()) {
ftrace_modify_all_code(param->command);
/*
* Make sure the patching store is effective *before* we
* increment the counter which releases all waiting CPUs
* by using the release variant of atomic increment. The
* release pairs with the call to local_flush_icache_all()
* on the waiting CPU.
* When using CALL_OPS, the function to call is associated with the
* call site, and we don't have a global function pointer to update.
*/
atomic_inc_return_release(&param->cpu_count);
} else {
while (atomic_read(&param->cpu_count) <= num_online_cpus())
cpu_relax();
local_flush_icache_all();
}
if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS))
return 0;
WRITE_ONCE(ftrace_call_dest, func);
/*
* The data fence ensure that the update to ftrace_call_dest happens
* before the write to function_trace_op later in the generic ftrace.
* If the sequence is not enforced, then an old ftrace_call_dest may
* race loading a new function_trace_op set in ftrace_modify_all_code
*/
smp_wmb();
/*
* Updating ftrace dpes not take stop_machine path, so irqs should not
* be disabled.
*/
WARN_ON(irqs_disabled());
smp_call_function(ftrace_sync_ipi, NULL, 1);
return 0;
}
void arch_ftrace_update_code(int command)
#else /* CONFIG_DYNAMIC_FTRACE */
unsigned long ftrace_call_adjust(unsigned long addr)
{
struct ftrace_modify_param param = { command, ATOMIC_INIT(0) };
stop_machine(__ftrace_modify_code, &param, cpu_online_mask);
return addr;
}
#endif
#endif /* CONFIG_DYNAMIC_FTRACE */
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr,
unsigned long addr)
{
unsigned int call[2];
unsigned long caller = rec->ip;
unsigned long caller = rec->ip - MCOUNT_AUIPC_SIZE;
int ret;
make_call_t0(caller, old_addr, call);
ret = ftrace_check_current_call(caller, call);
ret = ftrace_rec_update_ops(rec);
if (ret)
return ret;
return __ftrace_modify_call(caller, addr, true, false);
return __ftrace_modify_call(caller, FTRACE_ADDR, true);
}
#endif
@ -210,7 +237,6 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr,
}
#ifdef CONFIG_DYNAMIC_FTRACE
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
struct ftrace_ops *op, struct ftrace_regs *fregs)
{
@ -231,19 +257,5 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
if (!function_graph_enter_regs(old, ip, frame_pointer, parent, fregs))
*parent = return_hooker;
}
#else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
extern void ftrace_graph_call(void);
int ftrace_enable_ftrace_graph_caller(void)
{
return __ftrace_modify_call((unsigned long)&ftrace_graph_call,
(unsigned long)&prepare_ftrace_return, true, true);
}
int ftrace_disable_ftrace_graph_caller(void)
{
return __ftrace_modify_call((unsigned long)&ftrace_graph_call,
(unsigned long)&prepare_ftrace_return, false, true);
}
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
#endif /* CONFIG_DYNAMIC_FTRACE */
#endif /* CONFIG_FUNCTION_GRAPH_TRACER */

144
arch/riscv/kernel/kexec_elf.c Normal file
View file

@ -0,0 +1,144 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Load ELF vmlinux file for the kexec_file_load syscall.
*
* Copyright (C) 2021 Huawei Technologies Co, Ltd.
*
* Author: Liao Chang (liaochang1@huawei.com)
*
* Based on kexec-tools' kexec-elf-riscv.c, heavily modified
* for kernel.
*/
#define pr_fmt(fmt) "kexec_image: " fmt
#include <linux/elf.h>
#include <linux/kexec.h>
#include <linux/slab.h>
#include <linux/of.h>
#include <linux/libfdt.h>
#include <linux/types.h>
#include <linux/memblock.h>
#include <asm/setup.h>
static int riscv_kexec_elf_load(struct kimage *image, struct elfhdr *ehdr,
struct kexec_elf_info *elf_info, unsigned long old_pbase,
unsigned long new_pbase)
{
int i;
int ret = 0;
size_t size;
struct kexec_buf kbuf;
const struct elf_phdr *phdr;
kbuf.image = image;
for (i = 0; i < ehdr->e_phnum; i++) {
phdr = &elf_info->proghdrs[i];
if (phdr->p_type != PT_LOAD)
continue;
size = phdr->p_filesz;
if (size > phdr->p_memsz)
size = phdr->p_memsz;
kbuf.buffer = (void *) elf_info->buffer + phdr->p_offset;
kbuf.bufsz = size;
kbuf.buf_align = phdr->p_align;
kbuf.mem = phdr->p_paddr - old_pbase + new_pbase;
kbuf.memsz = phdr->p_memsz;
kbuf.top_down = false;
ret = kexec_add_buffer(&kbuf);
if (ret)
break;
}
return ret;
}
/*
* Go through the available phsyical memory regions and find one that hold
* an image of the specified size.
*/
static int elf_find_pbase(struct kimage *image, unsigned long kernel_len,
struct elfhdr *ehdr, struct kexec_elf_info *elf_info,
unsigned long *old_pbase, unsigned long *new_pbase)
{
int i;
int ret;
struct kexec_buf kbuf;
const struct elf_phdr *phdr;
unsigned long lowest_paddr = ULONG_MAX;
unsigned long lowest_vaddr = ULONG_MAX;
for (i = 0; i < ehdr->e_phnum; i++) {
phdr = &elf_info->proghdrs[i];
if (phdr->p_type != PT_LOAD)
continue;
if (lowest_paddr > phdr->p_paddr)
lowest_paddr = phdr->p_paddr;
if (lowest_vaddr > phdr->p_vaddr)
lowest_vaddr = phdr->p_vaddr;
}
kbuf.image = image;
kbuf.buf_min = lowest_paddr;
kbuf.buf_max = ULONG_MAX;
/*
* Current riscv boot protocol requires 2MB alignment for
* RV64 and 4MB alignment for RV32
*
*/
kbuf.buf_align = PMD_SIZE;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
kbuf.memsz = ALIGN(kernel_len, PAGE_SIZE);
kbuf.top_down = false;
ret = arch_kexec_locate_mem_hole(&kbuf);
if (!ret) {
*old_pbase = lowest_paddr;
*new_pbase = kbuf.mem;
image->start = ehdr->e_entry - lowest_vaddr + kbuf.mem;
}
return ret;
}
static void *elf_kexec_load(struct kimage *image, char *kernel_buf,
unsigned long kernel_len, char *initrd,
unsigned long initrd_len, char *cmdline,
unsigned long cmdline_len)
{
int ret;
unsigned long old_kernel_pbase = ULONG_MAX;
unsigned long new_kernel_pbase = 0UL;
struct elfhdr ehdr;
struct kexec_elf_info elf_info;
ret = kexec_build_elf_info(kernel_buf, kernel_len, &ehdr, &elf_info);
if (ret)
return ERR_PTR(ret);
ret = elf_find_pbase(image, kernel_len, &ehdr, &elf_info,
&old_kernel_pbase, &new_kernel_pbase);
if (ret)
goto out;
/* Add the kernel binary to the image */
ret = riscv_kexec_elf_load(image, &ehdr, &elf_info,
old_kernel_pbase, new_kernel_pbase);
if (ret)
goto out;
ret = load_extra_segments(image, image->start, kernel_len,
initrd, initrd_len, cmdline, cmdline_len);
out:
kexec_free_elf_info(&elf_info);
return ret ? ERR_PTR(ret) : NULL;
}
const struct kexec_file_ops elf_kexec_ops = {
.probe = kexec_elf_probe,
.load = elf_kexec_load,
};

96
arch/riscv/kernel/kexec_image.c Normal file
View file

@ -0,0 +1,96 @@
// SPDX-License-Identifier: GPL-2.0
/*
* RISC-V Kexec image loader
*
*/
#define pr_fmt(fmt) "kexec_file(Image): " fmt
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/kernel.h>
#include <linux/kexec.h>
#include <linux/pe.h>
#include <linux/string.h>
#include <asm/byteorder.h>
#include <asm/image.h>
static int image_probe(const char *kernel_buf, unsigned long kernel_len)
{
const struct riscv_image_header *h = (const struct riscv_image_header *)kernel_buf;
if (!h || kernel_len < sizeof(*h))
return -EINVAL;
/* According to Documentation/riscv/boot-image-header.rst,
* use "magic2" field to check when version >= 0.2.
*/
if (h->version >= RISCV_HEADER_VERSION &&
memcmp(&h->magic2, RISCV_IMAGE_MAGIC2, sizeof(h->magic2)))
return -EINVAL;
return 0;
}
static void *image_load(struct kimage *image,
char *kernel, unsigned long kernel_len,
char *initrd, unsigned long initrd_len,
char *cmdline, unsigned long cmdline_len)
{
struct riscv_image_header *h;
u64 flags;
bool be_image, be_kernel;
struct kexec_buf kbuf;
int ret;
/* Check Image header */
h = (struct riscv_image_header *)kernel;
if (!h->image_size) {
ret = -EINVAL;
goto out;
}
/* Check endianness */
flags = le64_to_cpu(h->flags);
be_image = riscv_image_flag_field(flags, RISCV_IMAGE_FLAG_BE);
be_kernel = IS_ENABLED(CONFIG_CPU_BIG_ENDIAN);
if (be_image != be_kernel) {
ret = -EINVAL;
goto out;
}
/* Load the kernel image */
kbuf.image = image;
kbuf.buf_min = 0;
kbuf.buf_max = ULONG_MAX;
kbuf.top_down = false;
kbuf.buffer = kernel;
kbuf.bufsz = kernel_len;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
kbuf.memsz = le64_to_cpu(h->image_size);
kbuf.buf_align = le64_to_cpu(h->text_offset);
ret = kexec_add_buffer(&kbuf);
if (ret) {
pr_err("Error add kernel image ret=%d\n", ret);
goto out;
}
image->start = kbuf.mem;
pr_info("Loaded kernel at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
kbuf.mem, kbuf.bufsz, kbuf.memsz);
ret = load_extra_segments(image, kbuf.mem, kbuf.memsz,
initrd, initrd_len, cmdline, cmdline_len);
out:
return ret ? ERR_PTR(ret) : NULL;
}
const struct kexec_file_ops image_kexec_ops = {
.probe = image_probe,
.load = image_load,
};

361
arch/riscv/kernel/machine_kexec_file.c
View file

@ -7,8 +7,369 @@
* Author: Liao Chang (liaochang1@huawei.com)
*/
#include <linux/kexec.h>
#include <linux/elf.h>
#include <linux/slab.h>
#include <linux/of.h>
#include <linux/libfdt.h>
#include <linux/types.h>
#include <linux/memblock.h>
#include <linux/vmalloc.h>
#include <asm/setup.h>
const struct kexec_file_ops * const kexec_file_loaders[] = {
&elf_kexec_ops,
&image_kexec_ops,
NULL
};
int arch_kimage_file_post_load_cleanup(struct kimage *image)
{
kvfree(image->arch.fdt);
image->arch.fdt = NULL;
vfree(image->elf_headers);
image->elf_headers = NULL;
image->elf_headers_sz = 0;
return kexec_image_post_load_cleanup_default(image);
}
#ifdef CONFIG_CRASH_DUMP
static int get_nr_ram_ranges_callback(struct resource *res, void *arg)
{
unsigned int *nr_ranges = arg;
(*nr_ranges)++;
return 0;
}
static int prepare_elf64_ram_headers_callback(struct resource *res, void *arg)
{
struct crash_mem *cmem = arg;
cmem->ranges[cmem->nr_ranges].start = res->start;
cmem->ranges[cmem->nr_ranges].end = res->end;
cmem->nr_ranges++;
return 0;
}
static int prepare_elf_headers(void **addr, unsigned long *sz)
{
struct crash_mem *cmem;
unsigned int nr_ranges;
int ret;
nr_ranges = 1; /* For exclusion of crashkernel region */
walk_system_ram_res(0, -1, &nr_ranges, get_nr_ram_ranges_callback);
cmem = kmalloc(struct_size(cmem, ranges, nr_ranges), GFP_KERNEL);
if (!cmem)
return -ENOMEM;
cmem->max_nr_ranges = nr_ranges;
cmem->nr_ranges = 0;
ret = walk_system_ram_res(0, -1, cmem, prepare_elf64_ram_headers_callback);
if (ret)
goto out;
/* Exclude crashkernel region */
ret = crash_exclude_mem_range(cmem, crashk_res.start, crashk_res.end);
if (!ret)
ret = crash_prepare_elf64_headers(cmem, true, addr, sz);
out:
kfree(cmem);
return ret;
}
static char *setup_kdump_cmdline(struct kimage *image, char *cmdline,
unsigned long cmdline_len)
{
int elfcorehdr_strlen;
char *cmdline_ptr;
cmdline_ptr = kzalloc(COMMAND_LINE_SIZE, GFP_KERNEL);
if (!cmdline_ptr)
return NULL;
elfcorehdr_strlen = sprintf(cmdline_ptr, "elfcorehdr=0x%lx ",
image->elf_load_addr);
if (elfcorehdr_strlen + cmdline_len > COMMAND_LINE_SIZE) {
pr_err("Appending elfcorehdr=<addr> exceeds cmdline size\n");
kfree(cmdline_ptr);
return NULL;
}
memcpy(cmdline_ptr + elfcorehdr_strlen, cmdline, cmdline_len);
/* Ensure it's nul terminated */
cmdline_ptr[COMMAND_LINE_SIZE - 1] = '\0';
return cmdline_ptr;
}
#endif
#define RV_X(x, s, n) (((x) >> (s)) & ((1 << (n)) - 1))
#define RISCV_IMM_BITS 12
#define RISCV_IMM_REACH (1LL << RISCV_IMM_BITS)
#define RISCV_CONST_HIGH_PART(x) \
(((x) + (RISCV_IMM_REACH >> 1)) & ~(RISCV_IMM_REACH - 1))
#define RISCV_CONST_LOW_PART(x) ((x) - RISCV_CONST_HIGH_PART(x))
#define ENCODE_ITYPE_IMM(x) \
(RV_X(x, 0, 12) << 20)
#define ENCODE_BTYPE_IMM(x) \
((RV_X(x, 1, 4) << 8) | (RV_X(x, 5, 6) << 25) | \
(RV_X(x, 11, 1) << 7) | (RV_X(x, 12, 1) << 31))
#define ENCODE_UTYPE_IMM(x) \
(RV_X(x, 12, 20) << 12)
#define ENCODE_JTYPE_IMM(x) \
((RV_X(x, 1, 10) << 21) | (RV_X(x, 11, 1) << 20) | \
(RV_X(x, 12, 8) << 12) | (RV_X(x, 20, 1) << 31))
#define ENCODE_CBTYPE_IMM(x) \
((RV_X(x, 1, 2) << 3) | (RV_X(x, 3, 2) << 10) | (RV_X(x, 5, 1) << 2) | \
(RV_X(x, 6, 2) << 5) | (RV_X(x, 8, 1) << 12))
#define ENCODE_CJTYPE_IMM(x) \
((RV_X(x, 1, 3) << 3) | (RV_X(x, 4, 1) << 11) | (RV_X(x, 5, 1) << 2) | \
(RV_X(x, 6, 1) << 7) | (RV_X(x, 7, 1) << 6) | (RV_X(x, 8, 2) << 9) | \
(RV_X(x, 10, 1) << 8) | (RV_X(x, 11, 1) << 12))
#define ENCODE_UJTYPE_IMM(x) \
(ENCODE_UTYPE_IMM(RISCV_CONST_HIGH_PART(x)) | \
(ENCODE_ITYPE_IMM(RISCV_CONST_LOW_PART(x)) << 32))
#define ENCODE_UITYPE_IMM(x) \
(ENCODE_UTYPE_IMM(x) | (ENCODE_ITYPE_IMM(x) << 32))
#define CLEAN_IMM(type, x) \
((~ENCODE_##type##_IMM((uint64_t)(-1))) & (x))
int arch_kexec_apply_relocations_add(struct purgatory_info *pi,
Elf_Shdr *section,
const Elf_Shdr *relsec,
const Elf_Shdr *symtab)
{
const char *strtab, *name, *shstrtab;
const Elf_Shdr *sechdrs;
Elf64_Rela *relas;
int i, r_type;
/* String & section header string table */
sechdrs = (void *)pi->ehdr + pi->ehdr->e_shoff;
strtab = (char *)pi->ehdr + sechdrs[symtab->sh_link].sh_offset;
shstrtab = (char *)pi->ehdr + sechdrs[pi->ehdr->e_shstrndx].sh_offset;
relas = (void *)pi->ehdr + relsec->sh_offset;
for (i = 0; i < relsec->sh_size / sizeof(*relas); i++) {
const Elf_Sym *sym; /* symbol to relocate */
unsigned long addr; /* final location after relocation */
unsigned long val; /* relocated symbol value */
unsigned long sec_base; /* relocated symbol value */
void *loc; /* tmp location to modify */
sym = (void *)pi->ehdr + symtab->sh_offset;
sym += ELF64_R_SYM(relas[i].r_info);
if (sym->st_name)
name = strtab + sym->st_name;
else
name = shstrtab + sechdrs[sym->st_shndx].sh_name;
loc = pi->purgatory_buf;
loc += section->sh_offset;
loc += relas[i].r_offset;
if (sym->st_shndx == SHN_ABS)
sec_base = 0;
else if (sym->st_shndx >= pi->ehdr->e_shnum) {
pr_err("Invalid section %d for symbol %s\n",
sym->st_shndx, name);
return -ENOEXEC;
} else
sec_base = pi->sechdrs[sym->st_shndx].sh_addr;
val = sym->st_value;
val += sec_base;
val += relas[i].r_addend;
addr = section->sh_addr + relas[i].r_offset;
r_type = ELF64_R_TYPE(relas[i].r_info);
switch (r_type) {
case R_RISCV_BRANCH:
*(u32 *)loc = CLEAN_IMM(BTYPE, *(u32 *)loc) |
ENCODE_BTYPE_IMM(val - addr);
break;
case R_RISCV_JAL:
*(u32 *)loc = CLEAN_IMM(JTYPE, *(u32 *)loc) |
ENCODE_JTYPE_IMM(val - addr);
break;
/*
* With no R_RISCV_PCREL_LO12_S, R_RISCV_PCREL_LO12_I
* sym is expected to be next to R_RISCV_PCREL_HI20
* in purgatory relsec. Handle it like R_RISCV_CALL
* sym, instead of searching the whole relsec.
*/
case R_RISCV_PCREL_HI20:
case R_RISCV_CALL_PLT:
case R_RISCV_CALL:
*(u64 *)loc = CLEAN_IMM(UITYPE, *(u64 *)loc) |
ENCODE_UJTYPE_IMM(val - addr);
break;
case R_RISCV_RVC_BRANCH:
*(u32 *)loc = CLEAN_IMM(CBTYPE, *(u32 *)loc) |
ENCODE_CBTYPE_IMM(val - addr);
break;
case R_RISCV_RVC_JUMP:
*(u32 *)loc = CLEAN_IMM(CJTYPE, *(u32 *)loc) |
ENCODE_CJTYPE_IMM(val - addr);
break;
case R_RISCV_ADD16:
*(u16 *)loc += val;
break;
case R_RISCV_SUB16:
*(u16 *)loc -= val;
break;
case R_RISCV_ADD32:
*(u32 *)loc += val;
break;
case R_RISCV_SUB32:
*(u32 *)loc -= val;
break;
/* It has been applied by R_RISCV_PCREL_HI20 sym */
case R_RISCV_PCREL_LO12_I:
case R_RISCV_ALIGN:
case R_RISCV_RELAX:
break;
case R_RISCV_64:
*(u64 *)loc = val;
break;
default:
pr_err("Unknown rela relocation: %d\n", r_type);
return -ENOEXEC;
}
}
return 0;
}
int load_extra_segments(struct kimage *image, unsigned long kernel_start,
unsigned long kernel_len, char *initrd,
unsigned long initrd_len, char *cmdline,
unsigned long cmdline_len)
{
int ret;
void *fdt;
unsigned long initrd_pbase = 0UL;
struct kexec_buf kbuf;
char *modified_cmdline = NULL;
kbuf.image = image;
kbuf.buf_min = kernel_start + kernel_len;
kbuf.buf_max = ULONG_MAX;
#ifdef CONFIG_CRASH_DUMP
/* Add elfcorehdr */
if (image->type == KEXEC_TYPE_CRASH) {
void *headers;
unsigned long headers_sz;
ret = prepare_elf_headers(&headers, &headers_sz);
if (ret) {
pr_err("Preparing elf core header failed\n");
goto out;
}
kbuf.buffer = headers;
kbuf.bufsz = headers_sz;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
kbuf.memsz = headers_sz;
kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
kbuf.top_down = true;
ret = kexec_add_buffer(&kbuf);
if (ret) {
vfree(headers);
goto out;
}
image->elf_headers = headers;
image->elf_load_addr = kbuf.mem;
image->elf_headers_sz = headers_sz;
kexec_dprintk("Loaded elf core header at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
image->elf_load_addr, kbuf.bufsz, kbuf.memsz);
/* Setup cmdline for kdump kernel case */
modified_cmdline = setup_kdump_cmdline(image, cmdline,
cmdline_len);
if (!modified_cmdline) {
pr_err("Setting up cmdline for kdump kernel failed\n");
ret = -EINVAL;
goto out;
}
cmdline = modified_cmdline;
}
#endif
#ifdef CONFIG_ARCH_SUPPORTS_KEXEC_PURGATORY
/* Add purgatory to the image */
kbuf.top_down = true;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
ret = kexec_load_purgatory(image, &kbuf);
if (ret) {
pr_err("Error loading purgatory ret=%d\n", ret);
goto out;
}
kexec_dprintk("Loaded purgatory at 0x%lx\n", kbuf.mem);
ret = kexec_purgatory_get_set_symbol(image, "riscv_kernel_entry",
&kernel_start,
sizeof(kernel_start), 0);
if (ret)
pr_err("Error update purgatory ret=%d\n", ret);
#endif /* CONFIG_ARCH_SUPPORTS_KEXEC_PURGATORY */
/* Add the initrd to the image */
if (initrd != NULL) {
kbuf.buffer = initrd;
kbuf.bufsz = kbuf.memsz = initrd_len;
kbuf.buf_align = PAGE_SIZE;
kbuf.top_down = true;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
ret = kexec_add_buffer(&kbuf);
if (ret)
goto out;
initrd_pbase = kbuf.mem;
kexec_dprintk("Loaded initrd at 0x%lx\n", initrd_pbase);
}
/* Add the DTB to the image */
fdt = of_kexec_alloc_and_setup_fdt(image, initrd_pbase,
initrd_len, cmdline, 0);
if (!fdt) {
pr_err("Error setting up the new device tree.\n");
ret = -EINVAL;
goto out;
}
fdt_pack(fdt);
kbuf.buffer = fdt;
kbuf.bufsz = kbuf.memsz = fdt_totalsize(fdt);
kbuf.buf_align = PAGE_SIZE;
kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
kbuf.top_down = true;
ret = kexec_add_buffer(&kbuf);
if (ret) {
pr_err("Error add DTB kbuf ret=%d\n", ret);
goto out_free_fdt;
}
/* Cache the fdt buffer address for memory cleanup */
image->arch.fdt = fdt;
kexec_dprintk("Loaded device tree at 0x%lx\n", kbuf.mem);
goto out;
out_free_fdt:
kvfree(fdt);
out:
kfree(modified_cmdline);
return ret;
}

117
arch/riscv/kernel/mcount-dyn.S
View file

@ -13,7 +13,6 @@
.text
#define FENTRY_RA_OFFSET 8
#define ABI_SIZE_ON_STACK 80
#define ABI_A0 0
#define ABI_A1 8
@ -56,16 +55,13 @@
addi sp, sp, ABI_SIZE_ON_STACK
.endm
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
/**
* SAVE_ABI_REGS - save regs against the ftrace_regs struct
*
* After the stack is established,
*
* 0(sp) stores the PC of the traced function which can be accessed
* by &(fregs)->epc in tracing function. Note that the real
* function entry address should be computed with -FENTRY_RA_OFFSET.
* by &(fregs)->epc in tracing function.
*
* 8(sp) stores the function return address (i.e. parent IP) that
* can be accessed by &(fregs)->ra in tracing function.
@ -86,17 +82,20 @@
* +++++++++
**/
.macro SAVE_ABI_REGS
mv t4, sp // Save original SP in T4
addi sp, sp, -FREGS_SIZE_ON_STACK
REG_S t0, FREGS_EPC(sp)
REG_S x1, FREGS_RA(sp)
REG_S t4, FREGS_SP(sp) // Put original SP on stack
#ifdef HAVE_FUNCTION_GRAPH_FP_TEST
REG_S x8, FREGS_S0(sp)
#endif
REG_S x6, FREGS_T1(sp)
#ifdef CONFIG_CC_IS_CLANG
REG_S x7, FREGS_T2(sp)
REG_S x28, FREGS_T3(sp)
REG_S x29, FREGS_T4(sp)
REG_S x30, FREGS_T5(sp)
REG_S x31, FREGS_T6(sp)
#endif
// save the arguments
REG_S x10, FREGS_A0(sp)
REG_S x11, FREGS_A1(sp)
@ -106,16 +105,25 @@
REG_S x15, FREGS_A5(sp)
REG_S x16, FREGS_A6(sp)
REG_S x17, FREGS_A7(sp)
mv a0, sp
addi a0, a0, FREGS_SIZE_ON_STACK
REG_S a0, FREGS_SP(sp) // Put original SP on stack
.endm
.macro RESTORE_ABI_REGS, all=0
.macro RESTORE_ABI_REGS
REG_L t0, FREGS_EPC(sp)
REG_L x1, FREGS_RA(sp)
#ifdef HAVE_FUNCTION_GRAPH_FP_TEST
REG_L x8, FREGS_S0(sp)
#endif
REG_L x6, FREGS_T1(sp)
#ifdef CONFIG_CC_IS_CLANG
REG_L x7, FREGS_T2(sp)
REG_L x28, FREGS_T3(sp)
REG_L x29, FREGS_T4(sp)
REG_L x30, FREGS_T5(sp)
REG_L x31, FREGS_T6(sp)
#endif
// restore the arguments
REG_L x10, FREGS_A0(sp)
REG_L x11, FREGS_A1(sp)
@ -130,60 +138,71 @@
.endm
.macro PREPARE_ARGS
addi a0, t0, -FENTRY_RA_OFFSET
addi a0, t0, -MCOUNT_JALR_SIZE // ip (callsite's jalr insn)
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
mv a1, ra // parent_ip
REG_L a2, -16(t0) // op
REG_L ra, FTRACE_OPS_FUNC(a2) // op->func
#else
la a1, function_trace_op
REG_L a2, 0(a1)
mv a1, ra
mv a3, sp
REG_L a2, 0(a1) // op
mv a1, ra // parent_ip
#endif
mv a3, sp // regs
.endm
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
#ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
SYM_FUNC_START(ftrace_caller)
SAVE_ABI
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
/*
* When CALL_OPS is enabled (2 or 4) nops [8B] are placed before the
* function entry, these are later overwritten with the pointer to the
* associated struct ftrace_ops.
*
* -8: &ftrace_ops of the associated tracer function.
*<ftrace enable>:
* 0: auipc t0/ra, 0x?
* 4: jalr t0/ra, ?(t0/ra)
*
* -8: &ftrace_nop_ops
*<ftrace disable>:
* 0: nop
* 4: nop
*
* t0 is set to ip+8 after the jalr is executed at the callsite,
* so we find the associated op at t0-16.
*/
REG_L t1, -16(t0) // op Should be SZ_REG instead of 16
addi a0, t0, -FENTRY_RA_OFFSET
la a1, function_trace_op
REG_L a2, 0(a1)
mv a1, ra
mv a3, sp
SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
call ftrace_stub
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
addi a0, sp, ABI_RA
REG_L a1, ABI_T0(sp)
addi a1, a1, -FENTRY_RA_OFFSET
#ifdef HAVE_FUNCTION_GRAPH_FP_TEST
mv a2, s0
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
/*
* If the op has a direct call, handle it immediately without
* saving/restoring registers.
*/
REG_L t1, FTRACE_OPS_DIRECT_CALL(t1)
bnez t1, ftrace_caller_direct
#endif
SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL)
call ftrace_stub
#endif
RESTORE_ABI
jr t0
SYM_FUNC_END(ftrace_caller)
#else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
SYM_FUNC_START(ftrace_caller)
mv t1, zero
SAVE_ABI_REGS
PREPARE_ARGS
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
jalr ra
#else
SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
call ftrace_stub
REG_L ra, ftrace_call_dest
jalr ra, 0(ra)
#endif
RESTORE_ABI_REGS
bnez t1, .Ldirect
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
bnez t1, ftrace_caller_direct
#endif
jr t0
.Ldirect:
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
SYM_INNER_LABEL(ftrace_caller_direct, SYM_L_LOCAL)
jr t1
#endif
SYM_FUNC_END(ftrace_caller)
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
SYM_CODE_START(ftrace_stub_direct_tramp)
jr t0

77
arch/riscv/kernel/module-sections.c
View file

@ -9,6 +9,7 @@
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/moduleloader.h>
#include <linux/sort.h>
unsigned long module_emit_got_entry(struct module *mod, unsigned long val)
{
@ -55,44 +56,70 @@ unsigned long module_emit_plt_entry(struct module *mod, unsigned long val)
return (unsigned long)&plt[i];
}
static int is_rela_equal(const Elf_Rela *x, const Elf_Rela *y)
#define cmp_3way(a, b) ((a) < (b) ? -1 : (a) > (b))
static int cmp_rela(const void *a, const void *b)
{
return x->r_info == y->r_info && x->r_addend == y->r_addend;
const Elf_Rela *x = a, *y = b;
int i;
/* sort by type, symbol index and addend */
i = cmp_3way(x->r_info, y->r_info);
if (i == 0)
i = cmp_3way(x->r_addend, y->r_addend);
return i;
}
static bool duplicate_rela(const Elf_Rela *rela, int idx)
{
int i;
for (i = 0; i < idx; i++) {
if (is_rela_equal(&rela[i], &rela[idx]))
return true;
}
return false;
/*
* Entries are sorted by type, symbol index and addend. That means
* that, if a duplicate entry exists, it must be in the preceding slot.
*/
return idx > 0 && cmp_rela(rela + idx, rela + idx - 1) == 0;
}
static void count_max_entries(Elf_Rela *relas, int num,
static void count_max_entries(const Elf_Rela *relas, size_t num,
unsigned int *plts, unsigned int *gots)
{
for (int i = 0; i < num; i++) {
for (size_t i = 0; i < num; i++) {
if (duplicate_rela(relas, i))
continue;
switch (ELF_R_TYPE(relas[i].r_info)) {
case R_RISCV_CALL_PLT:
case R_RISCV_PLT32:
if (!duplicate_rela(relas, i))
(*plts)++;
break;
case R_RISCV_GOT_HI20:
if (!duplicate_rela(relas, i))
(*gots)++;
break;
default:
unreachable();
}
}
}
static bool rela_needs_plt_got_entry(const Elf_Rela *rela)
{
switch (ELF_R_TYPE(rela->r_info)) {
case R_RISCV_CALL_PLT:
case R_RISCV_GOT_HI20:
case R_RISCV_PLT32:
return true;
default:
return false;
}
}
int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
char *secstrings, struct module *mod)
{
size_t num_scratch_relas = 0;
unsigned int num_plts = 0;
unsigned int num_gots = 0;
Elf_Rela *scratch = NULL;
size_t scratch_size = 0;
int i;
/*
@ -122,9 +149,10 @@ int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
/* Calculate the maxinum number of entries */
for (i = 0; i < ehdr->e_shnum; i++) {
size_t num_relas = sechdrs[i].sh_size / sizeof(Elf_Rela);
Elf_Rela *relas = (void *)ehdr + sechdrs[i].sh_offset;
int num_rela = sechdrs[i].sh_size / sizeof(Elf_Rela);
Elf_Shdr *dst_sec = sechdrs + sechdrs[i].sh_info;
size_t scratch_size_needed;
if (sechdrs[i].sh_type != SHT_RELA)
continue;
@ -133,7 +161,28 @@ int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
if (!(dst_sec->sh_flags & SHF_EXECINSTR))
continue;
count_max_entries(relas, num_rela, &num_plts, &num_gots);
/*
* apply_relocate_add() relies on HI20 and LO12 relocation pairs being
* close together, so sort a copy of the section to avoid interfering.
*/
scratch_size_needed = (num_scratch_relas + num_relas) * sizeof(*scratch);
if (scratch_size_needed > scratch_size) {
scratch_size = scratch_size_needed;
scratch = kvrealloc(scratch, scratch_size, GFP_KERNEL);
if (!scratch)
return -ENOMEM;
}
for (size_t j = 0; j < num_relas; j++)
if (rela_needs_plt_got_entry(&relas[j]))
scratch[num_scratch_relas++] = relas[j];
}
if (scratch) {
/* sort the accumulated PLT/GOT relocations so duplicates are adjacent */
sort(scratch, num_scratch_relas, sizeof(*scratch), cmp_rela, NULL);
count_max_entries(scratch, num_scratch_relas, &num_plts, &num_gots);
kvfree(scratch);
}
mod->arch.plt.shdr->sh_type = SHT_NOBITS;

2
arch/riscv/kernel/process.c
View file

@ -60,7 +60,7 @@ int get_unalign_ctl(struct task_struct *tsk, unsigned long adr)
if (!unaligned_ctl_available())
return -EINVAL;
return put_user(tsk->thread.align_ctl, (unsigned long __user *)adr);
return put_user(tsk->thread.align_ctl, (unsigned int __user *)adr);
}
void __show_regs(struct pt_regs *regs)

81
arch/riscv/kernel/sbi.c
View file

@ -299,6 +299,76 @@ static int __sbi_rfence_v02(int fid, const struct cpumask *cpu_mask,
return 0;
}
static bool sbi_fwft_supported;
struct fwft_set_req {
u32 feature;
unsigned long value;
unsigned long flags;
atomic_t error;
};
static void cpu_sbi_fwft_set(void *arg)
{
struct fwft_set_req *req = arg;
int ret;
ret = sbi_fwft_set(req->feature, req->value, req->flags);
if (ret)
atomic_set(&req->error, ret);
}
/**
* sbi_fwft_set() - Set a feature on the local hart
* @feature: The feature ID to be set
* @value: The feature value to be set
* @flags: FWFT feature set flags
*
* Return: 0 on success, appropriate linux error code otherwise.
*/
int sbi_fwft_set(u32 feature, unsigned long value, unsigned long flags)
{
struct sbiret ret;
if (!sbi_fwft_supported)
return -EOPNOTSUPP;
ret = sbi_ecall(SBI_EXT_FWFT, SBI_EXT_FWFT_SET,
feature, value, flags, 0, 0, 0);
return sbi_err_map_linux_errno(ret.error);
}
/**
* sbi_fwft_set_cpumask() - Set a feature for the specified cpumask
* @mask: CPU mask of cpus that need the feature to be set
* @feature: The feature ID to be set
* @value: The feature value to be set
* @flags: FWFT feature set flags
*
* Return: 0 on success, appropriate linux error code otherwise.
*/
int sbi_fwft_set_cpumask(const cpumask_t *mask, u32 feature,
unsigned long value, unsigned long flags)
{
struct fwft_set_req req = {
.feature = feature,
.value = value,
.flags = flags,
.error = ATOMIC_INIT(0),
};
if (!sbi_fwft_supported)
return -EOPNOTSUPP;
if (feature & SBI_FWFT_GLOBAL_FEATURE_BIT)
return -EINVAL;
on_each_cpu_mask(mask, cpu_sbi_fwft_set, &req, 1);
return atomic_read(&req.error);
}
/**
* sbi_set_timer() - Program the timer for next timer event.
* @stime_value: The value after which next timer event should fire.
@ -609,7 +679,7 @@ void __init sbi_init(void)
} else {
__sbi_rfence = __sbi_rfence_v01;
}
if ((sbi_spec_version >= sbi_mk_version(0, 3)) &&
if (sbi_spec_version >= sbi_mk_version(0, 3) &&
sbi_probe_extension(SBI_EXT_SRST)) {
pr_info("SBI SRST extension detected\n");
pm_power_off = sbi_srst_power_off;
@ -617,11 +687,16 @@ void __init sbi_init(void)
sbi_srst_reboot_nb.priority = 192;
register_restart_handler(&sbi_srst_reboot_nb);
}
if ((sbi_spec_version >= sbi_mk_version(2, 0)) &&
(sbi_probe_extension(SBI_EXT_DBCN) > 0)) {
if (sbi_spec_version >= sbi_mk_version(2, 0) &&
sbi_probe_extension(SBI_EXT_DBCN) > 0) {
pr_info("SBI DBCN extension detected\n");
sbi_debug_console_available = true;
}
if (sbi_spec_version >= sbi_mk_version(3, 0) &&
sbi_probe_extension(SBI_EXT_FWFT)) {
pr_info("SBI FWFT extension detected\n");
sbi_fwft_supported = true;
}
} else {
__sbi_set_timer = __sbi_set_timer_v01;
__sbi_send_ipi = __sbi_send_ipi_v01;

6
arch/riscv/kernel/sys_hwprobe.c
View file

@ -15,6 +15,7 @@
#include <asm/uaccess.h>
#include <asm/unistd.h>
#include <asm/vector.h>
#include <asm/vendor_extensions/sifive_hwprobe.h>
#include <asm/vendor_extensions/thead_hwprobe.h>
#include <vdso/vsyscall.h>
@ -96,6 +97,7 @@ static void hwprobe_isa_ext0(struct riscv_hwprobe *pair,
* presence in the hart_isa bitmap, are made.
*/
EXT_KEY(ZAAMO);
EXT_KEY(ZABHA);
EXT_KEY(ZACAS);
EXT_KEY(ZALRSC);
EXT_KEY(ZAWRS);
@ -300,6 +302,10 @@ static void hwprobe_one_pair(struct riscv_hwprobe *pair,
pair->value = riscv_timebase;
break;
case RISCV_HWPROBE_KEY_VENDOR_EXT_SIFIVE_0:
hwprobe_isa_vendor_ext_sifive_0(pair, cpus);
break;
case RISCV_HWPROBE_KEY_VENDOR_EXT_THEAD_0:
hwprobe_isa_vendor_ext_thead_0(pair, cpus);
break;

114
arch/riscv/kernel/traps_misaligned.c
View file

@ -16,6 +16,7 @@
#include <asm/entry-common.h>
#include <asm/hwprobe.h>
#include <asm/cpufeature.h>
#include <asm/sbi.h>
#include <asm/vector.h>
#define INSN_MATCH_LB 0x3
@ -368,9 +369,7 @@ static int handle_scalar_misaligned_load(struct pt_regs *regs)
perf_sw_event(PERF_COUNT_SW_ALIGNMENT_FAULTS, 1, regs, addr);
#ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS
*this_cpu_ptr(&misaligned_access_speed) = RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED;
#endif
if (!unaligned_enabled)
return -1;
@ -455,7 +454,7 @@ static int handle_scalar_misaligned_load(struct pt_regs *regs)
val.data_u64 = 0;
if (user_mode(regs)) {
if (copy_from_user(&val, (u8 __user *)addr, len))
if (copy_from_user_nofault(&val, (u8 __user *)addr, len))
return -1;
} else {
memcpy(&val, (u8 *)addr, len);
@ -556,7 +555,7 @@ static int handle_scalar_misaligned_store(struct pt_regs *regs)
return -EOPNOTSUPP;
if (user_mode(regs)) {
if (copy_to_user((u8 __user *)addr, &val, len))
if (copy_to_user_nofault((u8 __user *)addr, &val, len))
return -1;
} else {
memcpy((u8 *)addr, &val, len);
@ -626,6 +625,10 @@ bool __init check_vector_unaligned_access_emulated_all_cpus(void)
{
int cpu;
/*
* While being documented as very slow, schedule_on_each_cpu() is used since
* kernel_vector_begin() expects irqs to be enabled or it will panic()
*/
schedule_on_each_cpu(check_vector_unaligned_access_emulated);
for_each_online_cpu(cpu)
@ -642,11 +645,23 @@ bool __init check_vector_unaligned_access_emulated_all_cpus(void)
}
#endif
static bool all_cpus_unaligned_scalar_access_emulated(void)
{
int cpu;
for_each_online_cpu(cpu)
if (per_cpu(misaligned_access_speed, cpu) !=
RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED)
return false;
return true;
}
#ifdef CONFIG_RISCV_SCALAR_MISALIGNED
static bool unaligned_ctl __read_mostly;
void check_unaligned_access_emulated(struct work_struct *work __always_unused)
static void check_unaligned_access_emulated(void *arg __always_unused)
{
int cpu = smp_processor_id();
long *mas_ptr = per_cpu_ptr(&misaligned_access_speed, cpu);
@ -657,6 +672,13 @@ void check_unaligned_access_emulated(struct work_struct *work __always_unused)
__asm__ __volatile__ (
" "REG_L" %[tmp], 1(%[ptr])\n"
: [tmp] "=r" (tmp_val) : [ptr] "r" (&tmp_var) : "memory");
}
static int cpu_online_check_unaligned_access_emulated(unsigned int cpu)
{
long *mas_ptr = per_cpu_ptr(&misaligned_access_speed, cpu);
check_unaligned_access_emulated(NULL);
/*
* If unaligned_ctl is already set, this means that we detected that all
@ -665,25 +687,22 @@ void check_unaligned_access_emulated(struct work_struct *work __always_unused)
*/
if (unlikely(unaligned_ctl && (*mas_ptr != RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED))) {
pr_crit("CPU misaligned accesses non homogeneous (expected all emulated)\n");
while (true)
cpu_relax();
return -EINVAL;
}
return 0;
}
bool __init check_unaligned_access_emulated_all_cpus(void)
{
int cpu;
/*
* We can only support PR_UNALIGN controls if all CPUs have misaligned
* accesses emulated since tasks requesting such control can run on any
* CPU.
*/
schedule_on_each_cpu(check_unaligned_access_emulated);
on_each_cpu(check_unaligned_access_emulated, NULL, 1);
for_each_online_cpu(cpu)
if (per_cpu(misaligned_access_speed, cpu)
!= RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED)
if (!all_cpus_unaligned_scalar_access_emulated())
return false;
unaligned_ctl = true;
@ -699,4 +718,73 @@ bool __init check_unaligned_access_emulated_all_cpus(void)
{
return false;
}
static int cpu_online_check_unaligned_access_emulated(unsigned int cpu)
{
return 0;
}
#endif
static bool misaligned_traps_delegated;
#ifdef CONFIG_RISCV_SBI
static int cpu_online_sbi_unaligned_setup(unsigned int cpu)
{
if (sbi_fwft_set(SBI_FWFT_MISALIGNED_EXC_DELEG, 1, 0) &&
misaligned_traps_delegated) {
pr_crit("Misaligned trap delegation non homogeneous (expected delegated)");
return -EINVAL;
}
return 0;
}
void __init unaligned_access_init(void)
{
int ret;
ret = sbi_fwft_set_online_cpus(SBI_FWFT_MISALIGNED_EXC_DELEG, 1, 0);
if (ret)
return;
misaligned_traps_delegated = true;
pr_info("SBI misaligned access exception delegation ok\n");
/*
* Note that we don't have to take any specific action here, if
* the delegation is successful, then
* check_unaligned_access_emulated() will verify that indeed the
* platform traps on misaligned accesses.
*/
}
#else
void __init unaligned_access_init(void) {}
static int cpu_online_sbi_unaligned_setup(unsigned int cpu __always_unused)
{
return 0;
}
#endif
int cpu_online_unaligned_access_init(unsigned int cpu)
{
int ret;
ret = cpu_online_sbi_unaligned_setup(cpu);
if (ret)
return ret;
return cpu_online_check_unaligned_access_emulated(cpu);
}
bool misaligned_traps_can_delegate(void)
{
/*
* Either we successfully requested misaligned traps delegation for all
* CPUs, or the SBI does not implement the FWFT extension but delegated
* the exception by default.
*/
return misaligned_traps_delegated ||
all_cpus_unaligned_scalar_access_emulated();
}
EXPORT_SYMBOL_GPL(misaligned_traps_can_delegate);

8
arch/riscv/kernel/unaligned_access_speed.c
View file

@ -236,6 +236,11 @@ arch_initcall_sync(lock_and_set_unaligned_access_static_branch);
static int riscv_online_cpu(unsigned int cpu)
{
int ret = cpu_online_unaligned_access_init(cpu);
if (ret)
return ret;
/* We are already set since the last check */
if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN) {
goto exit;
@ -248,7 +253,6 @@ static int riscv_online_cpu(unsigned int cpu)
{
static struct page *buf;
check_unaligned_access_emulated(NULL);
buf = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER);
if (!buf) {
pr_warn("Allocation failure, not measuring misaligned performance\n");
@ -439,6 +443,8 @@ static int __init check_unaligned_access_all_cpus(void)
{
int cpu;
unaligned_access_init();
if (unaligned_scalar_speed_param != RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN) {
pr_info("scalar unaligned access speed set to '%s' (%lu) by command line\n",
speed_str[unaligned_scalar_speed_param], unaligned_scalar_speed_param);

2
arch/riscv/kernel/vdso.c
View file

@ -136,7 +136,7 @@ static int __setup_additional_pages(struct mm_struct *mm,
ret =
_install_special_mapping(mm, vdso_base, vdso_text_len,
(VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC),
(VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC | VM_SEALED_SYSMAP),
vdso_info->cm);
if (IS_ERR(ret))

15
arch/riscv/kernel/vdso/Makefile
View file

@ -13,9 +13,17 @@ vdso-syms += flush_icache
vdso-syms += hwprobe
vdso-syms += sys_hwprobe
ifdef CONFIG_VDSO_GETRANDOM
vdso-syms += getrandom
endif
# Files to link into the vdso
obj-vdso = $(patsubst %, %.o, $(vdso-syms)) note.o
ifdef CONFIG_VDSO_GETRANDOM
obj-vdso += vgetrandom-chacha.o
endif
ccflags-y := -fno-stack-protector
ccflags-y += -DDISABLE_BRANCH_PROFILING
ccflags-y += -fno-builtin
@ -24,6 +32,10 @@ ifneq ($(c-gettimeofday-y),)
CFLAGS_vgettimeofday.o += -fPIC -include $(c-gettimeofday-y)
endif
ifneq ($(c-getrandom-y),)
CFLAGS_getrandom.o += -fPIC -include $(c-getrandom-y)
endif
CFLAGS_hwprobe.o += -fPIC
# Build rules
@ -38,6 +50,7 @@ endif
# Disable -pg to prevent insert call site
CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS)
CFLAGS_REMOVE_getrandom.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS)
CFLAGS_REMOVE_hwprobe.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS)
# Force dependency
@ -47,7 +60,7 @@ $(obj)/vdso.o: $(obj)/vdso.so
$(obj)/vdso.so.dbg: $(obj)/vdso.lds $(obj-vdso) FORCE
$(call if_changed,vdsold_and_check)
LDFLAGS_vdso.so.dbg = -shared -soname=linux-vdso.so.1 \
--build-id=sha1 --hash-style=both --eh-frame-hdr
--build-id=sha1 --eh-frame-hdr
# strip rule for the .so file
$(obj)/%.so: OBJCOPYFLAGS := -S

10
arch/riscv/kernel/vdso/getrandom.c Normal file
View file

@ -0,0 +1,10 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Copyright (C) 2025 Xi Ruoyao <xry111@xry111.site>. All Rights Reserved.
*/
#include <linux/types.h>
ssize_t __vdso_getrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state, size_t opaque_len)
{
return __cvdso_getrandom(buffer, len, flags, opaque_state, opaque_len);
}

3
arch/riscv/kernel/vdso/vdso.lds.S
View file

@ -79,6 +79,9 @@ VERSION
__vdso_flush_icache;
#ifndef COMPAT_VDSO
__vdso_riscv_hwprobe;
#endif
#if defined(CONFIG_VDSO_GETRANDOM) && !defined(COMPAT_VDSO)
__vdso_getrandom;
#endif
local: *;
};

249
arch/riscv/kernel/vdso/vgetrandom-chacha.S Normal file
View file

@ -0,0 +1,249 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2025 Xi Ruoyao <xry111@xry111.site>. All Rights Reserved.
*
* Based on arch/loongarch/vdso/vgetrandom-chacha.S.
*/
#include <asm/asm.h>
#include <linux/linkage.h>
.text
.macro ROTRI rd rs imm
slliw t0, \rs, 32 - \imm
srliw \rd, \rs, \imm
or \rd, \rd, t0
.endm
.macro OP_4REG op d0 d1 d2 d3 s0 s1 s2 s3
\op \d0, \d0, \s0
\op \d1, \d1, \s1
\op \d2, \d2, \s2
\op \d3, \d3, \s3
.endm
/*
* a0: output bytes
* a1: 32-byte key input
* a2: 8-byte counter input/output
* a3: number of 64-byte blocks to write to output
*/
SYM_FUNC_START(__arch_chacha20_blocks_nostack)
#define output a0
#define key a1
#define counter a2
#define nblocks a3
#define i a4
#define state0 s0
#define state1 s1
#define state2 s2
#define state3 s3
#define state4 s4
#define state5 s5
#define state6 s6
#define state7 s7
#define state8 s8
#define state9 s9
#define state10 s10
#define state11 s11
#define state12 a5
#define state13 a6
#define state14 a7
#define state15 t1
#define cnt t2
#define copy0 t3
#define copy1 t4
#define copy2 t5
#define copy3 t6
/* Packs to be used with OP_4REG */
#define line0 state0, state1, state2, state3
#define line1 state4, state5, state6, state7
#define line2 state8, state9, state10, state11
#define line3 state12, state13, state14, state15
#define line1_perm state5, state6, state7, state4
#define line2_perm state10, state11, state8, state9
#define line3_perm state15, state12, state13, state14
#define copy copy0, copy1, copy2, copy3
#define _16 16, 16, 16, 16
#define _20 20, 20, 20, 20
#define _24 24, 24, 24, 24
#define _25 25, 25, 25, 25
/*
* The ABI requires s0-s9 saved.
* This does not violate the stack-less requirement: no sensitive data
* is spilled onto the stack.
*/
addi sp, sp, -12*SZREG
REG_S s0, (sp)
REG_S s1, SZREG(sp)
REG_S s2, 2*SZREG(sp)
REG_S s3, 3*SZREG(sp)
REG_S s4, 4*SZREG(sp)
REG_S s5, 5*SZREG(sp)
REG_S s6, 6*SZREG(sp)
REG_S s7, 7*SZREG(sp)
REG_S s8, 8*SZREG(sp)
REG_S s9, 9*SZREG(sp)
REG_S s10, 10*SZREG(sp)
REG_S s11, 11*SZREG(sp)
ld cnt, (counter)
li copy0, 0x61707865
li copy1, 0x3320646e
li copy2, 0x79622d32
li copy3, 0x6b206574
.Lblock:
/* state[0,1,2,3] = "expand 32-byte k" */
mv state0, copy0
mv state1, copy1
mv state2, copy2
mv state3, copy3
/* state[4,5,..,11] = key */
lw state4, (key)
lw state5, 4(key)
lw state6, 8(key)
lw state7, 12(key)
lw state8, 16(key)
lw state9, 20(key)
lw state10, 24(key)
lw state11, 28(key)
/* state[12,13] = counter */
mv state12, cnt
srli state13, cnt, 32
/* state[14,15] = 0 */
mv state14, zero
mv state15, zero
li i, 10
.Lpermute:
/* odd round */
OP_4REG addw line0, line1
OP_4REG xor line3, line0
OP_4REG ROTRI line3, _16
OP_4REG addw line2, line3
OP_4REG xor line1, line2
OP_4REG ROTRI line1, _20
OP_4REG addw line0, line1
OP_4REG xor line3, line0
OP_4REG ROTRI line3, _24
OP_4REG addw line2, line3
OP_4REG xor line1, line2
OP_4REG ROTRI line1, _25
/* even round */
OP_4REG addw line0, line1_perm
OP_4REG xor line3_perm, line0
OP_4REG ROTRI line3_perm, _16
OP_4REG addw line2_perm, line3_perm
OP_4REG xor line1_perm, line2_perm
OP_4REG ROTRI line1_perm, _20
OP_4REG addw line0, line1_perm
OP_4REG xor line3_perm, line0
OP_4REG ROTRI line3_perm, _24
OP_4REG addw line2_perm, line3_perm
OP_4REG xor line1_perm, line2_perm
OP_4REG ROTRI line1_perm, _25
addi i, i, -1
bnez i, .Lpermute
/* output[0,1,2,3] = copy[0,1,2,3] + state[0,1,2,3] */
OP_4REG addw line0, copy
sw state0, (output)
sw state1, 4(output)
sw state2, 8(output)
sw state3, 12(output)
/* from now on state[0,1,2,3] are scratch registers */
/* state[0,1,2,3] = lo(key) */
lw state0, (key)
lw state1, 4(key)
lw state2, 8(key)
lw state3, 12(key)
/* output[4,5,6,7] = state[0,1,2,3] + state[4,5,6,7] */
OP_4REG addw line1, line0
sw state4, 16(output)
sw state5, 20(output)
sw state6, 24(output)
sw state7, 28(output)
/* state[0,1,2,3] = hi(key) */
lw state0, 16(key)
lw state1, 20(key)
lw state2, 24(key)
lw state3, 28(key)
/* output[8,9,10,11] = tmp[0,1,2,3] + state[8,9,10,11] */
OP_4REG addw line2, line0
sw state8, 32(output)
sw state9, 36(output)
sw state10, 40(output)
sw state11, 44(output)
/* output[12,13,14,15] = state[12,13,14,15] + [cnt_lo, cnt_hi, 0, 0] */
addw state12, state12, cnt
srli state0, cnt, 32
addw state13, state13, state0
sw state12, 48(output)
sw state13, 52(output)
sw state14, 56(output)
sw state15, 60(output)
/* ++counter */
addi cnt, cnt, 1
/* output += 64 */
addi output, output, 64
/* --nblocks */
addi nblocks, nblocks, -1
bnez nblocks, .Lblock
/* counter = [cnt_lo, cnt_hi] */
sd cnt, (counter)
/* Zero out the potentially sensitive regs, in case nothing uses these
* again. As at now copy[0,1,2,3] just contains "expand 32-byte k" and
* state[0,...,11] are s0-s11 those we'll restore in the epilogue, we
* only need to zero state[12,...,15].
*/
mv state12, zero
mv state13, zero
mv state14, zero
mv state15, zero
REG_L s0, (sp)
REG_L s1, SZREG(sp)
REG_L s2, 2*SZREG(sp)
REG_L s3, 3*SZREG(sp)
REG_L s4, 4*SZREG(sp)
REG_L s5, 5*SZREG(sp)
REG_L s6, 6*SZREG(sp)
REG_L s7, 7*SZREG(sp)
REG_L s8, 8*SZREG(sp)
REG_L s9, 9*SZREG(sp)
REG_L s10, 10*SZREG(sp)
REG_L s11, 11*SZREG(sp)
addi sp, sp, 12*SZREG
ret
SYM_FUNC_END(__arch_chacha20_blocks_nostack)

10
arch/riscv/kernel/vendor_extensions.c
View file

@ -6,6 +6,7 @@
#include <asm/vendorid_list.h>
#include <asm/vendor_extensions.h>
#include <asm/vendor_extensions/andes.h>
#include <asm/vendor_extensions/sifive.h>
#include <asm/vendor_extensions/thead.h>
#include <linux/array_size.h>
@ -15,6 +16,9 @@ struct riscv_isa_vendor_ext_data_list *riscv_isa_vendor_ext_list[] = {
#ifdef CONFIG_RISCV_ISA_VENDOR_EXT_ANDES
&riscv_isa_vendor_ext_list_andes,
#endif
#ifdef CONFIG_RISCV_ISA_VENDOR_EXT_SIFIVE
&riscv_isa_vendor_ext_list_sifive,
#endif
#ifdef CONFIG_RISCV_ISA_VENDOR_EXT_THEAD
&riscv_isa_vendor_ext_list_thead,
#endif
@ -45,6 +49,12 @@ bool __riscv_isa_vendor_extension_available(int cpu, unsigned long vendor, unsig
cpu_bmap = riscv_isa_vendor_ext_list_andes.per_hart_isa_bitmap;
break;
#endif
#ifdef CONFIG_RISCV_ISA_VENDOR_EXT_SIFIVE
case SIFIVE_VENDOR_ID:
bmap = &riscv_isa_vendor_ext_list_sifive.all_harts_isa_bitmap;
cpu_bmap = riscv_isa_vendor_ext_list_sifive.per_hart_isa_bitmap;
break;
#endif
#ifdef CONFIG_RISCV_ISA_VENDOR_EXT_THEAD
case THEAD_VENDOR_ID:
bmap = &riscv_isa_vendor_ext_list_thead.all_harts_isa_bitmap;

2
arch/riscv/kernel/vendor_extensions/Makefile
View file

@ -1,5 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
obj-$(CONFIG_RISCV_ISA_VENDOR_EXT_ANDES) += andes.o
obj-$(CONFIG_RISCV_ISA_VENDOR_EXT_SIFIVE) += sifive.o
obj-$(CONFIG_RISCV_ISA_VENDOR_EXT_SIFIVE) += sifive_hwprobe.o
obj-$(CONFIG_RISCV_ISA_VENDOR_EXT_THEAD) += thead.o
obj-$(CONFIG_RISCV_ISA_VENDOR_EXT_THEAD) += thead_hwprobe.o

21
arch/riscv/kernel/vendor_extensions/sifive.c Normal file
View file

@ -0,0 +1,21 @@
// SPDX-License-Identifier: GPL-2.0-only
#include <asm/cpufeature.h>
#include <asm/vendor_extensions.h>
#include <asm/vendor_extensions/sifive.h>
#include <linux/array_size.h>
#include <linux/types.h>
/* All SiFive vendor extensions supported in Linux */
const struct riscv_isa_ext_data riscv_isa_vendor_ext_sifive[] = {
__RISCV_ISA_EXT_DATA(xsfvfnrclipxfqf, RISCV_ISA_VENDOR_EXT_XSFVFNRCLIPXFQF),
__RISCV_ISA_EXT_DATA(xsfvfwmaccqqq, RISCV_ISA_VENDOR_EXT_XSFVFWMACCQQQ),
__RISCV_ISA_EXT_DATA(xsfvqmaccdod, RISCV_ISA_VENDOR_EXT_XSFVQMACCDOD),
__RISCV_ISA_EXT_DATA(xsfvqmaccqoq, RISCV_ISA_VENDOR_EXT_XSFVQMACCQOQ),
};
struct riscv_isa_vendor_ext_data_list riscv_isa_vendor_ext_list_sifive = {
.ext_data_count = ARRAY_SIZE(riscv_isa_vendor_ext_sifive),
.ext_data = riscv_isa_vendor_ext_sifive,
};

22
arch/riscv/kernel/vendor_extensions/sifive_hwprobe.c Normal file
View file

@ -0,0 +1,22 @@
// SPDX-License-Identifier: GPL-2.0-only
#include <asm/vendor_extensions/sifive.h>
#include <asm/vendor_extensions/sifive_hwprobe.h>
#include <asm/vendor_extensions/vendor_hwprobe.h>
#include <linux/cpumask.h>
#include <linux/types.h>
#include <uapi/asm/hwprobe.h>
#include <uapi/asm/vendor/sifive.h>
void hwprobe_isa_vendor_ext_sifive_0(struct riscv_hwprobe *pair, const struct cpumask *cpus)
{
VENDOR_EXTENSION_SUPPORTED(pair, cpus,
riscv_isa_vendor_ext_list_sifive.per_hart_isa_bitmap, {
VENDOR_EXT_KEY(XSFVQMACCDOD);
VENDOR_EXT_KEY(XSFVQMACCQOQ);
VENDOR_EXT_KEY(XSFVFNRCLIPXFQF);
VENDOR_EXT_KEY(XSFVFWMACCQQQ);
});
}

11
arch/riscv/lib/riscv_v_helpers.c
View file

@ -16,8 +16,11 @@
#ifdef CONFIG_MMU
size_t riscv_v_usercopy_threshold = CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD;
int __asm_vector_usercopy(void *dst, void *src, size_t n);
int __asm_vector_usercopy_sum_enabled(void *dst, void *src, size_t n);
int fallback_scalar_usercopy(void *dst, void *src, size_t n);
asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n)
int fallback_scalar_usercopy_sum_enabled(void *dst, void *src, size_t n);
asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n,
bool enable_sum)
{
size_t remain, copied;
@ -26,7 +29,8 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n)
goto fallback;
kernel_vector_begin();
remain = __asm_vector_usercopy(dst, src, n);
remain = enable_sum ? __asm_vector_usercopy(dst, src, n) :
__asm_vector_usercopy_sum_enabled(dst, src, n);
kernel_vector_end();
if (remain) {
@ -40,6 +44,7 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n)
return remain;
fallback:
return fallback_scalar_usercopy(dst, src, n);
return enable_sum ? fallback_scalar_usercopy(dst, src, n) :
fallback_scalar_usercopy_sum_enabled(dst, src, n);
}
#endif

44
arch/riscv/lib/uaccess.S
View file

@ -17,14 +17,43 @@ SYM_FUNC_START(__asm_copy_to_user)
ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_ZVE32X, CONFIG_RISCV_ISA_V)
REG_L t0, riscv_v_usercopy_threshold
bltu a2, t0, fallback_scalar_usercopy
li a3, 1
tail enter_vector_usercopy
#endif
SYM_FUNC_START(fallback_scalar_usercopy)
SYM_FUNC_END(__asm_copy_to_user)
EXPORT_SYMBOL(__asm_copy_to_user)
SYM_FUNC_ALIAS(__asm_copy_from_user, __asm_copy_to_user)
EXPORT_SYMBOL(__asm_copy_from_user)
SYM_FUNC_START(fallback_scalar_usercopy)
/* Enable access to user memory */
li t6, SR_SUM
csrs CSR_STATUS, t6
mv t6, ra
call fallback_scalar_usercopy_sum_enabled
/* Disable access to user memory */
mv ra, t6
li t6, SR_SUM
csrc CSR_STATUS, t6
ret
SYM_FUNC_END(fallback_scalar_usercopy)
SYM_FUNC_START(__asm_copy_to_user_sum_enabled)
#ifdef CONFIG_RISCV_ISA_V
ALTERNATIVE("j fallback_scalar_usercopy_sum_enabled", "nop", 0, RISCV_ISA_EXT_ZVE32X, CONFIG_RISCV_ISA_V)
REG_L t0, riscv_v_usercopy_threshold
bltu a2, t0, fallback_scalar_usercopy_sum_enabled
li a3, 0
tail enter_vector_usercopy
#endif
SYM_FUNC_END(__asm_copy_to_user_sum_enabled)
SYM_FUNC_ALIAS(__asm_copy_from_user_sum_enabled, __asm_copy_to_user_sum_enabled)
EXPORT_SYMBOL(__asm_copy_from_user_sum_enabled)
EXPORT_SYMBOL(__asm_copy_to_user_sum_enabled)
SYM_FUNC_START(fallback_scalar_usercopy_sum_enabled)
/*
* Save the terminal address which will be used to compute the number
* of bytes copied in case of a fixup exception.
@ -178,23 +207,12 @@ SYM_FUNC_START(fallback_scalar_usercopy)
bltu a0, t0, 4b /* t0 - end of dst */
.Lout_copy_user:
/* Disable access to user memory */
csrc CSR_STATUS, t6
li a0, 0
ret
/* Exception fixup code */
10:
/* Disable access to user memory */
csrc CSR_STATUS, t6
sub a0, t5, a0
ret
SYM_FUNC_END(__asm_copy_to_user)
SYM_FUNC_END(fallback_scalar_usercopy)
EXPORT_SYMBOL(__asm_copy_to_user)
SYM_FUNC_ALIAS(__asm_copy_from_user, __asm_copy_to_user)
EXPORT_SYMBOL(__asm_copy_from_user)
SYM_FUNC_END(fallback_scalar_usercopy_sum_enabled)
SYM_FUNC_START(__clear_user)

15
arch/riscv/lib/uaccess_vector.S
View file

@ -24,7 +24,18 @@ SYM_FUNC_START(__asm_vector_usercopy)
/* Enable access to user memory */
li t6, SR_SUM
csrs CSR_STATUS, t6
mv t6, ra
call __asm_vector_usercopy_sum_enabled
/* Disable access to user memory */
mv ra, t6
li t6, SR_SUM
csrc CSR_STATUS, t6
ret
SYM_FUNC_END(__asm_vector_usercopy)
SYM_FUNC_START(__asm_vector_usercopy_sum_enabled)
loop:
vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
fixup vle8.v vData, (pSrc), 10f
@ -36,8 +47,6 @@ loop:
/* Exception fixup for vector load is shared with normal exit */
10:
/* Disable access to user memory */
csrc CSR_STATUS, t6
mv a0, iNum
ret
@ -49,4 +58,4 @@ loop:
csrr t2, CSR_VSTART
sub iNum, iNum, t2
j 10b
SYM_FUNC_END(__asm_vector_usercopy)
SYM_FUNC_END(__asm_vector_usercopy_sum_enabled)

29
arch/riscv/mm/cacheflush.c
View file

@ -24,7 +24,20 @@ void flush_icache_all(void)
if (num_online_cpus() < 2)
return;
else if (riscv_use_sbi_for_rfence())
/*
* Make sure all previous writes to the D$ are ordered before making
* the IPI. The RISC-V spec states that a hart must execute a data fence
* before triggering a remote fence.i in order to make the modification
* visable for remote harts.
*
* IPIs on RISC-V are triggered by MMIO writes to either CLINT or
* S-IMSIC, so the fence ensures previous data writes "happen before"
* the MMIO.
*/
RISCV_FENCE(w, o);
if (riscv_use_sbi_for_rfence())
sbi_remote_fence_i(NULL);
else
on_each_cpu(ipi_remote_fence_i, NULL, 1);
@ -101,6 +114,9 @@ EXPORT_SYMBOL_GPL(riscv_cbom_block_size);
unsigned int riscv_cboz_block_size;
EXPORT_SYMBOL_GPL(riscv_cboz_block_size);
unsigned int riscv_cbop_block_size;
EXPORT_SYMBOL_GPL(riscv_cbop_block_size);
static void __init cbo_get_block_size(struct device_node *node,
const char *name, u32 *block_size,
unsigned long *first_hartid)
@ -125,8 +141,8 @@ static void __init cbo_get_block_size(struct device_node *node,
void __init riscv_init_cbo_blocksizes(void)
{
unsigned long cbom_hartid, cboz_hartid;
u32 cbom_block_size = 0, cboz_block_size = 0;
unsigned long cbom_hartid, cboz_hartid, cbop_hartid;
u32 cbom_block_size = 0, cboz_block_size = 0, cbop_block_size = 0;
struct device_node *node;
struct acpi_table_header *rhct;
acpi_status status;
@ -138,13 +154,15 @@ void __init riscv_init_cbo_blocksizes(void)
&cbom_block_size, &cbom_hartid);
cbo_get_block_size(node, "riscv,cboz-block-size",
&cboz_block_size, &cboz_hartid);
cbo_get_block_size(node, "riscv,cbop-block-size",
&cbop_block_size, &cbop_hartid);
}
} else {
status = acpi_get_table(ACPI_SIG_RHCT, 0, &rhct);
if (ACPI_FAILURE(status))
return;
acpi_get_cbo_block_size(rhct, &cbom_block_size, &cboz_block_size, NULL);
acpi_get_cbo_block_size(rhct, &cbom_block_size, &cboz_block_size, &cbop_block_size);
acpi_put_table((struct acpi_table_header *)rhct);
}
@ -153,6 +171,9 @@ void __init riscv_init_cbo_blocksizes(void)
if (cboz_block_size)
riscv_cboz_block_size = cboz_block_size;
if (cbop_block_size)
riscv_cbop_block_size = cbop_block_size;
}
#ifdef CONFIG_SMP

10
arch/riscv/mm/pgtable.c
View file

@ -154,4 +154,14 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
flush_tlb_mm(vma->vm_mm);
return pmd;
}
pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address,
pud_t *pudp)
{
VM_WARN_ON_ONCE(!pud_present(*pudp));
pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp));
flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE);
return old;
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */

38
arch/riscv/mm/tlbflush.c
View file

@ -7,6 +7,27 @@
#include <linux/mmu_notifier.h>
#include <asm/sbi.h>
#include <asm/mmu_context.h>
#include <asm/cpufeature.h>
#define has_svinval() riscv_has_extension_unlikely(RISCV_ISA_EXT_SVINVAL)
static inline void local_sfence_inval_ir(void)
{
asm volatile(SFENCE_INVAL_IR() ::: "memory");
}
static inline void local_sfence_w_inval(void)
{
asm volatile(SFENCE_W_INVAL() ::: "memory");
}
static inline void local_sinval_vma(unsigned long vma, unsigned long asid)
{
if (asid != FLUSH_TLB_NO_ASID)
asm volatile(SINVAL_VMA(%0, %1) : : "r" (vma), "r" (asid) : "memory");
else
asm volatile(SINVAL_VMA(%0, zero) : : "r" (vma) : "memory");
}
/*
* Flush entire TLB if number of entries to be flushed is greater
@ -27,6 +48,16 @@ static void local_flush_tlb_range_threshold_asid(unsigned long start,
return;
}
if (has_svinval()) {
local_sfence_w_inval();
for (i = 0; i < nr_ptes_in_range; ++i) {
local_sinval_vma(start, asid);
start += stride;
}
local_sfence_inval_ir();
return;
}
for (i = 0; i < nr_ptes_in_range; ++i) {
local_flush_tlb_page_asid(start, asid);
start += stride;
@ -182,6 +213,13 @@ void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
__flush_tlb_range(vma->vm_mm, mm_cpumask(vma->vm_mm),
start, end - start, PMD_SIZE);
}
void flush_pud_tlb_range(struct vm_area_struct *vma, unsigned long start,
unsigned long end)
{
__flush_tlb_range(vma->vm_mm, mm_cpumask(vma->vm_mm),
start, end - start, PUD_SIZE);
}
#endif
bool arch_tlbbatch_should_defer(struct mm_struct *mm)

2
include/linux/ftrace.h
View file

@ -635,6 +635,8 @@ enum {
#define ftrace_get_symaddr(fentry_ip) (0)
#endif
void ftrace_sync_ipi(void *data);
#ifdef CONFIG_DYNAMIC_FTRACE
void ftrace_arch_code_modify_prepare(void);

5
include/linux/raid/pq.h
View file

@ -108,6 +108,10 @@ extern const struct raid6_calls raid6_vpermxor4;
extern const struct raid6_calls raid6_vpermxor8;
extern const struct raid6_calls raid6_lsx;
extern const struct raid6_calls raid6_lasx;
extern const struct raid6_calls raid6_rvvx1;
extern const struct raid6_calls raid6_rvvx2;
extern const struct raid6_calls raid6_rvvx4;
extern const struct raid6_calls raid6_rvvx8;
struct raid6_recov_calls {
void (*data2)(int, size_t, int, int, void **);
@ -125,6 +129,7 @@ extern const struct raid6_recov_calls raid6_recov_s390xc;
extern const struct raid6_recov_calls raid6_recov_neon;
extern const struct raid6_recov_calls raid6_recov_lsx;
extern const struct raid6_recov_calls raid6_recov_lasx;
extern const struct raid6_recov_calls raid6_recov_rvv;
extern const struct raid6_calls raid6_neonx1;
extern const struct raid6_calls raid6_neonx2;

2
kernel/trace/ftrace.c
View file

@ -188,7 +188,7 @@ static void ftrace_pid_func(unsigned long ip, unsigned long parent_ip,
op->saved_func(ip, parent_ip, op, fregs);
}
static void ftrace_sync_ipi(void *data)
void ftrace_sync_ipi(void *data)
{
/* Probably not needed, but do it anyway */
smp_rmb();

1
lib/raid6/Makefile
View file

@ -10,6 +10,7 @@ raid6_pq-$(CONFIG_ALTIVEC) += altivec1.o altivec2.o altivec4.o altivec8.o \
raid6_pq-$(CONFIG_KERNEL_MODE_NEON) += neon.o neon1.o neon2.o neon4.o neon8.o recov_neon.o recov_neon_inner.o
raid6_pq-$(CONFIG_S390) += s390vx8.o recov_s390xc.o
raid6_pq-$(CONFIG_LOONGARCH) += loongarch_simd.o recov_loongarch_simd.o
raid6_pq-$(CONFIG_RISCV_ISA_V) += rvv.o recov_rvv.o
hostprogs += mktables

9
lib/raid6/algos.c
View file

@ -76,6 +76,12 @@ const struct raid6_calls * const raid6_algos[] = {
#ifdef CONFIG_CPU_HAS_LSX
&raid6_lsx,
#endif
#endif
#ifdef CONFIG_RISCV_ISA_V
&raid6_rvvx1,
&raid6_rvvx2,
&raid6_rvvx4,
&raid6_rvvx8,
#endif
&raid6_intx8,
&raid6_intx4,
@ -109,6 +115,9 @@ const struct raid6_recov_calls *const raid6_recov_algos[] = {
#ifdef CONFIG_CPU_HAS_LSX
&raid6_recov_lsx,
#endif
#endif
#ifdef CONFIG_RISCV_ISA_V
&raid6_recov_rvv,
#endif
&raid6_recov_intx1,
NULL

229
lib/raid6/recov_rvv.c Normal file
View file

@ -0,0 +1,229 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Copyright 2024 Institute of Software, CAS.
* Author: Chunyan Zhang <zhangchunyan@iscas.ac.cn>
*/
#include <asm/simd.h>
#include <asm/vector.h>
#include <crypto/internal/simd.h>
#include <linux/raid/pq.h>
static int rvv_has_vector(void)
{
return has_vector();
}
static void __raid6_2data_recov_rvv(int bytes, u8 *p, u8 *q, u8 *dp,
u8 *dq, const u8 *pbmul,
const u8 *qmul)
{
asm volatile (".option push\n"
".option arch,+v\n"
"vsetvli x0, %[avl], e8, m1, ta, ma\n"
".option pop\n"
: :
[avl]"r"(16)
);
/*
* while ( bytes-- ) {
* uint8_t px, qx, db;
*
* px = *p ^ *dp;
* qx = qmul[*q ^ *dq];
* *dq++ = db = pbmul[px] ^ qx;
* *dp++ = db ^ px;
* p++; q++;
* }
*/
while (bytes) {
/*
* v0:px, v1:dp,
* v2:qx, v3:dq,
* v4:vx, v5:vy,
* v6:qm0, v7:qm1,
* v8:pm0, v9:pm1,
* v14:p/qm[vx], v15:p/qm[vy]
*/
asm volatile (".option push\n"
".option arch,+v\n"
"vle8.v v0, (%[px])\n"
"vle8.v v1, (%[dp])\n"
"vxor.vv v0, v0, v1\n"
"vle8.v v2, (%[qx])\n"
"vle8.v v3, (%[dq])\n"
"vxor.vv v4, v2, v3\n"
"vsrl.vi v5, v4, 4\n"
"vand.vi v4, v4, 0xf\n"
"vle8.v v6, (%[qm0])\n"
"vle8.v v7, (%[qm1])\n"
"vrgather.vv v14, v6, v4\n" /* v14 = qm[vx] */
"vrgather.vv v15, v7, v5\n" /* v15 = qm[vy] */
"vxor.vv v2, v14, v15\n" /* v2 = qmul[*q ^ *dq] */
"vsrl.vi v5, v0, 4\n"
"vand.vi v4, v0, 0xf\n"
"vle8.v v8, (%[pm0])\n"
"vle8.v v9, (%[pm1])\n"
"vrgather.vv v14, v8, v4\n" /* v14 = pm[vx] */
"vrgather.vv v15, v9, v5\n" /* v15 = pm[vy] */
"vxor.vv v4, v14, v15\n" /* v4 = pbmul[px] */
"vxor.vv v3, v4, v2\n" /* v3 = db = pbmul[px] ^ qx */
"vxor.vv v1, v3, v0\n" /* v1 = db ^ px; */
"vse8.v v3, (%[dq])\n"
"vse8.v v1, (%[dp])\n"
".option pop\n"
: :
[px]"r"(p),
[dp]"r"(dp),
[qx]"r"(q),
[dq]"r"(dq),
[qm0]"r"(qmul),
[qm1]"r"(qmul + 16),
[pm0]"r"(pbmul),
[pm1]"r"(pbmul + 16)
:);
bytes -= 16;
p += 16;
q += 16;
dp += 16;
dq += 16;
}
}
static void __raid6_datap_recov_rvv(int bytes, u8 *p, u8 *q,
u8 *dq, const u8 *qmul)
{
asm volatile (".option push\n"
".option arch,+v\n"
"vsetvli x0, %[avl], e8, m1, ta, ma\n"
".option pop\n"
: :
[avl]"r"(16)
);
/*
* while (bytes--) {
* *p++ ^= *dq = qmul[*q ^ *dq];
* q++; dq++;
* }
*/
while (bytes) {
/*
* v0:vx, v1:vy,
* v2:dq, v3:p,
* v4:qm0, v5:qm1,
* v10:m[vx], v11:m[vy]
*/
asm volatile (".option push\n"
".option arch,+v\n"
"vle8.v v0, (%[vx])\n"
"vle8.v v2, (%[dq])\n"
"vxor.vv v0, v0, v2\n"
"vsrl.vi v1, v0, 4\n"
"vand.vi v0, v0, 0xf\n"
"vle8.v v4, (%[qm0])\n"
"vle8.v v5, (%[qm1])\n"
"vrgather.vv v10, v4, v0\n"
"vrgather.vv v11, v5, v1\n"
"vxor.vv v0, v10, v11\n"
"vle8.v v1, (%[vy])\n"
"vxor.vv v1, v0, v1\n"
"vse8.v v0, (%[dq])\n"
"vse8.v v1, (%[vy])\n"
".option pop\n"
: :
[vx]"r"(q),
[vy]"r"(p),
[dq]"r"(dq),
[qm0]"r"(qmul),
[qm1]"r"(qmul + 16)
:);
bytes -= 16;
p += 16;
q += 16;
dq += 16;
}
}
static void raid6_2data_recov_rvv(int disks, size_t bytes, int faila,
int failb, void **ptrs)
{
u8 *p, *q, *dp, *dq;
const u8 *pbmul; /* P multiplier table for B data */
const u8 *qmul; /* Q multiplier table (for both) */
p = (u8 *)ptrs[disks - 2];
q = (u8 *)ptrs[disks - 1];
/*
* Compute syndrome with zero for the missing data pages
* Use the dead data pages as temporary storage for
* delta p and delta q
*/
dp = (u8 *)ptrs[faila];
ptrs[faila] = (void *)raid6_empty_zero_page;
ptrs[disks - 2] = dp;
dq = (u8 *)ptrs[failb];
ptrs[failb] = (void *)raid6_empty_zero_page;
ptrs[disks - 1] = dq;
raid6_call.gen_syndrome(disks, bytes, ptrs);
/* Restore pointer table */
ptrs[faila] = dp;
ptrs[failb] = dq;
ptrs[disks - 2] = p;
ptrs[disks - 1] = q;
/* Now, pick the proper data tables */
pbmul = raid6_vgfmul[raid6_gfexi[failb - faila]];
qmul = raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila] ^
raid6_gfexp[failb]]];
kernel_vector_begin();
__raid6_2data_recov_rvv(bytes, p, q, dp, dq, pbmul, qmul);
kernel_vector_end();
}
static void raid6_datap_recov_rvv(int disks, size_t bytes, int faila,
void **ptrs)
{
u8 *p, *q, *dq;
const u8 *qmul; /* Q multiplier table */
p = (u8 *)ptrs[disks - 2];
q = (u8 *)ptrs[disks - 1];
/*
* Compute syndrome with zero for the missing data page
* Use the dead data page as temporary storage for delta q
*/
dq = (u8 *)ptrs[faila];
ptrs[faila] = (void *)raid6_empty_zero_page;
ptrs[disks - 1] = dq;
raid6_call.gen_syndrome(disks, bytes, ptrs);
/* Restore pointer table */
ptrs[faila] = dq;
ptrs[disks - 1] = q;
/* Now, pick the proper data tables */
qmul = raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila]]];
kernel_vector_begin();
__raid6_datap_recov_rvv(bytes, p, q, dq, qmul);
kernel_vector_end();
}
const struct raid6_recov_calls raid6_recov_rvv = {
.data2 = raid6_2data_recov_rvv,
.datap = raid6_datap_recov_rvv,
.valid = rvv_has_vector,
.name = "rvv",
.priority = 1,
};

1212
lib/raid6/rvv.c Normal file
View file

File diff suppressed because it is too large Load diff

39
lib/raid6/rvv.h Normal file
View file

@ -0,0 +1,39 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* Copyright 2024 Institute of Software, CAS.
*
* raid6/rvv.h
*
* Definitions for RISC-V RAID-6 code
*/
#define RAID6_RVV_WRAPPER(_n) \
static void raid6_rvv ## _n ## _gen_syndrome(int disks, \
size_t bytes, void **ptrs) \
{ \
void raid6_rvv ## _n ## _gen_syndrome_real(int d, \
unsigned long b, void **p); \
kernel_vector_begin(); \
raid6_rvv ## _n ## _gen_syndrome_real(disks, \
(unsigned long)bytes, ptrs); \
kernel_vector_end(); \
} \
static void raid6_rvv ## _n ## _xor_syndrome(int disks, \
int start, int stop, \
size_t bytes, void **ptrs) \
{ \
void raid6_rvv ## _n ## _xor_syndrome_real(int d, \
int s1, int s2, \
unsigned long b, void **p); \
kernel_vector_begin(); \
raid6_rvv ## _n ## _xor_syndrome_real(disks, \
start, stop, (unsigned long)bytes, ptrs); \
kernel_vector_end(); \
} \
struct raid6_calls const raid6_rvvx ## _n = { \
raid6_rvv ## _n ## _gen_syndrome, \
raid6_rvv ## _n ## _xor_syndrome, \
rvv_has_vector, \
"rvvx" #_n, \
0 \
}

6
tools/perf/util/symbol-elf.c
View file

@ -1668,6 +1668,12 @@ dso__load_sym_internal(struct dso *dso, struct map *map, struct symsrc *syms_ss,
continue;
}
/* Reject RISCV ELF "mapping symbols" */
if (ehdr.e_machine == EM_RISCV) {
if (elf_name[0] == '$' && strchr("dx", elf_name[1]))
continue;
}
if (runtime_ss->opdsec && sym.st_shndx == runtime_ss->opdidx) {
u32 offset = sym.st_value - syms_ss->opdshdr.sh_addr;
u64 *opd = opddata->d_buf + offset;

2
tools/testing/selftests/vDSO/vgetrandom-chacha.S
View file

@ -11,6 +11,8 @@
#include "../../../../arch/loongarch/vdso/vgetrandom-chacha.S"
#elif defined(__powerpc__) || defined(__powerpc64__)
#include "../../../../arch/powerpc/kernel/vdso/vgetrandom-chacha.S"
#elif defined(__riscv) && __riscv_xlen == 64
#include "../../../../arch/riscv/kernel/vdso/vgetrandom-chacha.S"
#elif defined(__s390x__)
#include "../../../../arch/s390/kernel/vdso64/vgetrandom-chacha.S"
#elif defined(__x86_64__)