An early return triggered by read_prog_test_msg leaves uninitialized elements,
which leads to memory corruption during free_test_states cleanup.
Signed-off-by: Feng Yang <yangfeng@kylinos.cn>
Link: https://lore.kernel.org/bpf/20260723085100.482147-5-yangfeng59949@163.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
pthread_create returns 0 on success and a positive error code on failure;
it never returns a negative value. The current conditional branch can never be taken.
Failures during thread creation are silently ignored, which will lead to
invalid memory access when waiting on threads or dereferencing thread handles later.
Fixes: 91b2c0afd0 ("selftests/bpf: Add parallelism to test_progs")
Signed-off-by: Feng Yang <yangfeng@kylinos.cn>
Link: https://lore.kernel.org/bpf/20260723085100.482147-3-yangfeng59949@163.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The name has already been freed in the free_subtest_state function
and does not need to be freed again. The extra free is noop since the
pointer was already set to NULL.
Signed-off-by: Feng Yang <yangfeng@kylinos.cn>
Link: https://lore.kernel.org/bpf/20260723085100.482147-2-yangfeng59949@163.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Maxim Khmelevskii says:
====================
s390/bpf: Support load-acquire and store-release instructions
Support load-acquire (BPF_LOAD_ACQ) and store-release (BPF_STORE_REL)
instructions. Since s390 has strong memory model, implement
them as regular BPF_LDX/BPF_STX instructions.
====================
Link: https://patch.msgid.link/20260723140648.583055-5-max@linux.ibm.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add new functions for load and store to reuse
them in the load-acquire and store-release logic.
Signed-off-by: Maxim Khmelevskii <max@linux.ibm.com>
Reviewed-by: Ilya Leoshkevich <iii@linux.ibm.com>
Link: https://lore.kernel.org/bpf/20260723140648.583055-6-max@linux.ibm.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Currently, there is no RISC-V support for get_preempt_count() and
its fallback path always returns 0.
Add it so that bpf_in_interrupt(), bpf_in_nmi(), bpf_in_hardirq(),
bpf_in_serving_softirq(), and bpf_in_task() work for RISC-V as well.
Given that RISC-V has supported CONFIG_THREAD_INFO_IN_TASK since its
initial commit fbe934d69e ("RISC-V: Build Infrastructure") in 2017,
directly retrieve preempt_count from the thread_info embedded within
task_struct via bpf_get_current_task_btf().
This aligns the implementation with arm64, powerpc, and loongarch.
Tested on a RISC-V virtual machine.
Before:
$ sudo ./test_progs -t exe_ctx
...
#114 exe_ctx:FAIL
Summary: 0/0 PASSED, 0 SKIPPED, 1 FAILED
After:
$ sudo ./test_progs -t exe_ctx
#114 exe_ctx:OK
Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/bpf/20260722022906.8778-1-yangtiezhu@loongson.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Leon Hwang says:
====================
bpf: Fix WARNING in bpf_tracing_link_release
The trampoline could be corrupted by the blindly
'tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX' in verifier.
1. A fexit attached to a tail_call_reachable prog. 'tr->flags' became
'BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_TAIL_CALL_CTX'. And, the
trampoline would poke the target prog's nop insn using jmp insn instead
of call insn.
2. Another fexit loaded with the same tail_call_reachable prog target.
'tr->flags' became 'BPF_TRAMP_F_TAIL_CALL_CTX'.
3. Close the first fexit link. Due to no BPF_TRAMP_F_CALL_ORIG in
'tr->flags', the trampoline will fail to restore the prog's nop insn
using call insn.
[ 3.410719] WARNING: kernel/bpf/syscall.c:3551 at bpf_tracing_link_release+0x53/0x60, CPU#1: test_progs/98
...
[ 3.428793] bpf_link_free+0x58/0x130
[ 3.429293] bpf_link_release+0x23/0x30
Fix the warning by updating 'tr->flags' with '|=' and lock.
Changes:
v1 -> v2:
* Update the patch #1 message with 'tr->flags' change. (per Jiri)
* Drop the 'link' and the last 'if' in patch #2. (per Jiri)
* v1: https://lore.kernel.org/bpf/20260721133036.49265-1-leon.hwang@linux.dev/
====================
Link: https://patch.msgid.link/20260722151909.69142-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add a test to verify that there's no WARNING when detaching fexit link by
following the repro steps of previous commit.
Without the fix, the WARNING could be triggered by this test.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260722151909.69142-3-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The trampoline could be corrupted by the blindly
'tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX' in verifier.
1. A fexit attached to a tail_call_reachable prog. 'tr->flags' became
'BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_TAIL_CALL_CTX'. And, the
trampoline would poke the target prog's nop insn using jmp insn instead
of call insn.
2. Another fexit loaded with the same tail_call_reachable prog target.
'tr->flags' became 'BPF_TRAMP_F_TAIL_CALL_CTX'.
3. Close the first fexit link. Due to no BPF_TRAMP_F_CALL_ORIG in
'tr->flags', the trampoline will fail to restore the prog's nop insn
using call insn.
[ 3.410719] WARNING: kernel/bpf/syscall.c:3551 at bpf_tracing_link_release+0x53/0x60, CPU#1: test_progs/98
...
[ 3.428793] bpf_link_free+0x58/0x130
[ 3.429293] bpf_link_release+0x23/0x30
Fix the warning by updating 'tr->flags' with '|=' and lock.
Fixes: 2b5dcb31a1 ("bpf, x64: Fix tailcall infinite loop")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260722151909.69142-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
BPF subprograms use the bpf_callback_t ABI, but task work invokes the
callback through a three-argument function pointer. This trips kCFI.
Store and invoke the callback as bpf_callback_t.
Fixes: 38aa7003e3 ("bpf: task work scheduling kfuncs")
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://lore.kernel.org/bpf/20260724-task_work_cfi-v1-1-2616691781ed@meta.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Feng Jiang says:
====================
bpf, riscv: add timed may_goto support
This series adds RISC-V JIT support for the timed may_goto loop bound.
Patch 1 implements arch_bpf_timed_may_goto() and enables
bpf_jit_supports_timed_may_goto() so the verifier uses the timed
expansion path.
Patch 2 adds a test that checks R0-R5 are preserved across
arch_bpf_timed_may_goto() calls.
Patch 3 enables the verifier_may_goto_1, stream_cond_break, and
may_goto_interaction fastcall tests on riscv64.
Tested on riscv64 QEMU (rva23s64): may_goto programs load and JIT
correctly, and the 250ms timeout path works as expected.
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn>
Tested-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
---
Changes in v5:
- Use REG_S/REG_L/SZREG in arch_bpf_timed_may_goto assembly. (Pu Lehui)
- Add __arch_s390x to the timed_may_goto_preserves_regs test. (Pu Lehui)
- Switch the preserves-regs test to SEC("syscall") to fix
bpf_prog_test_run() EINVAL.
- Link to v4: https://lore.kernel.org/r/20260722-riscv-bpf-timed-may-goto-v4-0-e117e6337bc7@kylinos.cn
Changes in v4:
- Add 'bpf-next' prefix to match the BPF kernel tree workflow.
- Add a test checking that R0-R5 are preserved across
arch_bpf_timed_may_goto() calls. Use bpf_get_prandom_u32() to
prevent the verifier from removing the checks via DCE.
- Rename may_goto_interaction_arm64() to may_goto_interaction().
- Wrap the arch_bpf_timed_may_goto address check to a single line.
- Link to v3: https://lore.kernel.org/r/20260715-riscv-bpf-timed-may-goto-v3-0-cf2a9c3d843f@kylinos.cn
Changes in v3:
- Set up the frame pointer in arch_bpf_timed_may_goto() so the function
does not break stack unwinding under CONFIG_FRAME_POINTER.
Changes in v2:
- Fix BPF_REG_0 being clobbered after arch_bpf_timed_may_goto() calls.
- Enable the may_goto_interaction fastcall test on riscv64.
---
====================
Link: https://patch.msgid.link/20260723-riscv-bpf-timed-may-goto-v5-0-86acb54e5642@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Enable verifier_may_goto_1 (raw instruction tests), stream_cond_break
(250ms timeout path), and the may_goto_interaction fastcall test on
riscv64 now that the JIT supports timed may_goto.
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260723-riscv-bpf-timed-may-goto-v5-3-86acb54e5642@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add a test that checks R0-R5 are preserved across
arch_bpf_timed_may_goto() calls.
Use bpf_get_prandom_u32() to avoid the verifier removing the checks
via DCE.
Suggested-by: Björn Töpel <bjorn@kernel.org>
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260723-riscv-bpf-timed-may-goto-v5-2-86acb54e5642@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Implement arch_bpf_timed_may_goto() for the RV64 JIT. The argument and
return value are carried in BPF_REG_AX, and BPF R0-R5 are preserved
across the call to the generic bpf_check_timed_may_goto().
Enable bpf_jit_supports_timed_may_goto() so the verifier uses the timed
expansion path.
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260723-riscv-bpf-timed-may-goto-v5-1-86acb54e5642@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Teach libarena's BPF atomic primitives to use compiler builtins for
load-acquire and store-release when Clang advertises
__BPF_FEATURE_LOAD_ACQ_STORE_REL. Older compilers continue to use the
existing barrier-based fallback.
Notably, as BPF programs begin running on arm64, it is better to use the
more appropriate variants since we can no longer rely on x86 TSO ordering.
Commit 880442305a ("bpf: Introduce load-acquire and store-release instructions")
introduced support, hence kernels from 6.15 onwards are needed when
compiling with compilers supporting these instructions. We have
relatively recent kernel version requirements in libarena anyway, and
have not cut first release, hence declare such a dependency.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260722141003.2841007-1-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Mike Rapoport says:
====================
bpf, x86: enable EXECMEM_ROX_CACHE for BPF allocations
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most
paths that call bpf_jit_alloc_exec() immediately make it ROX with
set_memory_rox().
The code generation, at least on x86, uses separately allocated writable
buffers and then updates the actual text memory with text_poke().
These patches do several small adjustments to how BPF allocates executable
memory and enable EXECMEM_ROX_CACHE for BPF allocations on x86.
Acked-by: Song Liu <song@kernel.org>
---
v3 changes:
* replace vmalloc() with vzalloc() in bpf_dispatcher_change_prog()
* rebase on the current bpf-next
v2: https://patch.msgid.link/20260711-execmem-x86-rox-bpf-v0-v2-0-bfd956d35119@kernel.org
* rebase on the current bpf-next
v1: https://patch.msgid.link/20260626-execmem-x86-rox-bpf-v0-v1-0-45a0b0ed4fe9@kernel.org
---
---
Mike Rapoport (Microsoft) (5):
bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
bpf: drop __weak from bpf_jit_alloc_exec() and bpf_jit_free_exec()
bpf: alloc_prog_pack(): skip ROX management for already ROX memory
bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
x86/bpf: enable EXECMEM_ROX_CACHE for BPF allocations
arch/x86/mm/init.c | 4 ++--
arch/x86/net/bpf_jit_comp.c | 5 ++---
include/linux/filter.h | 1 +
kernel/bpf/core.c | 30 +++++++++++++++++++++---------
kernel/bpf/dispatcher.c | 5 ++++-
5 files changed, 30 insertions(+), 15 deletions(-)
---
base-commit: d1f4b56417
change-id: 20260626-execmem-x86-rox-bpf-v0-b4241ade80df
--
Sincerely yours,
Mike.
====================
Link: https://patch.msgid.link/20260716-execmem-x86-rox-bpf-v0-v3-0-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
BPF core and x86 JIT use text poking and temporary writable buffers and
thus can handle ROX memory.
Enable ROX cache for EXECMEM_BPF when configuration and CPU features allow
that.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-5-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.
This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to
that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
execmem_alloc() can return ROX memory that is already filled with
architecture defined trapping instructions.
In preparation for enabling this mode for BPF on x86, make sure that there
is no redundant management of the ROX memory.
There is no need to fill allocated memory with trapping instructions, to
request permissions reset on free and to set ROX permissions as this all
is handled by execmem_alloc().
Add bpf_jit_mem_is_rox() wrapper for execmem_is_rox(), use it to check if
execmem_alloc() returns ROX memory and skip the redundant steps in that
case.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-3-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_jit_alloc_exec() and bpf_jit_free_exec() are wrappers for the
corresponding execmem APIs.
Architectures define the properties of the memory range needed by BPF in
their initialization of execmem and don't need to override neither of
them.
Drop the __weak qualifier from bpf_jit_alloc_exec() and
bpf_jit_free_exec().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-2-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_dispatcher->rw_image is a temporary writable buffer that
arch_prepare_bpf_dispatcher() fills and then copies into
bpf_dispatcher->image using bpf_arch_text_copy().
The rel32 offsets emitted by emit_bpf_dispatcher() are calculated against
->image, so ->rw_image does not need to live in the module address range.
Allocate ->rw_image with vzalloc() to avoid permissions dance when
EXECMEM_BPF will be backed by ROX caches.
Using vzalloc() rather than vmalloc() ensures that the memory that
bpf_dispatcher_update() unconditionally copies into the executable buffer
is zeroed, which is not ideal but still better than random memory returned
by the existing bpf_jit_alloc_exec() or plain vmalloc().
Switching from bpf_jit_alloc_exec() to vzalloc() also saves a bit of
space in the more scarce module address space.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-1-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Fix unique field logic in BTF
Fix constraint of certain fields that have to be unique when nested
structs are present, BPF_REFCOUNT which should be marked as unique.
See commit logs for details. While at it, fix improper offset check.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/20260719142401.2420111-1-memxor@gmail.com
* Add fix for BPF_REFCOUNT not being unique. (Sashiko)
* Roll improper offset warning check into series. (Sashiko)
====================
Link: https://patch.msgid.link/20260719153634.2908692-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add a raw BTF test with two bpf_refcount fields. The duplicate must be
rejected during BTF loading instead of reaching the duplicate-field
invariant in btf_parse_fields().
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add a raw BTF test with a spin lock directly in a struct and another in a
nested struct. The duplicate must now be rejected during BTF loading.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
BPF_REFCOUNT is not marked as a unique field, while it should be. Fix
this oversight.
Fixes: d54730b50b ("bpf: Introduce opaque bpf_refcount struct and add btf_record plumbing")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
btf_find_struct_field() initializes a fresh seen mask for every recursive
descent. Unique special fields in different levels of the same aggregate
therefore do not see one another. The duplicate fields can reach
btf_parse_fields(), where they trigger an invariant WARN_ON_ONCE(). A
crafted user BTF can consequently trigger the warning before map creation
checks capabilities.
Initialize the seen mask once in btf_find_field() and pass the same pointer
through struct, datasec, and nested-struct walks. This gives the entire field
traversal one shared uniqueness state.
Fixes: 64e8ee8148 ("bpf: look into the types of the fields of a struct type recursively.")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Sashiko pointed out correctly that the case statement for
BPF_RES_SPIN_LOCK incorrectly checks offset for BPF_SPIN_LOCK.
Fix it by checking res_spin_lock_off instead.
Fixes: 0de2046137 ("bpf: Implement verifier support for rqspinlock")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add myself as a reviewer for the BPF areas where I've been active.
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260721172104.3689982-1-ihor.solodrai@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kuan-Wei Chiu says:
====================
riscv, bpf: Add support for signed operations and 32-bit atomics
Add support for missing signed operations and 32-bit atomics in the
RV32 BPF JIT compiler.
The current implementation lacks support for BPF_SDIV, BPF_SMOD, and
BPF_MOVSX, ignoring the instruction offset field and treating them as
unsigned or zero-extended. Introduce support for these operations by
checking the offset field and emitting the corresponding instructions.
Additionally, we leverage the mandatory A extension to natively support
32-bit BPF atomics (and, or, xor, xchg) by mapping them directly to
amo*.w instructions. BPF_CMPXCHG continues to fall back to the
interpreter.
As a result, test_bpf.ko now runs with 0 failures, and the total number
of successfully JIT'ed test cases increases from 843 to 902.
Tested-by: Pu Lehui <pulehui@huawei.com>
---
Changes in v4:
- Add a 'dst' param to emit_bpf_atomic().
Changes in v3:
- Pass insn directly to emit_alu_r32().
- Remove the Fixes: tag and update title.
- Consolidate size, mode, and off into insn for emit_store_r64().
Changes in v2:
- Add missing Fixes tags.
- Fix memory ordering by emitting aq=1, rl=1
Kuan-Wei Chiu (3):
riscv, bpf: Add support for BPF_SDIV and BPF_SMOD in RV32 JIT
riscv, bpf: Add support for BPF_MOVSX in RV32 JIT
riscv, bpf: Add 32 bit atomic operations to RV32 JIT
arch/riscv/net/bpf_jit_comp32.c | 111 +++++++++++++++++++++++++++-----
1 file changed, 95 insertions(+), 16 deletions(-)
====================
Link: https://patch.msgid.link/20260714002451.4091139-1-visitorckw@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The RV32 BPF JIT compiler currently only supports the BPF_ADD atomic
operation. Other 32 bit atomic operations (and, or, xor, xchg) and
their BPF_FETCH variants are not supported and gracefully fall back to
the interpreter.
Since the RISC-V A extension is required for Linux on RV32, we can
natively support these 32-bit BPF atomic operations by mapping them
directly to the corresponding RISC-V amo*.w instructions.
Implement BPF_ADD, BPF_AND, BPF_OR, BPF_XOR, and BPF_XCHG with and
without BPF_FETCH. BPF_CMPXCHG requires a more complex lr.w/sc.w
loop and is left to fall back to the interpreter.
Before this patch:
[ 138.862161] test_bpf: Summary: 1054 PASSED, 0 FAILED, [843/1042 JIT'ed]
After this patch:
[ 157.024124] test_bpf: Summary: 1054 PASSED, 0 FAILED, [902/1042 JIT'ed]
Signed-off-by: Kuan-Wei Chiu <visitorckw@gmail.com>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/bpf/20260714002451.4091139-4-visitorckw@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
emit_atomic_ld_st() returns 1 to have build_body() skip the zext after
a sub-word load_acquire. The caller does "ret = ret ?:
add_exception_handler(...)", which skips add_exception_handler() on any
non-zero ret, so the extable entry is missing and a faulting
PROBE_ATOMIC load_acquire oopses.
REG_DONT_CLEAR_MARKER leaves rd stale on fault, and the verifier still
thinks the load overwrote it, so a program can leak it through a map.
Check ret >= 0 before calling add_exception_handler(), and pass rd for
LOAD_ACQ so the fault zeroes rd like a PROBE_MEM load. Return ret
unchanged for the zext skip.
Fixes: fb7cefabae ("riscv, bpf: Add support arena atomics for RV64")
Suggested-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260720-bpf-riscv-fix-extable-v4-1-165c0b3b07d5@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Pu Lehui says:
====================
Mixing bpf2bpf and tailcalls for RV64
In the current RV64 JIT, if we just don't initialize the TCC in subprog,
the TCC can be propagated from the parent process to the subprocess, but
the updated TCC of the parent process cannot be restored when the
subprocess exits. Since the RV64 TCC is initialized before saving the
callee saved registers into the stack, we cannot use the callee saved
register to pass the TCC, otherwise the original value of the callee
saved register will be destroyed. So we implemented mixing bpf2bpf and
tailcalls similar to x86_64, i.e. using a non-callee saved register to
transfer the TCC between functions, and saving that register to the
stack to protect the TCC value. As for the tailcall hierarchy issue,
inspired by the s390's low-overhead approach, we store TCC from
RV_REG_TCC back to stack after calling bpf2bpf call or calling orig bpf
func in bpf trampoline.
In addition, some code cleans are also attached to this patchset.
Tests test_bpf.ko and test_verifier have passed, as well as the relative
testcases of test_progs*.
v6:
- Add new patch to fix memory leak in bpf_jit_free. (Sashiko)
v5: https://lore.kernel.org/bpf/20260707142219.2871758-1-pulehui@huaweicloud.com
- Fix TCC value was not restored to RV_REG_TCC upon trampoline exit,
resulting in an infinite tail call. (Sashiko)
- Fix epilogue not restored TCC value to RV_REG_TCC, resulting in
leaf tailcall callee may clobbered RV_REG_TCC. (Sashiko)
- Remove patch of `Remove ctx->offset initialization` as it would hit
some corner case. (Sashiko)
- Fix double-count kcfi insn when tailcall to target. (Sashiko)
- Add detail for selftest commit message. (BPF BOT)
v4: https://lore.kernel.org/bpf/20260629140048.733346-1-pulehui@huaweicloud.com
- Fix tailcall hierarchy issue.
- use is_struct_ops_tramp helper in bpf trampoline
v3: https://lore.kernel.org/bpf/20240201083351.943121-1-pulehui@huaweicloud.com
- Remove duplicate RV_REG_TCC load in epiloguei. (Björn Töpel)
v2: https://lore.kernel.org/bpf/20240130040958.230673-1-pulehui@huaweicloud.com
- Fix emit restore RV_REG_TCC double times when `flags &
BPF_TRAMP_F_CALL_ORIG`
- Use bpf_is_subprog helper
v1: https://lore.kernel.org/bpf/20230919035711.3297256-1-pulehui@huaweicloud.com
====================
Link: https://patch.msgid.link/20260708064436.2971933-1-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The RV64 BPF JIT now supports mixing bpf2bpf and tailcalls.
Therefore, the tailcall_bpf2bpf tests can be safely removed
from the riscv64 denylist.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-8-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
In the current RV64 JIT, if we just don't initialize the TCC in subprog,
the TCC can be propagated from the parent process to the subprocess, but
the updated TCC of the parent process cannot be restored when the
subprocess exits. Since the RV64 TCC is initialized before saving the
callee saved registers into the stack, we cannot use the callee saved
register to pass the TCC, otherwise the original value of the callee
saved register will be destroyed. So we implemented mixing bpf2bpf and
tailcalls similar to x86_64, i.e. using a non-callee saved register to
transfer the TCC between functions, and saving that register to the
stack to protect the TCC value. As for the tailcall hierarchy issue,
inspired by the s390's low-overhead approach, we store TCC from
RV_REG_TCC back to stack after calling bpf2bpf call or calling orig bpf
func in bpf trampoline.
Tests test_bpf.ko and test_verifier have passed, as well as the relative
testcases of test_progs*.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-7-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add RV_TAILCALL_OFFSET macro to format tailcall offset, and correct the
relevant comments.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-6-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When CONFIG_CFI_CLANG is enabled, prog->bpf_func already skips the kcfi
instruction during setup. Including it again in the tailcall jump offset
causes it to jump over an extra 4 bytes, skipping the stack pointer
adjustment, which will result in kernel stack corruption.
Fixes: 30a59cc797 ("riscv, bpf: Fix possible infinite tailcall when CONFIG_CFI_CLANG is enabled")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-5-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
It is unnecessary to allocate continuous physical memory for cache
buffer, and when ebpf program is too large, it may cause memory
allocation failure.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-4-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When bpf_int_jit_compile() is called for subprograms, it returns early
during the first pass (!prog->is_func || extra_pass is false), keeping
ctx->offset alive for the subsequent extra pass.
If JIT compilation fails for a later subprogram, the BPF core aborts
and calls bpf_jit_free() to clean up the first subprogram. However,
bpf_jit_free() fails to free jit_data->ctx.offset, which causes a
memory leak of the JIT context offsets array.
Fix this by adding the missing kfree(jit_data->ctx.offset) in
bpf_jit_free().
Fixes: 48a8f78c50 ("bpf, riscv: use prog pack allocator in the BPF JIT")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-3-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Extract the is_struct_ops_tramp helper, and use it in riscv
as the current checks are somewhat hacky.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-2-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
attach_probe/uprobe-lib and uprobe_autoattach selftests fail with "failed
to resolve full path for libc.so.6" on older non-usrmerged distros, where
libc.so.6 lives under a top-level /lib64 or /lib rather than /usr/lib64 or
/usr/lib. Add /lib64:/lib to the search paths, alongside the existing
/usr/lib64:/usr/lib and Debian multiarch entries.
Fixes: 1ce3a60e3c ("libbpf: auto-resolve programs/libraries when necessary for uprobes")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-3-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
progs/lsm_bdev.c keys its verity_devices hashmap with the raw kernel dev_t
read straight off bdev->bd_dev, i.e. MKDEV(major, minor) = (major << 20) |
minor. prog_tests/lsm_bdev.c instead builds its lookup key with dev_key =
(__u32)st.st_rdev from stat(2), but the stat(2) syscall fills st_rdev via
the kernel's new_encode_dev(), a different bit layout: (minor & 0xff) |
(major << 8) | ((minor & ~0xff) << 12).
For any device with a non-trivial major these two values differ, so the
lookup can never find what the BPF program stored, and test_lsm_bdev()
always fails with:
test_lsm_bdev:FAIL:map lookup unexpected error: -2 (errno 2)
Reconstruct the raw kernel dev_t from the decoded major/minor instead of
casting st_rdev directly, restoring the layout the BPF program actually
reads.
Fixes: 96f4c251a0 ("selftests/bpf: add block device management selftests")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-2-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
After "make install", test_progs fails because two files end up in the
wrong place:
- bpftool: TEST_GEN_PROGS_EXTENDED flattens it into $(INSTALL_PATH), losing
the tools/sbin/ prefix that detect_bpftool_path() expects. Remove it from
TEST_GEN_PROGS_EXTENDED and install it explicitly under tools/sbin/
instead.
- *.BTF: resolve_btfids writes resolve_btfids.test.o.BTF as a side-effect
of the build but INSTALL_RULE never copies it over. Install all *.BTF
files alongside the rest of the per-flavor output.
Fixes: f21fae5774 ("selftests/bpf: Add a few helpers for bpftool testing")
Fixes: 522397d05e ("resolve_btfids: Change in-place update with raw binary output")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-1-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Queue and stack pop/peek helpers accept an uninitialized output buffer
because the verifier expects the helper to initialize it. The empty-map
error path clears the buffer, but a failed lock acquisition returns
-EBUSY without writing it.
Clear the output before returning -EBUSY so BPF programs cannot observe
uninitialized stack contents after a failed helper call.
Fixes: a34a9f1a19 ("bpf: Avoid deadlock when using queue and stack maps from NMI")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260719125419.1782196-1-memxor@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
