Add the S[m|s]csrind ISA extension description.
Acked-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Atish Patra <atishp@rivosinc.com>
[pjw@kernel.org: use official extension names in the patch description]
Link: https://patch.msgid.link/20260807-counter_delegation-v9-4-58658104e487@meta.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
The S[m|s]csrind extensions extend the indirect CSR access mechanism
defined in Smaia/Ssaia extensions.
This patch just enables the definition and parsing.
Signed-off-by: Atish Patra <atishp@rivosinc.com>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
[pjw@kernel.org: use official RISC-V extension names in the patch description]
Link: https://patch.msgid.link/20260807-counter_delegation-v9-3-58658104e487@meta.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
This adds definitions of new CSRs and bits defined in the Smcsrind and
Sscsrind ISA extensions. These CSRs enable the indirect CSR accesses
mechanism to access any indirect CSRs in M-, S-, and VS-mode. The
range of the select values and ireg will be defined by the ISA
extension that are based on the Smcsrind and Sscsrind extensions.
Signed-off-by: Kaiwen Xue <kaiwenx@rivosinc.com>
Reviewed-by: Clément Léger <cleger@rivosinc.com>
Signed-off-by: Atish Patra <atishp@rivosinc.com>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
[pjw@kernel.org: clean up the patch description; use official RISC-V extension names]
Link: https://patch.msgid.link/20260807-counter_delegation-v9-2-58658104e487@meta.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Sashiko pointed out various UAF and memory leak issues around
pmu_sbi_device_probe() error paths.
If the probe fails, here are list of cleanups needed.
a. Already registered pmu must be freed
b. per cpu IRQ must be released
c. pmu_ctr_list data structure must be freed
d. cpu hotplug state must be cleaned up only if added.
Fix the resource cleanup by reorganizing the code around probe failure.
Reported-by: Sashiko AI <sashiko-bot@kernel.org>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Signed-off-by: Atish Patra <atishp@meta.com>
Link: https://patch.msgid.link/20260807-counter_delegation-v9-1-58658104e487@meta.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
This reverts commit 5d15d2ad36 ("riscv: hwprobe: Fix stale vDSO data for
late-initialized keys at boot"). The commit ensures synchronization between
the unaligned vector access speed probe kthread and vDSO data read. But now
that the kthread has been removed, this commit can be reverted.
Signed-off-by: Nam Cao <namcao@linutronix.de>
Tested-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
Link: https://patch.msgid.link/50ca78a649faf53f8941bc94c9cf8268b3644d38.1781666867.git.namcao@linutronix.de
Signed-off-by: Paul Walmsley <pjw@kernel.org>
A kthread is used to run check_vector_unaligned_access() to optimize boot
time, allowing the kernel to continue booting without waiting for the
unaligned vector speed probe to finish.
However, this asynchronous approach introduces several complications.
First, the kthread may not complete before a user reads vDSO data,
resulting in incorrect values. This was previously addressed by
commit 5d15d2ad36 ("riscv: hwprobe: Fix stale vDSO data for
late-initialized keys at boot"), which added complex synchronization
between the kthread and vDSO reads.
Second, it was discovered that the kthread may not finish before
vec_check_unaligned_access_speed_all_cpus() (marked with __init) is freed,
triggering a page fault.
These issues raise the question of whether the kthread is worth the added
complexity. A past boot time regression report was actually unrelated to
synchronous probing; it was caused by the probe running serially. Since
switching to a parallel probe, no further complaints have been made.
Furthermore, the unaligned scalar access speed probe takes the same amount
of time, runs synchronously, and has caused no issues.
Testing shows no noticeable boot time slowdown when running the vector
probe synchronously (0.464474s with kthread vs. 0.457991s without).
Remove the kthread usage and run the probe synchronously. This simplifies
the boot flow and allows for the revert of commit 5d15d2ad36 ("riscv:
hwprobe: Fix stale vDSO data for late-initialized keys at boot")
Reported-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
Closes: https://lore.kernel.org/linux-riscv/20260612-vec_unaligned_drop_init-v1-1-df969210ae34@oss.tenstorrent.com/
Fixes: e7c9d66e31 ("RISC-V: Report vector unaligned access speed hwprobe")
Cc: stable@vger.kernel.org
Signed-off-by: Nam Cao <namcao@linutronix.de>
Acked-by: Jesse Taube <jtaubepe@redhat.com>
Tested-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
Link: https://patch.msgid.link/1c378963f27c5960e8a57c50b8b444d30954cb54.1781666867.git.namcao@linutronix.de
[pjw@kernel.org: updated to apply; adjusted Fixes: tag; fixed my own manual patch application error]
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Since default_power_off() never returns, annotate it with the __noreturn
attribute to improve compiler optimizations.
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Link: https://patch.msgid.link/20260727100339.410466-2-thorsten.blum@linux.dev
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Change the shadow stack size calculation from RLIMIT_STACK/2 (capped at
2GB) to RLIMIT_STACK/8 (capped at 512MB), following David Laight's
analysis and recommendation.
Rationale:
David Laight pointed out that the focus should be on the ratio between
shadow stack size and the normal stack size, rather than just the
absolute upper limit. His analysis showed that while there are many
functions with small stack frames, the majority have stack deltas of
over 64 bytes due to saved registers and local variables.
Shadow stacks only store return addresses (8 bytes per entry on 64-bit
systems), whereas normal stack frames typically consume 64+ bytes. This
8:64 byte ratio means that programs using a lot of stack space are
dominated by large buffer allocations and local variables, not extreme
recursion depths with minimal local data.
For example, with the default RLIMIT_STACK of 8MB:
- RLIMIT_STACK/2 gives a 4MB shadow stack supporting 512K nested calls
- RLIMIT_STACK/8 gives a 1MB shadow stack supporting 128K nested calls
Given typical stack frame sizes of 64+ bytes, RLIMIT_STACK/8 is still
conservative and provides adequate depth for practical applications.
David noted that this could even be safely halved again.
This reduction also better accommodates memory-constrained platforms.
On systems with limited physical memory, allocating large shadow stacks
can cause virtual memory allocation failures when overcommit mode is set
to OVERCOMMIT_GUESS or OVERCOMMIT_NEVER.
Suggested-by: David Laight <david.laight.linux@gmail.com>
Link: https://lore.kernel.org/all/20260518105725.7afe7a4c@pumpkin/
Signed-off-by: Zong Li <zong.li@sifive.com>
Link: https://patch.msgid.link/20260522093634.3530233-1-zong.li@sifive.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Use Svinval in update_mmu_cache_range() when the extension is available.
Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
Link: https://patch.msgid.link/20260715132009.10634-3-luxu.kernel@bytedance.com
Tested-by: Klara Modin <klarasmodin@gmail.com>
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Add a test case validating that kprobes correctly simulates the
c.jal instruction on RV32.
The test uses two probe points: a forward c.jal and a backward
c.jal, and verifies that the containing function returns the
expected magic value KPROBE_TEST_MAGIC after kprobe interception.
Co-developed-by: Xiaofeng Yuan <xiaofengmian@163.com>
Signed-off-by: Nam Cao <namcao@linutronix.de>
Signed-off-by: Xiaofeng Yuan <xiaofengmian@163.com>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Link: https://patch.msgid.link/20260701081033.49871-3-xiaofengmian@163.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
The c.jal instruction is currently marked REJECTED in kprobes
instruction decoding, but it should be SIMULATED like other
compressed jump instructions.
Add simulate_c_jal() which saves the return address to RA and
sets the program counter to the target offset, reusing
simulate_c_j for the common jump logic.
Although c.jal is RV32-only, the function compiles unconditionally.
On RV64, riscv_insn_is_c_jal() always returns 0, so the simulation
code is never invoked and the small overhead in kernel size is
acceptable.
Signed-off-by: Xiaofeng Yuan <xiaofengmian@163.com>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Reviewed-by: Nam Cao <namcao@linutronix.de>
Tested-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/20260701081033.49871-2-xiaofengmian@163.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Implement the various required hooks and enable
ARCH_HAS_ACPI_TABLE_UPGRADE to allow use for ACPI_TABLE_UPGRADE, which
is useful for debugging ACPI table problems.
The implementation is based on arm64's of the same feature due to the
similarities of the requirements of the two platforms.
Signed-off-by: Vivian Wang <wangruikang@iscas.ac.cn>
Link: https://patch.msgid.link/20260616-riscv-acpi-table-upgrade-v1-2-45902d2dedf9@iscas.ac.cn
[pjw@kernel.org: updated to apply]
Signed-off-by: Paul Walmsley <pjw@kernel.org>
drivers/acpi/tables.c uses NR_FIX_BTMAPS without including
<asm/fixmap.h>. This isn't a problem for existing archs, but would be
when ARCH_HAS_ACPI_TABLE_UPGRADE is enabled for RISC-V. Add the missing
include.
Signed-off-by: Vivian Wang <wangruikang@iscas.ac.cn>
Link: https://patch.msgid.link/20260616-riscv-acpi-table-upgrade-v1-1-45902d2dedf9@iscas.ac.cn
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Firmware-preferred reset and EFI capsule update support requires reset
via EFI runtime services rather than direction M-mode firmware invocation
via SBI. Unlike poweroff, restart mechanism is directly controlled from
machine_restart function though.
Prefer the EFI runtime ResetSystem() service for restart when UEFI
runtime services are available.
Signed-off-by: Atish Patra <atishp@meta.com>
Reviewed-by: Sunil V L <sunilvl@oss.qualcomm.com>
Link: https://patch.msgid.link/20260615-efi_reset_shutdown-v1-2-9414edcbbab0@meta.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
When booted via UEFI with runtime services enabled, EFI Reset Shutdown is
the firmware-preferred shutdown path: it lets firmware run its own
shutdown hooks which may invoke SBI SRST extension internally. However,
RISC-V always powers off via the SBI SRST extension today and EFI runtime
path is never used even when firmware provides it.
Enable the poweroff via EFI by overriding efi_poweroff_required()
Signed-off-by: Atish Patra <atishp@meta.com>
Reviewed-by: Sunil V L <sunilvl@oss.qualcomm.com>
Link: https://patch.msgid.link/20260615-efi_reset_shutdown-v1-1-9414edcbbab0@meta.com
Signed-off-by: Paul Walmsley <pjw@kernel.org>
Ivan has been continuously active in DPLL development and discussion
since April 2025. His contributions cover the DPLL core and API,
netlink, bindings, ICE/SyncE integration, and the ZL3073x driver.
He also regularly reviews and tests DPLL patches from other contributors
and already maintains the Microchip ZL3073x driver. Add him as a reviewer
to reflect his ongoing involvement across the subsystem.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Acked-by: Vadim Fedorenko <vadim.fedorenko@linux.dev>
Acked-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>
Link: https://patch.msgid.link/20260806094432.163833-1-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Map the email addresses used for previous kernel contributions to the
current OpenAI address. This prevents get_maintainer.pl from listing
historical addresses as patch recipients.
Suggested-by: Matthieu Baerts <matttbe@kernel.org>
Signed-off-by: Christoph Paasch <cpaasch@openai.com>
Acked-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://patch.msgid.link/20260806-b4-mailman-v1-1-b4d7bc0ffd1c@openai.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Jiawen Wu says:
====================
net: wangxun: timeout and error
It is about adding the Tx timeout process and pci_error_handlers.
When a PCIe error occurs, the txgbe device is able to recover on platform
that support AER interrupt. And for Tx timeout, the txgbe driver can
recover the device by reset process.
For ngbe devices, due to the absence of the current function, it cannot
br fully recovered once there is a PCIe error or Tx timeout. Its
function will be completed in the future.
====================
Link: https://patch.msgid.link/20260803064334.21876-1-jiawenwu@trustnetic.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Support AER driver to handle the PCIe errors. Sometimes netdev watchdog
Tx timeout happens before the AER error report when a PCIe error occurs,
CPU blocking would be caused by MMIO during the reset process. To
prevent it, check PCIe error status in .ndo_tx_timeout. The current
function of ngbe is not yet fully developed, it will be completed in the
future.
Signed-off-by: Jiawen Wu <jiawenwu@trustnetic.com>
Link: https://patch.msgid.link/20260803064334.21876-6-jiawenwu@trustnetic.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Function wx_soft_quiesce() provide a lightweight shutdown path during
PCIe error recovery. It avoids MMIO-dependent operations in PCIe error
status.
Waiting for the service task to complete may unnecessarily delay PCIe
error recovery, especially if the work item is already blocked by the
hardware failure that triggered AER. So the service task is not
explicitly cancelled in quiesce path. As a measure to block the service
task, the checking of WX_STATE_DOWN and WX_STATE_RESETTING is added at
the entry of relevant work item.
Signed-off-by: Jiawen Wu <jiawenwu@trustnetic.com>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Link: https://patch.msgid.link/20260803064334.21876-5-jiawenwu@trustnetic.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
To implement a simple hardware reset without tearing down the network
interface state, introduce a boolean 'reinit' parameter to wx->do_reset
callback.
Signed-off-by: Jiawen Wu <jiawenwu@trustnetic.com>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Link: https://patch.msgid.link/20260803064334.21876-4-jiawenwu@trustnetic.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Implement .ndo_tx_timeout to handle Tx side timeout event. When a Tx
timeout event occur, it will trigger driver into reset process. And
allocate a separate work queue for reset process.
The WX_HANG_CHECK_ARMED bit is set to indicate a potential hang. It will
be cleared if a pause frame is received to avoid false hang detection
caused by pause frames.
Signed-off-by: Jiawen Wu <jiawenwu@trustnetic.com>
Link: https://patch.msgid.link/20260803064334.21876-3-jiawenwu@trustnetic.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
In mmio_print_mark(), a raw C cast (struct print_entry *)entry is used to
obtain the print_entry pointer.
Use the standard trace_assign_type() macro instead, matching the usage in
mmio_print_rw() and mmio_print_map().
Link: https://patch.msgid.link/178524301013.56416.9116249028160618790.stgit@devnote2
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Simon Schippers says:
====================
tun/tap & vhost-net: apply qdisc backpressure on full ptr_ring to reduce TX drops
This patch series deals with tun/tap & vhost-net which drop incoming
SKBs whenever their internal ptr_ring buffer is full. Instead, with this
patch series, the associated netdev queue is stopped, but only when the
new IFF_BACKPRESSURE flag is set and a qdisc is attached. Without the
flag, or if no qdisc is present, the existing behavior is preserved. The
XDP transmit path is not affected. This patch series touches tun/tap and
vhost-net, as they share common logic and must be updated together.
Modifying only one of them would break the other.
By applying proper backpressure, this change allows the connected qdisc to
operate correctly, as reported in [1], and significantly improves
performance in real-world scenarios, as demonstrated in our paper [2]. For
example, we observed a 36% TCP throughput improvement for an OpenVPN
connection between Germany and the USA.
The previous version of this work was applied and then reverted in 7.2,
because the backpressure was unconditional: it caused a significant
throughput drop in an IPv6 multicast testcase with multiple iperf3 TCP
threads sending on Brett Sheffield's librecast testbed [3]. This version
therefore makes the behavior opt-in via IFF_BACKPRESSURE from the very
first patch, so that a tun/tap device which does not set the flag behaves
exactly as before.
The series is ordered so that no patch changes how packets are handled
unless the flag is set:
- Patch 1 adds the flag only. It has no effect yet: TUNSETIFF silently
masks it off, as it does for any flag outside TUN_FEATURES, until
patch 5 adds it there.
- Patches 2 and 3 add the consumer side, which wakes a stopped netdev
queue. __tun_wake_queue() returns early unless IFF_BACKPRESSURE is set,
and no queue is stopped at this point anyway.
- Patch 4 is a pure ptr_ring refactor required by patch 5.
- Patch 5 adds the queue stopping, gated on IFF_BACKPRESSURE, together
with the wake needed when the flag is cleared again, and only there is
the flag added to TUN_FEATURES.
That way no intermediate commit changes the behavior of an existing
tun/tap user beyond the added checks, and bisecting inside the series can
not hit the regression that led to the revert.
[1] https://unix.stackexchange.com/questions/762935/traffic-shaping-ineffective-on-tun-device
[2] https://cni.etit.tu-dortmund.de/storages/cni-etit/r/Research/Publications/2025/Gebauer_2025_VTCFall/Gebauer_VTCFall2025_AuthorsVersion.pdf
[3] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
====================
Link: https://patch.msgid.link/20260803183641.96882-1-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.
IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.
The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.
A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.
The documentation in tuntap.rst is updated accordingly.
Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
This patch moves the check for available free space for a new entry into
a separate function. Existing callers that only check for a non-zero
return value are unaffected. __ptr_ring_produce() now returns -EINVAL
for a zero-size ring and -ENOSPC when full, whereas before both cases
returned -ENOSPC. The new helper allows callers to determine in advance
whether a single subsequent __ptr_ring_produce() call will succeed. This
information can, for example, be used to temporarily stop producing until
__ptr_ring_check_produce() indicates that space is available again.
The return values are documented above the helper, as a caller that waits
for space must distinguish the transient -ENOSPC from the permanent
-EINVAL.
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-5-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add tun_wake_queue() to tun.c and export it for use by vhost-net. The
function validates that the file belongs to a device implemented by
drivers/net/tun.c, in IFF_TUN as well as in IFF_TAP mode, and that the
tfile exists, dereferences the tun_struct under RCU, and delegates to
__tun_wake_queue().
vhost_net_buf_produce() now calls tun_wake_queue() after a successful
batched consume of the ring to allow the netdev subqueue to be woken up.
The point is to allow the queue to be stopped when it gets full, which is
required for traffic shaping, implemented by the following
"stop tail-drop when IFF_BACKPRESSURE is set".
As __tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, a
tun/tap device that does not opt in only pays for the added check.
macvtap and ipvtap rings, which get_tap_ptr_ring() accepts too, are
unaffected: their producer is the tap_handle_frame() rx_handler and not
ndo_start_xmit, so stopping a netdev TX queue would not hold it back.
drivers/net/tap.c has no netdev_ops of its own either. No
tap_wake_queue() is needed.
cons_cnt and the wake decision are best-effort and are not reverted by
ptr_ring_unconsume(), so vhost_net_buf_unproduce() can leave the subqueue
woken over a full ring. The producer re-stops it on the next packet, and
that path only runs from vhost_net_stop_vq() and vhost_net_set_backend(),
when the consumer is going away, so a stopped queue is the correct end
state rather than a stall.
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-4-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add the IFF_BACKPRESSURE flag to the UAPI header and to its tools/ copy.
The flag has no effect yet, it is the opt-in switch for the qdisc
backpressure logic added by the following patches.
It is added to TUN_FEATURES only in the last patch of the series, once the
implementation is complete. Until then TUNSETIFF silently masks it off, as
it does for any flag outside TUN_FEATURES.
Keeping the flag and its users in separate patches would either leave a
window where backpressure is unconditional, or make the opt-in a later
add-on. Adding the flag first lets every following patch be a no-op
unless it is set.
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-2-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
In tc_new_tfilter() the create branch sets tp_created = 1 before calling
tcf_chain_tp_insert_unique(). When the caller loses the race (another
request inserted a proto at the same chain/prio first), insert_unique()
destroys the caller's own tp_new and returns the winner's proto with an
extra reference. tp_created was never cleared, so the loser's errout
path treated the winner's live proto as its own and called
tcf_chain_tp_delete_empty() on it, silently unlinking an active
classifier that the winning request already advertised via
RTM_NEWTFILTER.
Track the outcome of the insert step in a single tri-state variable so
each errout path reacts correctly:
- TP_NOT_CREATED: no proto created; pursue the old path.
- TP_CREATED: proto inserted successfully; same code path as before.
- TP_NOT_OWNED: New - lost the insert race; tp is another request's proto
(chain ref already released by tp_new's destroy)
Both errout reactions are single expressions derived from the state.
This fix is motivated by the Sashiko's automated review of Patch
(net/sched: cls_api: Always acquire rtnl_lock when destroying locked
classifiers) [1][2]. The review identified the silent-unlink behaviour of
an adopted proto's teardown when a request loses the
tcf_chain_tp_insert_unique() race.
[1] https://sashiko.dev/#/patchset/20260801125632.360365-1-jhs%40mojatatu.com
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260801125632.360365-1-jhs%40mojatatu.com
Fixes: 8b64678e0a ("net: sched: refactor tp insert/delete for concurrent execution")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260801125632.360365-1-jhs%40mojatatu.com
Closes: https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260801125632.360365-1-jhs%40mojatatu.com
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: Victor Nogueira <victor@mojatatu.com>
Reported-by: TencentOS Corvus AI <corvus@tencent.com>
Tested-by: Aohan Mei <henrymei@tencent.com>
Link: https://patch.msgid.link/20260805134049.927864-1-victor@mojatatu.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Sun Jian says:
====================
fix skb length accounting after XDP frag adjustment
This series fixes skb length accounting after an XDP program adjusts its
fragment area, in both the generic XDP path (net/core/dev.c) and the veth
native path (drivers/net/veth.c). When the fragment area is resized,
skb->len and skb->data_len can go out of sync, and in the reproduced UDP
receive path this leaked skb_shared_info contents (including a kernel
pointer) to userspace while truncating real payload.
====================
Link: https://patch.msgid.link/20260804054040.613675-1-sun.jian.kdev@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
veth exposes non-linear skb fragments through an xdp_buff. If an XDP
program adjusts the fragment area, veth_xdp_rcv_skb() copies
xdp_frags_size back to skb->data_len but leaves skb->len containing the
old fragment contribution.
After a fragment shrink, this makes skb_headlen() larger than the actual
linear area. In the reproduced UDP receive path, __skb_datagram_iter()
copied 1024 bytes past the actual linear tail to userspace, starting at
struct skb_shared_info. The copied bytes included the affected skb's
nr_frags, xdp_frags_size, and a kernel pointer from
skb_shinfo(skb)->frags[0]. Real packet data was displaced by the same
amount and truncated at the end.
Subtract the old data_len before replacing it and add the new data_len
afterwards, keeping skb->len and skb->data_len synchronized.
Additionally, bpf_xdp_pull_data() can advance data_end while leaving
frags present. The skb is then still non-linear, so the old
__skb_put(skb, off) triggers SKB_LINEAR_ASSERT().
Use skb_set_tail_pointer() and update skb->len explicitly instead,
following bpf_prog_run_generic_xdp(). Unlike __skb_put(),
skb_set_tail_pointer() does not require a linear skb.
A 60000-byte UDP datagram on a veth pair with MTU 64000 was shortened by
1024 bytes from its fragment area. Before the fix, all 10 runs produced
corrupted payloads. After the fix, all 10 runs matched the expected
payload exactly. A forced-tailroom reproducer also exercises
bpf_xdp_pull_data() with frags still present; the old code triggers
SKB_LINEAR_ASSERT(), while this fix passes 10/10 runs.
Fixes: 718a18a0c8 ("veth: Rework veth_xdp_rcv_skb in order to accept non-linear skb")
Cc: stable@vger.kernel.org
Reported-by: Mohsin Bashir <mohsin.bashr@gmail.com>
Link: https://lore.kernel.org/bpf/80687d9c-9c27-494c-b3f2-efd0230b1895@gmail.com/
Suggested-by: Lorenzo Bianconi <lorenzo@kernel.org>
Acked-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Sun Jian <sun.jian.kdev@gmail.com>
Link: https://patch.msgid.link/20260804054040.613675-3-sun.jian.kdev@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Generic XDP exposes non-linear skb fragments through an xdp_buff. If an
XDP program adjusts the fragment area, bpf_prog_run_generic_xdp() copies
xdp_frags_size back to skb->data_len but leaves skb->len containing the
old fragment contribution.
After a fragment shrink, this makes skb_headlen() larger than the actual
linear area. In the reproduced UDP receive path, __skb_datagram_iter()
copied 1024 bytes past the actual linear tail to userspace, starting at
struct skb_shared_info. The copied bytes included the affected skb's
nr_frags, xdp_frags_size and a kernel pointer from
skb_shinfo(skb)->frags[0]. Real packet data was displaced by the same
amount and truncated at the end.
Subtract the old data_len before replacing it and add the new data_len
afterwards, keeping skb->len and skb->data_len synchronized.
A 60000-byte UDP datagram on a veth pair with MTU 64000 was shortened by
1024 bytes from its fragment area. Before the fix, all 10 runs produced
corrupted payloads. After the fix, all 10 runs matched the expected
payload exactly.
Fixes: e6d5dbdd20 ("xdp: add multi-buff support for xdp running in generic mode")
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/al9T9Eto%2FhRIzP5W@boxer/
Reviewed-by: Mohsin Bashir <hmohsin@meta.com>
Acked-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Sun Jian <sun.jian.kdev@gmail.com>
Link: https://patch.msgid.link/20260804054040.613675-2-sun.jian.kdev@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Mohsin Bashir says:
====================
selftests: net: shaper: Expand shaper API coverage
Add more net shaper selftest coverage for group operations and nested
node management.
The series first prepares shared cleanup and capability helpers, and
separates the basic netdev grouping coverage from the rate-limited
variant. It then adds tests for updating node shaper rates through both
.set and .group, discovering the supported nesting depth, deleting child
nodes and reparenting their leaves, moving queues between nodes, and
rejecting node reparenting.
Further patches broaden API coverage: exercising the full set of scalar
shaper attributes, rejecting invalid .set requests while leaving the
existing configuration intact, grouping leaves drawn from different
parents (which requires an explicit parent), and recursively cleaning up
nodes left empty.
The new tests use the capability helper to skip unsupported devices
instead of depending on earlier test ordering, size their queue
requirements from the number of TX queues exposed in sysfs, and register
cleanup for created shapers as soon as the operation succeeds.
TAP version 13
1..22
ok 1 shaper.get_shapers
ok 2 shaper.get_caps
ok 3 shaper.set_qshapers
ok 4 shaper.del_qshapers
ok 5 shaper.set_nshapers
ok 6 shaper.del_nshapers
ok 7 shaper.set_all_supported_attrs
ok 8 shaper.invalid_set_preserves_state
ok 9 shaper.mixed_parent_group_requires_parent
ok 10 shaper.recursive_empty_node_cleanup
ok 11 shaper.basic_groups
ok 12 shaper.basic_groups_with_rate
ok 13 shaper.qgroups
ok 14 shaper.set_node_shaper
ok 15 shaper.group_update_rate
ok 16 shaper.delegation
ok 17 shaper.nested_depth_limit
ok 18 shaper.delete_child_reparent
ok 19 shaper.move_queue_between_nodes
ok 20 shaper.reject_reparenting
ok 21 shaper.dup_leaves
ok 22 shaper.queue_update
====================
Link: https://patch.msgid.link/20260805030936.1092907-1-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Exercise cleanup of nested nodes after deleting their last queue leaf. The
test builds a two-level node hierarchy and checks that removing the queue
also removes both now-empty node shapers.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-15-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add coverage for grouping leaves that currently belong to different parent
nodes. The test verifies that an implicit parent is rejected, an explicit
parent succeeds, and the old empty parent nodes are cleaned up.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-14-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Verify that invalid set requests fail without corrupting existing queue
shaper state. The test covers invalid node creation through set and invalid
queue identifiers, then confirms the original queue configuration remains
unchanged.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-13-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Exercise queue-scope scalar shaper attributes reported by the device,
including rate limits, burst, priority and weight. Build the set request
from advertised capabilities so devices are tested for the attributes they
claim rather than skipped for missing unrelated fields.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-12-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add reject_reparenting to verify that the group operation rejects attempts
to change an existing node's parent. The test creates two node shapers
under netdev and verifies that re-grouping the first node under the second
fails with EOPNOTSUPP. It also verifies that updating the node with the
same parent succeeds, and that updating the node without specifying a
parent keeps the queue leaves under the original node while updating their
weights.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-11-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add move_queue_between_nodes to verify that a queue can be moved
from one node to another via re-grouping. Creates N1 with Q1,Q2
and N2 with Q3, then re-groups N2 with Q1,Q3 to steal Q1 from
N1. Verifies Q1 moved to N2 and Q2 remains under N1.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-10-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add delete_child_reparent to verify that deleting a child node
reparents its queue leaves to the parent node. Creates a two-level
hierarchy (N1 with Q1,Q2 and child N2 with Q3), deletes N2, and
verifies Q3's parent becomes N1.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-9-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add nested_depth_limit to incrementally create deeper nesting
levels until the driver rejects. Reports the maximum supported
nesting depth on both pass and fail. A device advertising nesting
support must support at least depth 2, otherwise nesting is
meaningless.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-8-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add group_update_rate to test updating an existing node's rate
via the .group callback. Creates a node with bw_max=10000,
re-groups with bw_max=50000, and verifies the rate changed while
leaves remain under the same node.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-7-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add set_node_shaper to test updating a NODE scope shaper's rate
via the .set callback. Creates a node group with bw_max=10000,
updates to 20000 via .set, and verifies the change.
Signed-off-by: Mohsin Bashir <hmohsin@meta.com>
Link: https://patch.msgid.link/20260805030936.1092907-6-mohsin.bashr@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
