2019-05-27 08:55:06 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-01 17:40:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-07-12 19:33:09 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-02 07:18:10 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-02-02 19:15:33 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-25 13:00:33 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2024-01-25 16:25:11 -08:00
|
|
|
|
2008-04-16 00:41:16 -07:00
|
|
|
|
2008-07-03 03:48:02 -07:00
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
2007-09-17 11:56:21 -07:00
|
|
|
|
2008-04-16 00:40:46 -07:00
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
xdp: change ndo_xdp_xmit API to support bulking
This patch change the API for ndo_xdp_xmit to support bulking
xdp_frames.
When kernel is compiled with CONFIG_RETPOLINE, XDP sees a huge slowdown.
Most of the slowdown is caused by DMA API indirect function calls, but
also the net_device->ndo_xdp_xmit() call.
Benchmarked patch with CONFIG_RETPOLINE, using xdp_redirect_map with
single flow/core test (CPU E5-1650 v4 @ 3.60GHz), showed
performance improved:
for driver ixgbe: 6,042,682 pps -> 6,853,768 pps = +811,086 pps
for driver i40e : 6,187,169 pps -> 6,724,519 pps = +537,350 pps
With frames avail as a bulk inside the driver ndo_xdp_xmit call,
further optimizations are possible, like bulk DMA-mapping for TX.
Testing without CONFIG_RETPOLINE show the same performance for
physical NIC drivers.
The virtual NIC driver tun sees a huge performance boost, as it can
avoid doing per frame producer locking, but instead amortize the
locking cost over the bulk.
V2: Fix compile errors reported by kbuild test robot <lkp@intel.com>
V4: Isolated ndo, driver changes and callers.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-05-24 16:46:12 +02:00
|
|
|
|
2020-06-29 19:06:22 -06:00
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2021-04-06 18:45:54 +01:00
|
|
|
|
2024-09-30 14:29:53 +01:00
|
|
|
|
2021-04-06 18:45:54 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-06 16:00:30 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2016-12-24 11:46:01 -08:00
|
|
|
|
2018-02-14 16:40:14 +03:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2025-02-07 15:10:55 +09:00
|
|
|
|
|
|
|
|
|
2018-06-02 17:49:53 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-04 11:36:08 +08:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
2014-11-19 14:44:40 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set
This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.
IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.
The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.
A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.
The documentation in tuntap.rst is updated accordingly.
Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:41 +02:00
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-01-12 11:41:29 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-23 03:59:13 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2013-12-05 20:42:58 -08:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
2012-11-25 22:07:40 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
|
|
|
|
|
2012-11-25 22:07:40 +00:00
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun: Add ability to create tun device with given index
Tun devices cannot be created with ifidex user wants, but it's
required by checkpoint-restore project.
Long time ago such ability was implemented for rtnl_ops-based
interface for creating links (9c7dafbf net: Allow to create links
with given ifindex), but the only API for creating and managing
tuntap devices is ioctl-based and is evolving with adding new ones
(cde8b15f tuntap: add ioctl to attach or detach a file form tuntap
device).
Following that trend, here's how a new ioctl that sets the ifindex
for device, that _will_ be created by TUNSETIFF ioctl looks like.
So those who want a tuntap device with the ifindex N, should open
the tun device, call ioctl(fd, TUNSETIFINDEX, &N), then call TUNSETIFF.
If the index N is busy, then the register_netdev will find this out
and the ioctl would be failed with -EBUSY.
If setifindex is not called, then it will be generated as before.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-21 14:31:38 +04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2017-10-18 12:12:09 -07:00
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
|
|
|
|
|
2018-01-03 11:25:59 +01:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
|
|
|
|
|
2018-11-15 17:43:10 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2018-12-06 16:08:17 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-08-03 15:50:02 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2012-11-25 22:07:40 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
2008-04-12 18:48:58 -07:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2012-02-07 16:48:55 -08:00
|
|
|
|
|
|
|
|
|
2008-04-12 18:48:58 -07:00
|
|
|
|
|
|
|
|
|
2011-11-15 15:29:55 +00:00
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
2010-03-17 17:45:01 +02:00
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
2010-03-17 17:45:01 +02:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
2013-01-14 07:12:19 +00:00
|
|
|
|
2013-01-23 03:59:13 +00:00
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
2018-06-02 17:49:53 -04:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-04-12 18:48:58 -07:00
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-04-12 18:48:58 -07:00
|
|
|
|
|
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2017-10-18 12:12:09 -07:00
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2022-05-04 09:37:24 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-06-22 21:21:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-28 14:51:47 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2017-10-18 12:12:09 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-28 14:51:47 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2017-10-18 12:12:09 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
2018-08-03 15:50:02 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
treewide: Replace kmalloc with kmalloc_obj for non-scalar types
This is the result of running the Coccinelle script from
scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to
avoid scalar types (which need careful case-by-case checking), and
instead replace kmalloc-family calls that allocate struct or union
object instances:
Single allocations: kmalloc(sizeof(TYPE), ...)
are replaced with: kmalloc_obj(TYPE, ...)
Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...)
are replaced with: kmalloc_objs(TYPE, COUNT, ...)
Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...)
are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...)
(where TYPE may also be *VAR)
The resulting allocations no longer return "void *", instead returning
"TYPE *".
Signed-off-by: Kees Cook <kees@kernel.org>
2026-02-20 23:49:23 -08:00
|
|
|
|
2012-12-21 07:17:21 +00:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-23 03:59:13 +00:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2012-12-21 07:17:21 +00:00
|
|
|
|
2013-01-23 03:59:13 +00:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
treewide: setup_timer() -> timer_setup()
This converts all remaining cases of the old setup_timer() API into using
timer_setup(), where the callback argument is the structure already
holding the struct timer_list. These should have no behavioral changes,
since they just change which pointer is passed into the callback with
the same available pointers after conversion. It handles the following
examples, in addition to some other variations.
Casting from unsigned long:
void my_callback(unsigned long data)
{
struct something *ptr = (struct something *)data;
...
}
...
setup_timer(&ptr->my_timer, my_callback, ptr);
and forced object casts:
void my_callback(struct something *ptr)
{
...
}
...
setup_timer(&ptr->my_timer, my_callback, (unsigned long)ptr);
become:
void my_callback(struct timer_list *t)
{
struct something *ptr = from_timer(ptr, t, my_timer);
...
}
...
timer_setup(&ptr->my_timer, my_callback, 0);
Direct function assignments:
void my_callback(unsigned long data)
{
struct something *ptr = (struct something *)data;
...
}
...
ptr->my_timer.function = my_callback;
have a temporary cast added, along with converting the args:
void my_callback(struct timer_list *t)
{
struct something *ptr = from_timer(ptr, t, my_timer);
...
}
...
ptr->my_timer.function = (TIMER_FUNC_TYPE)my_callback;
And finally, callbacks without a data assignment:
void my_callback(unsigned long data)
{
...
}
...
setup_timer(&ptr->my_timer, my_callback, 0);
have their argument renamed to verify they're unused during conversion:
void my_callback(struct timer_list *unused)
{
...
}
...
timer_setup(&ptr->my_timer, my_callback, 0);
The conversion is done with the following Coccinelle script:
spatch --very-quiet --all-includes --include-headers \
-I ./arch/x86/include -I ./arch/x86/include/generated \
-I ./include -I ./arch/x86/include/uapi \
-I ./arch/x86/include/generated/uapi -I ./include/uapi \
-I ./include/generated/uapi --include ./include/linux/kconfig.h \
--dir . \
--cocci-file ~/src/data/timer_setup.cocci
@fix_address_of@
expression e;
@@
setup_timer(
-&(e)
+&e
, ...)
// Update any raw setup_timer() usages that have a NULL callback, but
// would otherwise match change_timer_function_usage, since the latter
// will update all function assignments done in the face of a NULL
// function initialization in setup_timer().
@change_timer_function_usage_NULL@
expression _E;
identifier _timer;
type _cast_data;
@@
(
-setup_timer(&_E->_timer, NULL, _E);
+timer_setup(&_E->_timer, NULL, 0);
|
-setup_timer(&_E->_timer, NULL, (_cast_data)_E);
+timer_setup(&_E->_timer, NULL, 0);
|
-setup_timer(&_E._timer, NULL, &_E);
+timer_setup(&_E._timer, NULL, 0);
|
-setup_timer(&_E._timer, NULL, (_cast_data)&_E);
+timer_setup(&_E._timer, NULL, 0);
)
@change_timer_function_usage@
expression _E;
identifier _timer;
struct timer_list _stl;
identifier _callback;
type _cast_func, _cast_data;
@@
(
-setup_timer(&_E->_timer, _callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, &_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, &_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)&_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)&_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, &_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, &_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)&_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)&_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
_E->_timer@_stl.function = _callback;
|
_E->_timer@_stl.function = &_callback;
|
_E->_timer@_stl.function = (_cast_func)_callback;
|
_E->_timer@_stl.function = (_cast_func)&_callback;
|
_E._timer@_stl.function = _callback;
|
_E._timer@_stl.function = &_callback;
|
_E._timer@_stl.function = (_cast_func)_callback;
|
_E._timer@_stl.function = (_cast_func)&_callback;
)
// callback(unsigned long arg)
@change_callback_handle_cast
depends on change_timer_function_usage@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _origtype;
identifier _origarg;
type _handletype;
identifier _handle;
@@
void _callback(
-_origtype _origarg
+struct timer_list *t
)
{
(
... when != _origarg
_handletype *_handle =
-(_handletype *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle =
-(void *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle;
... when != _handle
_handle =
-(_handletype *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle;
... when != _handle
_handle =
-(void *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
)
}
// callback(unsigned long arg) without existing variable
@change_callback_handle_cast_no_arg
depends on change_timer_function_usage &&
!change_callback_handle_cast@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _origtype;
identifier _origarg;
type _handletype;
@@
void _callback(
-_origtype _origarg
+struct timer_list *t
)
{
+ _handletype *_origarg = from_timer(_origarg, t, _timer);
+
... when != _origarg
- (_handletype *)_origarg
+ _origarg
... when != _origarg
}
// Avoid already converted callbacks.
@match_callback_converted
depends on change_timer_function_usage &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg@
identifier change_timer_function_usage._callback;
identifier t;
@@
void _callback(struct timer_list *t)
{ ... }
// callback(struct something *handle)
@change_callback_handle_arg
depends on change_timer_function_usage &&
!match_callback_converted &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _handletype;
identifier _handle;
@@
void _callback(
-_handletype *_handle
+struct timer_list *t
)
{
+ _handletype *_handle = from_timer(_handle, t, _timer);
...
}
// If change_callback_handle_arg ran on an empty function, remove
// the added handler.
@unchange_callback_handle_arg
depends on change_timer_function_usage &&
change_callback_handle_arg@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _handletype;
identifier _handle;
identifier t;
@@
void _callback(struct timer_list *t)
{
- _handletype *_handle = from_timer(_handle, t, _timer);
}
// We only want to refactor the setup_timer() data argument if we've found
// the matching callback. This undoes changes in change_timer_function_usage.
@unchange_timer_function_usage
depends on change_timer_function_usage &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg &&
!change_callback_handle_arg@
expression change_timer_function_usage._E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type change_timer_function_usage._cast_data;
@@
(
-timer_setup(&_E->_timer, _callback, 0);
+setup_timer(&_E->_timer, _callback, (_cast_data)_E);
|
-timer_setup(&_E._timer, _callback, 0);
+setup_timer(&_E._timer, _callback, (_cast_data)&_E);
)
// If we fixed a callback from a .function assignment, fix the
// assignment cast now.
@change_timer_function_assignment
depends on change_timer_function_usage &&
(change_callback_handle_cast ||
change_callback_handle_cast_no_arg ||
change_callback_handle_arg)@
expression change_timer_function_usage._E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type _cast_func;
typedef TIMER_FUNC_TYPE;
@@
(
_E->_timer.function =
-_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-&_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-(_cast_func)_callback;
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-(_cast_func)&_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-&_callback;
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-(_cast_func)_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-(_cast_func)&_callback
+(TIMER_FUNC_TYPE)_callback
;
)
// Sometimes timer functions are called directly. Replace matched args.
@change_timer_function_calls
depends on change_timer_function_usage &&
(change_callback_handle_cast ||
change_callback_handle_cast_no_arg ||
change_callback_handle_arg)@
expression _E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type _cast_data;
@@
_callback(
(
-(_cast_data)_E
+&_E->_timer
|
-(_cast_data)&_E
+&_E._timer
|
-_E
+&_E->_timer
)
)
// If a timer has been configured without a data argument, it can be
// converted without regard to the callback argument, since it is unused.
@match_timer_function_unused_data@
expression _E;
identifier _timer;
identifier _callback;
@@
(
-setup_timer(&_E->_timer, _callback, 0);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, 0L);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, 0UL);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0L);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0UL);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0L);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0UL);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0);
+timer_setup(_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0L);
+timer_setup(_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0UL);
+timer_setup(_timer, _callback, 0);
)
@change_callback_unused_data
depends on match_timer_function_unused_data@
identifier match_timer_function_unused_data._callback;
type _origtype;
identifier _origarg;
@@
void _callback(
-_origtype _origarg
+struct timer_list *unused
)
{
... when != _origarg
}
Signed-off-by: Kees Cook <keescook@chromium.org>
2017-10-16 14:43:17 -07:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2025-05-09 07:51:14 +02:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-10-20 11:29:55 -07:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2017-10-20 11:29:56 -07:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2017-10-20 11:29:56 -07:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2017-10-20 11:29:56 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-10-20 11:29:55 -07:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
2012-12-12 19:22:57 +00:00
|
|
|
|
2013-01-28 01:05:19 +00:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-28 01:05:19 +00:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2018-12-06 16:28:11 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-10-09 09:20:02 -07:00
|
|
|
|
|
|
|
|
|
2018-12-06 16:08:17 +08:00
|
|
|
|
|
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
2013-01-23 03:59:13 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-03-04 17:23:59 +01:00
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: rfs: add hash collision detection
Receive Flow Steering is a nice solution but suffers from
hash collisions when a mix of connected and unconnected traffic
is received on the host, when flow hash table is populated.
Also, clearing flow in inet_release() makes RFS not very good
for short lived flows, as many packets can follow close().
(FIN , ACK packets, ...)
This patch extends the information stored into global hash table
to not only include cpu number, but upper part of the hash value.
I use a 32bit value, and dynamically split it in two parts.
For host with less than 64 possible cpus, this gives 6 bits for the
cpu number, and 26 (32-6) bits for the upper part of the hash.
Since hash bucket selection use low order bits of the hash, we have
a full hash match, if /proc/sys/net/core/rps_sock_flow_entries is big
enough.
If the hash found in flow table does not match, we fallback to RPS (if
it is enabled for the rxqueue).
This means that a packet for an non connected flow can avoid the
IPI through a unrelated/victim CPU.
This also means we no longer have to clear the table at socket
close time, and this helps short lived flows performance.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Tom Herbert <therbert@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-02-06 12:59:01 -08:00
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-10-09 10:32:04 +08:00
|
|
|
|
2013-12-05 20:42:58 -08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
2018-10-09 10:32:04 +08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2024-01-25 16:25:11 -08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE()
Please do not apply this to mainline directly, instead please re-run the
coccinelle script shown below and apply its output.
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't harmful, and changing them results in
churn.
However, for some features, the read/write distinction is critical to
correct operation. To distinguish these cases, separate read/write
accessors must be used. This patch migrates (most) remaining
ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following
coccinelle script:
----
// Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and
// WRITE_ONCE()
// $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davem@davemloft.net
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-23 14:07:29 -07:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2017-06-06 14:09:49 +08:00
|
|
|
|
2018-10-09 10:32:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-01-25 16:25:11 -08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2019-05-08 23:20:17 -04:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2019-05-08 23:20:17 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-05-08 23:20:17 -04:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-20 11:02:06 +01:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:10:06 -05:00
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
|
|
|
|
|
2012-11-18 21:34:11 +00:00
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
2025-02-04 11:10:06 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
2012-12-18 11:00:27 +08:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-09 14:50:34 +08:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
xdp: transition into using xdp_frame for return API
Changing API xdp_return_frame() to take struct xdp_frame as argument,
seems like a natural choice. But there are some subtle performance
details here that needs extra care, which is a deliberate choice.
When de-referencing xdp_frame on a remote CPU during DMA-TX
completion, result in the cache-line is change to "Shared"
state. Later when the page is reused for RX, then this xdp_frame
cache-line is written, which change the state to "Modified".
This situation already happens (naturally) for, virtio_net, tun and
cpumap as the xdp_frame pointer is the queued object. In tun and
cpumap, the ptr_ring is used for efficiently transferring cache-lines
(with pointers) between CPUs. Thus, the only option is to
de-referencing xdp_frame.
It is only the ixgbe driver that had an optimization, in which it can
avoid doing the de-reference of xdp_frame. The driver already have
TX-ring queue, which (in case of remote DMA-TX completion) have to be
transferred between CPUs anyhow. In this data area, we stored a
struct xdp_mem_info and a data pointer, which allowed us to avoid
de-referencing xdp_frame.
To compensate for this, a prefetchw is used for telling the cache
coherency protocol about our access pattern. My benchmarks show that
this prefetchw is enough to compensate the ixgbe driver.
V7: Adjust for commit d9314c474d4f ("i40e: add support for XDP_REDIRECT")
V8: Adjust for commit bd658dda4237 ("net/mlx5e: Separate dma base address
and offset in dma_sync call")
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 16:46:32 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-09 14:50:34 +08:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2013-09-05 17:53:59 +08:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
2013-09-05 17:53:59 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2022-06-29 11:19:10 -07:00
|
|
|
|
|
|
|
|
|
2018-09-28 14:51:47 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
|
|
|
|
|
2013-01-28 01:05:19 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2024-02-20 11:12:07 +08:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
|
|
|
|
|
2019-05-08 23:20:18 -04:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2013-01-28 01:05:19 +00:00
|
|
|
|
2014-03-24 00:02:32 +05:30
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2022-06-22 21:21:05 -07:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2022-06-22 21:21:05 -07:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2013-09-05 17:53:59 +08:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2013-01-11 16:59:34 +00:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2013-01-11 16:59:34 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
2013-01-28 00:38:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2013-01-28 00:38:02 +00:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2013-01-28 00:38:02 +00:00
|
|
|
|
2018-05-11 10:49:25 +08:00
|
|
|
|
|
|
|
|
|
2018-05-16 20:39:33 +08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2022-11-25 02:51:34 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2018-09-28 14:51:47 -07:00
|
|
|
|
2016-05-19 13:36:51 +08:00
|
|
|
|
2014-05-16 15:11:48 -07:00
|
|
|
|
2014-03-24 00:02:32 +05:30
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2013-01-28 01:05:19 +00:00
|
|
|
|
2016-05-19 13:36:51 +08:00
|
|
|
|
2014-05-16 15:11:48 -07:00
|
|
|
|
2014-03-24 00:02:32 +05:30
|
|
|
|
2013-01-28 01:05:19 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
2018-09-28 14:51:47 -07:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2013-09-05 17:53:59 +08:00
|
|
|
|
2018-05-11 10:49:25 +08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2022-06-22 21:20:39 -07:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2013-09-05 17:53:59 +08:00
|
|
|
|
2018-05-11 10:49:25 +08:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-11 16:59:34 +00:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2013-01-11 16:59:34 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2019-09-10 18:56:57 +08:00
|
|
|
|
|
|
|
|
|
2009-01-20 10:57:48 +00:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2009-01-20 11:02:28 +00:00
|
|
|
|
2009-01-20 10:57:48 +00:00
|
|
|
|
2013-01-14 07:12:19 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:02:28 +00:00
|
|
|
|
2013-01-28 01:05:19 +00:00
|
|
|
|
2009-01-20 11:02:28 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
2009-01-20 11:02:28 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2013-12-05 20:42:58 -08:00
|
|
|
|
tun: Allow to skip filter on attach
There's a small problem with sk-filters on tun devices. Consider
an application doing this sequence of steps:
fd = open("/dev/net/tun");
ioctl(fd, TUNSETIFF, { .ifr_name = "tun0" });
ioctl(fd, TUNATTACHFILTER, &my_filter);
ioctl(fd, TUNSETPERSIST, 1);
close(fd);
At that point the tun0 will remain in the system and will keep in
mind that there should be a socket filter at address '&my_filter'.
If after that we do
fd = open("/dev/net/tun");
ioctl(fd, TUNSETIFF, { .ifr_name = "tun0" });
we most likely receive the -EFAULT error, since tun_attach() would
try to connect the filter back. But (!) if we provide a filter at
address &my_filter, then tun0 will be created and the "new" filter
would be attached, but application may not know about that.
This may create certain problems to anyone using tun-s, but it's
critical problem for c/r -- if we meet a persistent tun device
with a filter in mind, we will not be able to attach to it to dump
its state (flags, owner, address, vnethdr size, etc.).
The proposal is to allow to attach to tun device (with TUNSETIFF)
w/o attaching the filter to the tun-file's socket. After this
attach app may e.g clean the device by dropping the filter, it
doesn't want to have one, or (in case of c/r) get information
about the device with tun ioctls.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-21 14:32:21 +04:00
|
|
|
|
2016-04-05 17:10:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2018-05-11 10:49:25 +08:00
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2016-05-19 13:36:51 +08:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2018-01-03 11:25:59 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-11-30 19:52:01 +01:00
|
|
|
|
2018-01-03 11:25:59 +01:00
|
|
|
|
|
|
|
|
|
xdp: rhashtable with allocator ID to pointer mapping
Use the IDA infrastructure for getting a cyclic increasing ID number,
that is used for keeping track of each registered allocator per
RX-queue xdp_rxq_info. Instead of using the IDR infrastructure, which
uses a radix tree, use a dynamic rhashtable, for creating ID to
pointer lookup table, because this is faster.
The problem that is being solved here is that, the xdp_rxq_info
pointer (stored in xdp_buff) cannot be used directly, as the
guaranteed lifetime is too short. The info is needed on a
(potentially) remote CPU during DMA-TX completion time . In an
xdp_frame the xdp_mem_info is stored, when it got converted from an
xdp_buff, which is sufficient for the simple page refcnt based recycle
schemes.
For more advanced allocators there is a need to store a pointer to the
registered allocator. Thus, there is a need to guard the lifetime or
validity of the allocator pointer, which is done through this
rhashtable ID map to pointer. The removal and validity of of the
allocator and helper struct xdp_mem_allocator is guarded by RCU. The
allocator will be created by the driver, and registered with
xdp_rxq_info_reg_mem_model().
It is up-to debate who is responsible for freeing the allocator
pointer or invoking the allocator destructor function. In any case,
this must happen via RCU freeing.
Use the IDA infrastructure for getting a cyclic increasing ID number,
that is used for keeping track of each registered allocator per
RX-queue xdp_rxq_info.
V4: Per req of Jason Wang
- Use xdp_rxq_info_reg_mem_model() in all drivers implementing
XDP_REDIRECT, even-though it's not strictly necessary when
allocator==NULL for type MEM_TYPE_PAGE_SHARED (given it's zero).
V6: Per req of Alex Duyck
- Introduce rhashtable_lookup() call in later patch
V8: Address sparse should be static warnings (from kbuild test robot)
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 16:46:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-03 11:25:59 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2022-06-22 21:21:05 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2018-09-12 11:16:59 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 10:57:48 +00:00
|
|
|
|
2019-01-07 13:38:38 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 18:56:57 +08:00
|
|
|
|
|
|
|
|
|
2019-01-07 13:38:38 -08:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2019-01-29 22:50:13 -05:00
|
|
|
|
2009-01-20 11:02:28 +00:00
|
|
|
|
|
|
|
|
|
2009-01-20 10:57:48 +00:00
|
|
|
|
|
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
|
|
|
|
|
2011-03-02 07:18:10 +00:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-08-20 08:54:15 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-02-08 17:49:17 -08:00
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2009-02-08 17:49:17 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-08-20 09:00:34 +02:00
|
|
|
|
2009-02-08 17:49:17 -08:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2009-02-08 17:49:17 -08:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-08-20 09:00:34 +02:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
drivers/net: Convert compare_ether_addr to ether_addr_equal
Use the new bool function ether_addr_equal to add
some clarity and reduce the likelihood for misuse
of compare_ether_addr for sorting.
Done via cocci script:
$ cat compare_ether_addr.cocci
@@
expression a,b;
@@
- !compare_ether_addr(a, b)
+ ether_addr_equal(a, b)
@@
expression a,b;
@@
- compare_ether_addr(a, b)
+ !ether_addr_equal(a, b)
@@
expression a,b;
@@
- !ether_addr_equal(a, b) == 0
+ ether_addr_equal(a, b)
@@
expression a,b;
@@
- !ether_addr_equal(a, b) != 0
+ !ether_addr_equal(a, b)
@@
expression a,b;
@@
- ether_addr_equal(a, b) == 0
+ !ether_addr_equal(a, b)
@@
expression a,b;
@@
- ether_addr_equal(a, b) != 0
+ ether_addr_equal(a, b)
@@
expression a,b;
@@
- !!ether_addr_equal(a, b)
+ ether_addr_equal(a, b)
Signed-off-by: Joe Perches <joe@perches.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-05-09 17:17:46 +00:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2006-09-13 14:30:00 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-04 10:38:07 -08:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-04 10:38:07 -08:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
2024-08-29 14:33:37 +02:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-08-29 14:33:37 +02:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2017-03-13 00:00:26 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2016-04-25 23:13:42 -04:00
|
|
|
|
2019-03-22 08:56:38 -07:00
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-10-09 10:32:04 +08:00
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
|
|
|
|
|
2017-06-06 14:09:49 +08:00
|
|
|
|
2018-10-09 10:32:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
2016-04-25 23:13:42 -04:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-16 18:26:53 +02:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2021-11-12 08:56:03 +01:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set
This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.
IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.
The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.
A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.
The documentation in tuntap.rst is updated accordingly.
Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:41 +02:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-12-22 18:54:32 +08:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2026-04-09 14:56:22 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
|
|
|
|
|
2022-03-04 06:55:06 -08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2022-03-04 06:55:06 -08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
|
|
|
|
|
2013-09-05 17:54:00 +08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2013-09-05 17:54:00 +08:00
|
|
|
|
2016-08-23 18:22:33 -04:00
|
|
|
|
2013-07-19 19:40:10 +02:00
|
|
|
|
tun: orphan an skb on tx
The following situation was observed in the field:
tap1 sends packets, tap2 does not consume them, as a result
tap1 can not be closed. This happens because
tun/tap devices can hang on to skbs undefinitely.
As noted by Herbert, possible solutions include a timeout followed by a
copy/change of ownership of the skb, or always copying/changing
ownership if we're going into a hostile device.
This patch implements the second approach.
Note: one issue still remaining is that since skbs
keep reference to tun socket and tun socket has a
reference to tun device, we won't flush backlog,
instead simply waiting for all skbs to get transmitted.
At least this is not user-triggerable, and
this was not reported in practice, my assumption is
other devices besides tap complete an skb
within finite time after it has been queued.
A possible solution for the second issue
would not to have socket reference the device,
instead, implement dev->destructor for tun, and
wait for all skbs to complete there, but this
needs some thought, probably too risky for 2.6.34.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Tested-by: Yan Vugenfirer <yvugenfi@redhat.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-13 04:59:44 +00:00
|
|
|
|
2013-09-05 17:54:00 +08:00
|
|
|
|
|
|
|
|
|
tun: orphan an skb on tx
The following situation was observed in the field:
tap1 sends packets, tap2 does not consume them, as a result
tap1 can not be closed. This happens because
tun/tap devices can hang on to skbs undefinitely.
As noted by Herbert, possible solutions include a timeout followed by a
copy/change of ownership of the skb, or always copying/changing
ownership if we're going into a hostile device.
This patch implements the second approach.
Note: one issue still remaining is that since skbs
keep reference to tun socket and tun socket has a
reference to tun device, we won't flush backlog,
instead simply waiting for all skbs to get transmitted.
At least this is not user-triggerable, and
this was not reported in practice, my assumption is
other devices besides tap complete an skb
within finite time after it has been queued.
A possible solution for the second issue
would not to have socket reference the device,
instead, implement dev->destructor for tun, and
wait for all skbs to complete there, but this
needs some thought, probably too risky for 2.6.34.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Tested-by: Yan Vugenfirer <yvugenfi@redhat.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-13 04:59:44 +00:00
|
|
|
|
|
|
|
|
|
2019-09-29 20:54:03 +02:00
|
|
|
|
2013-03-06 11:02:37 +00:00
|
|
|
|
tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set
This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.
IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.
The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.
A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.
The documentation in tuntap.rst is updated accordingly.
Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:41 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2024-08-29 14:33:37 +02:00
|
|
|
|
2022-04-12 15:58:52 +02:00
|
|
|
|
2021-11-12 08:56:03 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
2014-05-16 15:11:48 -07:00
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
|
|
|
|
|
2009-06-23 06:03:08 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2022-03-10 21:14:20 -08:00
|
|
|
|
2012-11-01 09:16:32 +00:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
2014-11-18 13:20:41 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2011-11-15 15:29:55 +00:00
|
|
|
|
|
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-12 01:21:53 +00:00
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
2026-08-12 01:21:53 +00:00
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
2026-08-12 01:21:53 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
|
|
|
|
|
2017-01-06 19:12:52 -08:00
|
|
|
|
2016-04-13 10:52:20 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
2016-04-13 10:52:20 +02:00
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
|
|
|
|
|
2016-04-13 10:52:20 +02:00
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:16:59 +08:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2018-09-12 11:16:59 +08:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:16:59 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-11-03 13:56:16 -07:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-28 19:12:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
|
|
|
|
|
2008-11-20 20:14:53 -08:00
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
2016-04-13 10:52:20 +02:00
|
|
|
|
2018-11-28 19:12:56 +01:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
|
|
|
|
|
2018-05-31 11:00:03 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-05-31 10:59:47 +02:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-03-08 12:06:58 +01:00
|
|
|
|
xdp: change ndo_xdp_xmit API to support bulking
This patch change the API for ndo_xdp_xmit to support bulking
xdp_frames.
When kernel is compiled with CONFIG_RETPOLINE, XDP sees a huge slowdown.
Most of the slowdown is caused by DMA API indirect function calls, but
also the net_device->ndo_xdp_xmit() call.
Benchmarked patch with CONFIG_RETPOLINE, using xdp_redirect_map with
single flow/core test (CPU E5-1650 v4 @ 3.60GHz), showed
performance improved:
for driver ixgbe: 6,042,682 pps -> 6,853,768 pps = +811,086 pps
for driver i40e : 6,187,169 pps -> 6,724,519 pps = +537,350 pps
With frames avail as a bulk inside the driver ndo_xdp_xmit call,
further optimizations are possible, like bulk DMA-mapping for TX.
Testing without CONFIG_RETPOLINE show the same performance for
physical NIC drivers.
The virtual NIC driver tun sees a huge performance boost, as it can
avoid doing per frame producer locking, but instead amortize the
locking cost over the bulk.
V2: Fix compile errors reported by kbuild test robot <lkp@intel.com>
V4: Isolated ndo, driver changes and callers.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-05-24 16:46:12 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2018-05-31 11:00:03 +02:00
|
|
|
|
2018-05-31 10:59:47 +02:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
2019-05-08 23:20:18 -04:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
xdp: change ndo_xdp_xmit API to support bulking
This patch change the API for ndo_xdp_xmit to support bulking
xdp_frames.
When kernel is compiled with CONFIG_RETPOLINE, XDP sees a huge slowdown.
Most of the slowdown is caused by DMA API indirect function calls, but
also the net_device->ndo_xdp_xmit() call.
Benchmarked patch with CONFIG_RETPOLINE, using xdp_redirect_map with
single flow/core test (CPU E5-1650 v4 @ 3.60GHz), showed
performance improved:
for driver ixgbe: 6,042,682 pps -> 6,853,768 pps = +811,086 pps
for driver i40e : 6,187,169 pps -> 6,724,519 pps = +537,350 pps
With frames avail as a bulk inside the driver ndo_xdp_xmit call,
further optimizations are possible, like bulk DMA-mapping for TX.
Testing without CONFIG_RETPOLINE show the same performance for
physical NIC drivers.
The virtual NIC driver tun sees a huge performance boost, as it can
avoid doing per frame producer locking, but instead amortize the
locking cost over the bulk.
V2: Fix compile errors reported by kbuild test robot <lkp@intel.com>
V4: Isolated ndo, driver changes and callers.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-05-24 16:46:12 +02:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-05-08 23:20:18 -04:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
xdp: change ndo_xdp_xmit API to support bulking
This patch change the API for ndo_xdp_xmit to support bulking
xdp_frames.
When kernel is compiled with CONFIG_RETPOLINE, XDP sees a huge slowdown.
Most of the slowdown is caused by DMA API indirect function calls, but
also the net_device->ndo_xdp_xmit() call.
Benchmarked patch with CONFIG_RETPOLINE, using xdp_redirect_map with
single flow/core test (CPU E5-1650 v4 @ 3.60GHz), showed
performance improved:
for driver ixgbe: 6,042,682 pps -> 6,853,768 pps = +811,086 pps
for driver i40e : 6,187,169 pps -> 6,724,519 pps = +537,350 pps
With frames avail as a bulk inside the driver ndo_xdp_xmit call,
further optimizations are possible, like bulk DMA-mapping for TX.
Testing without CONFIG_RETPOLINE show the same performance for
physical NIC drivers.
The virtual NIC driver tun sees a huge performance boost, as it can
avoid doing per frame producer locking, but instead amortize the
locking cost over the bulk.
V2: Fix compile errors reported by kbuild test robot <lkp@intel.com>
V4: Isolated ndo, driver changes and callers.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-05-24 16:46:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-03-10 21:14:20 -08:00
|
|
|
|
2021-03-08 12:06:58 +01:00
|
|
|
|
xdp: change ndo_xdp_xmit API to support bulking
This patch change the API for ndo_xdp_xmit to support bulking
xdp_frames.
When kernel is compiled with CONFIG_RETPOLINE, XDP sees a huge slowdown.
Most of the slowdown is caused by DMA API indirect function calls, but
also the net_device->ndo_xdp_xmit() call.
Benchmarked patch with CONFIG_RETPOLINE, using xdp_redirect_map with
single flow/core test (CPU E5-1650 v4 @ 3.60GHz), showed
performance improved:
for driver ixgbe: 6,042,682 pps -> 6,853,768 pps = +811,086 pps
for driver i40e : 6,187,169 pps -> 6,724,519 pps = +537,350 pps
With frames avail as a bulk inside the driver ndo_xdp_xmit call,
further optimizations are possible, like bulk DMA-mapping for TX.
Testing without CONFIG_RETPOLINE show the same performance for
physical NIC drivers.
The virtual NIC driver tun sees a huge performance boost, as it can
avoid doing per frame producer locking, but instead amortize the
locking cost over the bulk.
V2: Fix compile errors reported by kbuild test robot <lkp@intel.com>
V4: Isolated ndo, driver changes and callers.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-05-24 16:46:12 +02:00
|
|
|
|
2021-03-08 12:06:58 +01:00
|
|
|
|
xdp: change ndo_xdp_xmit API to support bulking
This patch change the API for ndo_xdp_xmit to support bulking
xdp_frames.
When kernel is compiled with CONFIG_RETPOLINE, XDP sees a huge slowdown.
Most of the slowdown is caused by DMA API indirect function calls, but
also the net_device->ndo_xdp_xmit() call.
Benchmarked patch with CONFIG_RETPOLINE, using xdp_redirect_map with
single flow/core test (CPU E5-1650 v4 @ 3.60GHz), showed
performance improved:
for driver ixgbe: 6,042,682 pps -> 6,853,768 pps = +811,086 pps
for driver i40e : 6,187,169 pps -> 6,724,519 pps = +537,350 pps
With frames avail as a bulk inside the driver ndo_xdp_xmit call,
further optimizations are possible, like bulk DMA-mapping for TX.
Testing without CONFIG_RETPOLINE show the same performance for
physical NIC drivers.
The virtual NIC driver tun sees a huge performance boost, as it can
avoid doing per frame producer locking, but instead amortize the
locking cost over the bulk.
V2: Fix compile errors reported by kbuild test robot <lkp@intel.com>
V4: Isolated ndo, driver changes and callers.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-05-24 16:46:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-05-31 11:00:03 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2021-03-08 12:06:58 +01:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
2018-04-17 16:46:37 +02:00
|
|
|
|
|
|
|
|
|
2020-05-28 22:47:29 +02:00
|
|
|
|
2021-03-08 12:06:58 +01:00
|
|
|
|
2018-04-17 16:46:37 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-03-08 12:06:58 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
|
|
|
|
|
2008-11-20 20:14:53 -08:00
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2011-08-16 06:29:01 +00:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2015-07-31 15:03:27 +09:00
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
2017-11-03 13:56:16 -07:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2018-11-28 19:12:56 +01:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
|
|
|
|
|
2013-06-11 17:01:08 +04:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
treewide: setup_timer() -> timer_setup()
This converts all remaining cases of the old setup_timer() API into using
timer_setup(), where the callback argument is the structure already
holding the struct timer_list. These should have no behavioral changes,
since they just change which pointer is passed into the callback with
the same available pointers after conversion. It handles the following
examples, in addition to some other variations.
Casting from unsigned long:
void my_callback(unsigned long data)
{
struct something *ptr = (struct something *)data;
...
}
...
setup_timer(&ptr->my_timer, my_callback, ptr);
and forced object casts:
void my_callback(struct something *ptr)
{
...
}
...
setup_timer(&ptr->my_timer, my_callback, (unsigned long)ptr);
become:
void my_callback(struct timer_list *t)
{
struct something *ptr = from_timer(ptr, t, my_timer);
...
}
...
timer_setup(&ptr->my_timer, my_callback, 0);
Direct function assignments:
void my_callback(unsigned long data)
{
struct something *ptr = (struct something *)data;
...
}
...
ptr->my_timer.function = my_callback;
have a temporary cast added, along with converting the args:
void my_callback(struct timer_list *t)
{
struct something *ptr = from_timer(ptr, t, my_timer);
...
}
...
ptr->my_timer.function = (TIMER_FUNC_TYPE)my_callback;
And finally, callbacks without a data assignment:
void my_callback(unsigned long data)
{
...
}
...
setup_timer(&ptr->my_timer, my_callback, 0);
have their argument renamed to verify they're unused during conversion:
void my_callback(struct timer_list *unused)
{
...
}
...
timer_setup(&ptr->my_timer, my_callback, 0);
The conversion is done with the following Coccinelle script:
spatch --very-quiet --all-includes --include-headers \
-I ./arch/x86/include -I ./arch/x86/include/generated \
-I ./include -I ./arch/x86/include/uapi \
-I ./arch/x86/include/generated/uapi -I ./include/uapi \
-I ./include/generated/uapi --include ./include/linux/kconfig.h \
--dir . \
--cocci-file ~/src/data/timer_setup.cocci
@fix_address_of@
expression e;
@@
setup_timer(
-&(e)
+&e
, ...)
// Update any raw setup_timer() usages that have a NULL callback, but
// would otherwise match change_timer_function_usage, since the latter
// will update all function assignments done in the face of a NULL
// function initialization in setup_timer().
@change_timer_function_usage_NULL@
expression _E;
identifier _timer;
type _cast_data;
@@
(
-setup_timer(&_E->_timer, NULL, _E);
+timer_setup(&_E->_timer, NULL, 0);
|
-setup_timer(&_E->_timer, NULL, (_cast_data)_E);
+timer_setup(&_E->_timer, NULL, 0);
|
-setup_timer(&_E._timer, NULL, &_E);
+timer_setup(&_E._timer, NULL, 0);
|
-setup_timer(&_E._timer, NULL, (_cast_data)&_E);
+timer_setup(&_E._timer, NULL, 0);
)
@change_timer_function_usage@
expression _E;
identifier _timer;
struct timer_list _stl;
identifier _callback;
type _cast_func, _cast_data;
@@
(
-setup_timer(&_E->_timer, _callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, &_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, &_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)&_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)&_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, &_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, &_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)&_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)&_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
_E->_timer@_stl.function = _callback;
|
_E->_timer@_stl.function = &_callback;
|
_E->_timer@_stl.function = (_cast_func)_callback;
|
_E->_timer@_stl.function = (_cast_func)&_callback;
|
_E._timer@_stl.function = _callback;
|
_E._timer@_stl.function = &_callback;
|
_E._timer@_stl.function = (_cast_func)_callback;
|
_E._timer@_stl.function = (_cast_func)&_callback;
)
// callback(unsigned long arg)
@change_callback_handle_cast
depends on change_timer_function_usage@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _origtype;
identifier _origarg;
type _handletype;
identifier _handle;
@@
void _callback(
-_origtype _origarg
+struct timer_list *t
)
{
(
... when != _origarg
_handletype *_handle =
-(_handletype *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle =
-(void *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle;
... when != _handle
_handle =
-(_handletype *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle;
... when != _handle
_handle =
-(void *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
)
}
// callback(unsigned long arg) without existing variable
@change_callback_handle_cast_no_arg
depends on change_timer_function_usage &&
!change_callback_handle_cast@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _origtype;
identifier _origarg;
type _handletype;
@@
void _callback(
-_origtype _origarg
+struct timer_list *t
)
{
+ _handletype *_origarg = from_timer(_origarg, t, _timer);
+
... when != _origarg
- (_handletype *)_origarg
+ _origarg
... when != _origarg
}
// Avoid already converted callbacks.
@match_callback_converted
depends on change_timer_function_usage &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg@
identifier change_timer_function_usage._callback;
identifier t;
@@
void _callback(struct timer_list *t)
{ ... }
// callback(struct something *handle)
@change_callback_handle_arg
depends on change_timer_function_usage &&
!match_callback_converted &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _handletype;
identifier _handle;
@@
void _callback(
-_handletype *_handle
+struct timer_list *t
)
{
+ _handletype *_handle = from_timer(_handle, t, _timer);
...
}
// If change_callback_handle_arg ran on an empty function, remove
// the added handler.
@unchange_callback_handle_arg
depends on change_timer_function_usage &&
change_callback_handle_arg@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _handletype;
identifier _handle;
identifier t;
@@
void _callback(struct timer_list *t)
{
- _handletype *_handle = from_timer(_handle, t, _timer);
}
// We only want to refactor the setup_timer() data argument if we've found
// the matching callback. This undoes changes in change_timer_function_usage.
@unchange_timer_function_usage
depends on change_timer_function_usage &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg &&
!change_callback_handle_arg@
expression change_timer_function_usage._E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type change_timer_function_usage._cast_data;
@@
(
-timer_setup(&_E->_timer, _callback, 0);
+setup_timer(&_E->_timer, _callback, (_cast_data)_E);
|
-timer_setup(&_E._timer, _callback, 0);
+setup_timer(&_E._timer, _callback, (_cast_data)&_E);
)
// If we fixed a callback from a .function assignment, fix the
// assignment cast now.
@change_timer_function_assignment
depends on change_timer_function_usage &&
(change_callback_handle_cast ||
change_callback_handle_cast_no_arg ||
change_callback_handle_arg)@
expression change_timer_function_usage._E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type _cast_func;
typedef TIMER_FUNC_TYPE;
@@
(
_E->_timer.function =
-_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-&_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-(_cast_func)_callback;
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-(_cast_func)&_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-&_callback;
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-(_cast_func)_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-(_cast_func)&_callback
+(TIMER_FUNC_TYPE)_callback
;
)
// Sometimes timer functions are called directly. Replace matched args.
@change_timer_function_calls
depends on change_timer_function_usage &&
(change_callback_handle_cast ||
change_callback_handle_cast_no_arg ||
change_callback_handle_arg)@
expression _E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type _cast_data;
@@
_callback(
(
-(_cast_data)_E
+&_E->_timer
|
-(_cast_data)&_E
+&_E._timer
|
-_E
+&_E->_timer
)
)
// If a timer has been configured without a data argument, it can be
// converted without regard to the callback argument, since it is unused.
@match_timer_function_unused_data@
expression _E;
identifier _timer;
identifier _callback;
@@
(
-setup_timer(&_E->_timer, _callback, 0);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, 0L);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, 0UL);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0L);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0UL);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0L);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0UL);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0);
+timer_setup(_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0L);
+timer_setup(_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0UL);
+timer_setup(_timer, _callback, 0);
)
@change_callback_unused_data
depends on match_timer_function_unused_data@
identifier match_timer_function_unused_data._callback;
type _origtype;
identifier _origarg;
@@
void _callback(
-_origtype _origarg
+struct timer_list *unused
)
{
... when != _origarg
}
Signed-off-by: Kees Cook <keescook@chromium.org>
2017-10-16 14:43:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-04-05 10:17:26 +02:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: use core MTU range checking in core net infra
geneve:
- Merge __geneve_change_mtu back into geneve_change_mtu, set max_mtu
- This one isn't quite as straight-forward as others, could use some
closer inspection and testing
macvlan:
- set min/max_mtu
tun:
- set min/max_mtu, remove tun_net_change_mtu
vxlan:
- Merge __vxlan_change_mtu back into vxlan_change_mtu
- Set max_mtu to IP_MAX_MTU and retain dynamic MTU range checks in
change_mtu function
- This one is also not as straight-forward and could use closer inspection
and testing from vxlan folks
bridge:
- set max_mtu of IP_MAX_MTU and retain dynamic MTU range checks in
change_mtu function
openvswitch:
- set min/max_mtu, remove internal_dev_change_mtu
- note: max_mtu wasn't checked previously, it's been set to 65535, which
is the largest possible size supported
sch_teql:
- set min/max_mtu (note: max_mtu previously unchecked, used max of 65535)
macsec:
- min_mtu = 0, max_mtu = 65535
macvlan:
- min_mtu = 0, max_mtu = 65535
ntb_netdev:
- min_mtu = 0, max_mtu = 65535
veth:
- min_mtu = 68, max_mtu = 65535
8021q:
- min_mtu = 0, max_mtu = 65535
CC: netdev@vger.kernel.org
CC: Nicolas Dichtel <nicolas.dichtel@6wind.com>
CC: Hannes Frederic Sowa <hannes@stressinduktion.org>
CC: Tom Herbert <tom@herbertland.com>
CC: Daniel Borkmann <daniel@iogearbox.net>
CC: Alexander Duyck <alexander.h.duyck@intel.com>
CC: Paolo Abeni <pabeni@redhat.com>
CC: Jiri Benc <jbenc@redhat.com>
CC: WANG Cong <xiyou.wangcong@gmail.com>
CC: Roopa Prabhu <roopa@cumulusnetworks.com>
CC: Pravin B Shelar <pshelar@ovn.org>
CC: Sabrina Dubroca <sd@queasysnail.net>
CC: Patrick McHardy <kaber@trash.net>
CC: Stephen Hemminger <stephen@networkplumber.org>
CC: Pravin Shelar <pshelar@nicira.com>
CC: Maxim Krasnyansky <maxk@qti.qualcomm.com>
Signed-off-by: Jarod Wilson <jarod@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-10-20 13:55:20 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
2020-06-29 19:06:22 -06:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2008-12-29 18:23:28 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2011-07-26 06:05:38 +00:00
|
|
|
|
2012-12-10 15:16:00 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-02-15 06:45:39 +00:00
|
|
|
|
2007-04-26 01:00:55 -07:00
|
|
|
|
drivers: net: turn on XDP features
A summary of the flags being set for various drivers is given below.
Note that XDP_F_REDIRECT_TARGET and XDP_F_FRAG_TARGET are features
that can be turned off and on at runtime. This means that these flags
may be set and unset under RTNL lock protection by the driver. Hence,
READ_ONCE must be used by code loading the flag value.
Also, these flags are not used for synchronization against the availability
of XDP resources on a device. It is merely a hint, and hence the read
may race with the actual teardown of XDP resources on the device. This
may change in the future, e.g. operations taking a reference on the XDP
resources of the driver, and in turn inhibiting turning off this flag.
However, for now, it can only be used as a hint to check whether device
supports becoming a redirection target.
Turn 'hw-offload' feature flag on for:
- netronome (nfp)
- netdevsim.
Turn 'native' and 'zerocopy' features flags on for:
- intel (i40e, ice, ixgbe, igc)
- mellanox (mlx5).
- stmmac
- netronome (nfp)
Turn 'native' features flags on for:
- amazon (ena)
- broadcom (bnxt)
- freescale (dpaa, dpaa2, enetc)
- funeth
- intel (igb)
- marvell (mvneta, mvpp2, octeontx2)
- mellanox (mlx4)
- mtk_eth_soc
- qlogic (qede)
- sfc
- socionext (netsec)
- ti (cpsw)
- tap
- tsnep
- veth
- xen
- virtio_net.
Turn 'basic' (tx, pass, aborted and drop) features flags on for:
- netronome (nfp)
- cavium (thunder)
- hyperv.
Turn 'redirect_target' feature flag on for:
- amanzon (ena)
- broadcom (bnxt)
- freescale (dpaa, dpaa2)
- intel (i40e, ice, igb, ixgbe)
- ti (cpsw)
- marvell (mvneta, mvpp2)
- sfc
- socionext (netsec)
- qlogic (qede)
- mellanox (mlx5)
- tap
- veth
- virtio_net
- xen
Reviewed-by: Gerhard Engleder <gerhard@engleder-embedded.com>
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Acked-by: Stanislav Fomichev <sdf@google.com>
Acked-by: Jakub Kicinski <kuba@kernel.org>
Co-developed-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Co-developed-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Marek Majtyka <alardam@gmail.com>
Link: https://lore.kernel.org/r/3eca9fafb308462f7edb1f58e451d59209aa07eb.1675245258.git.lorenzo@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2023-02-01 11:24:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
net: use core MTU range checking in core net infra
geneve:
- Merge __geneve_change_mtu back into geneve_change_mtu, set max_mtu
- This one isn't quite as straight-forward as others, could use some
closer inspection and testing
macvlan:
- set min/max_mtu
tun:
- set min/max_mtu, remove tun_net_change_mtu
vxlan:
- Merge __vxlan_change_mtu back into vxlan_change_mtu
- Set max_mtu to IP_MAX_MTU and retain dynamic MTU range checks in
change_mtu function
- This one is also not as straight-forward and could use closer inspection
and testing from vxlan folks
bridge:
- set max_mtu of IP_MAX_MTU and retain dynamic MTU range checks in
change_mtu function
openvswitch:
- set min/max_mtu, remove internal_dev_change_mtu
- note: max_mtu wasn't checked previously, it's been set to 65535, which
is the largest possible size supported
sch_teql:
- set min/max_mtu (note: max_mtu previously unchecked, used max of 65535)
macsec:
- min_mtu = 0, max_mtu = 65535
macvlan:
- min_mtu = 0, max_mtu = 65535
ntb_netdev:
- min_mtu = 0, max_mtu = 65535
veth:
- min_mtu = 68, max_mtu = 65535
8021q:
- min_mtu = 0, max_mtu = 65535
CC: netdev@vger.kernel.org
CC: Nicolas Dichtel <nicolas.dichtel@6wind.com>
CC: Hannes Frederic Sowa <hannes@stressinduktion.org>
CC: Tom Herbert <tom@herbertland.com>
CC: Daniel Borkmann <daniel@iogearbox.net>
CC: Alexander Duyck <alexander.h.duyck@intel.com>
CC: Paolo Abeni <pabeni@redhat.com>
CC: Jiri Benc <jbenc@redhat.com>
CC: WANG Cong <xiyou.wangcong@gmail.com>
CC: Roopa Prabhu <roopa@cumulusnetworks.com>
CC: Pravin B Shelar <pshelar@ovn.org>
CC: Sabrina Dubroca <sd@queasysnail.net>
CC: Patrick McHardy <kaber@trash.net>
CC: Stephen Hemminger <stephen@networkplumber.org>
CC: Pravin Shelar <pshelar@nicira.com>
CC: Maxim Krasnyansky <maxk@qti.qualcomm.com>
Signed-off-by: Jarod Wilson <jarod@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-10-20 13:55:20 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2018-05-22 14:21:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-07-03 06:39:46 -04:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2009-01-20 11:03:21 +00:00
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2009-07-05 19:48:35 +00:00
|
|
|
|
2017-07-03 06:39:46 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2009-07-05 19:48:35 +00:00
|
|
|
|
2014-05-16 15:11:48 -07:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-05-22 14:21:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
2009-01-20 11:07:17 +00:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: tun: fix bugs for oversize packet when napi frags enabled
Recently, we got two syzkaller problems because of oversize packet
when napi frags enabled.
One of the problems is because the first seg size of the iov_iter
from user space is very big, it is 2147479538 which is bigger than
the threshold value for bail out early in __alloc_pages(). And
skb->pfmemalloc is true, __kmalloc_reserve() would use pfmemalloc
reserves without __GFP_NOWARN flag. Thus we got a warning as following:
========================================================
WARNING: CPU: 1 PID: 17965 at mm/page_alloc.c:5295 __alloc_pages+0x1308/0x16c4 mm/page_alloc.c:5295
...
Call trace:
__alloc_pages+0x1308/0x16c4 mm/page_alloc.c:5295
__alloc_pages_node include/linux/gfp.h:550 [inline]
alloc_pages_node include/linux/gfp.h:564 [inline]
kmalloc_large_node+0x94/0x350 mm/slub.c:4038
__kmalloc_node_track_caller+0x620/0x8e4 mm/slub.c:4545
__kmalloc_reserve.constprop.0+0x1e4/0x2b0 net/core/skbuff.c:151
pskb_expand_head+0x130/0x8b0 net/core/skbuff.c:1654
__skb_grow include/linux/skbuff.h:2779 [inline]
tun_napi_alloc_frags+0x144/0x610 drivers/net/tun.c:1477
tun_get_user+0x31c/0x2010 drivers/net/tun.c:1835
tun_chr_write_iter+0x98/0x100 drivers/net/tun.c:2036
The other problem is because odd IPv6 packets without NEXTHDR_NONE
extension header and have big packet length, it is 2127925 which is
bigger than ETH_MAX_MTU(65535). After ipv6_gso_pull_exthdrs() in
ipv6_gro_receive(), network_header offset and transport_header offset
are all bigger than U16_MAX. That would trigger skb->network_header
and skb->transport_header overflow error, because they are all '__u16'
type. Eventually, it would affect the value for __skb_push(skb, value),
and make it be a big value. After __skb_push() in ipv6_gro_receive(),
skb->data would less than skb->head, an out of bounds memory bug occurred.
That would trigger the problem as following:
==================================================================
BUG: KASAN: use-after-free in eth_type_trans+0x100/0x260
...
Call trace:
dump_backtrace+0xd8/0x130
show_stack+0x1c/0x50
dump_stack_lvl+0x64/0x7c
print_address_description.constprop.0+0xbc/0x2e8
print_report+0x100/0x1e4
kasan_report+0x80/0x120
__asan_load8+0x78/0xa0
eth_type_trans+0x100/0x260
napi_gro_frags+0x164/0x550
tun_get_user+0xda4/0x1270
tun_chr_write_iter+0x74/0x130
do_iter_readv_writev+0x130/0x1ec
do_iter_write+0xbc/0x1e0
vfs_writev+0x13c/0x26c
To fix the problems, restrict the packet size less than
(ETH_MAX_MTU - NET_SKB_PAD - NET_IP_ALIGN) which has considered reserved
skb space in napi_alloc_skb() because transport_header is an offset from
skb->head. Add len check in tun_napi_alloc_frags() simply.
Fixes: 90e33d459407 ("tun: enable napi_gro_frags() for TUN/TAP driver")
Signed-off-by: Ziyang Xuan <william.xuanziyang@huawei.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Link: https://lore.kernel.org/r/20221029094101.1653855-1-william.xuanziyang@huawei.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2022-10-29 17:41:01 +08:00
|
|
|
|
|
|
|
|
|
2020-12-25 10:52:16 +08:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-12-12 22:22:47 +00:00
|
|
|
|
2023-03-29 08:52:15 -06:00
|
|
|
|
2018-11-18 07:37:33 -08:00
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-18 07:37:33 -08:00
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-18 07:37:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2011-06-08 14:33:08 +00:00
|
|
|
|
|
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
|
|
|
|
|
2023-08-09 09:47:52 -07:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
2023-08-01 20:52:52 +00:00
|
|
|
|
|
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2023-08-01 20:52:52 +00:00
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
2008-08-15 15:15:10 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-18 00:46:00 -07:00
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-18 00:46:00 -07:00
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
2018-11-18 00:46:00 -07:00
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-03 20:59:48 +02:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-07-23 16:23:01 +02:00
|
|
|
|
|
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2018-09-12 11:17:03 +08:00
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
|
|
|
|
|
2019-07-23 16:23:01 +02:00
|
|
|
|
2018-09-12 11:17:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-01-19 18:22:35 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2024-01-19 18:22:35 +08:00
|
|
|
|
2024-01-19 18:22:56 +08:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-01-19 18:22:35 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2024-01-19 18:22:35 +08:00
|
|
|
|
2024-01-19 18:22:56 +08:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-11-30 11:08:07 +01:00
|
|
|
|
2020-08-23 17:36:59 -05:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
|
|
|
|
|
2020-08-23 17:36:59 -05:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2022-03-10 21:14:20 -08:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2017-09-04 11:36:09 +08:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
2017-08-16 22:14:33 +08:00
|
|
|
|
2024-07-04 16:48:15 +02:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2017-09-04 11:36:08 +08:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2017-09-04 11:36:08 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:00 +08:00
|
|
|
|
2017-09-04 11:36:08 +08:00
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
2017-10-27 11:05:44 +08:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-04 11:36:08 +08:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-04 11:36:08 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:03 +08:00
|
|
|
|
2017-09-04 11:36:09 +08:00
|
|
|
|
2019-07-23 16:23:01 +02:00
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
2018-09-12 11:17:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
2018-05-28 19:37:49 +09:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2024-07-04 16:48:15 +02:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-12-22 22:09:28 +01:00
|
|
|
|
net: tun: Enable XDP metadata support
Enable the support for the bpf_xdp_adjust_meta helper function for XDP
buffers initialized by the tun driver. This allows to reserve a metadata
area that is useful to pass any information from one XDP program to
another one, for example when using tail-calls.
Whether this helper function can be used in an XDP program depends on
how the xdp_buff was initialized. Most net drivers initialize the
xdp_buff in a way, that allows bpf_xdp_adjust_meta to be used. In case
of the tun driver, this is currently not the case.
There are two code paths in the tun driver that lead to a
bpf_prog_run_xdp and where metadata support should be enabled:
1. tun_build_skb, which is called by tun_get_user and is used when
writing packets from userspace into the device. In this case, the
xdp_buff created in tun_build_skb has no support for
bpf_xdp_adjust_meta and calls of that helper function result in
ENOTSUPP.
For this code path, it's sufficient to set the meta_valid argument of
the xdp_prepare_buff call. The reserved headroom is large enough
already.
2. tun_xdp_one, which is called by tun_sendmsg which again is called by
other drivers (e.g. vhost_net). When the TUN_MSG_PTR mode is used,
another driver may pass a batch of xdp_buffs to the tun driver. In
this case, that other driver is the one initializing the xdp_buff.
See commit 043d222f93ab ("tuntap: accept an array of XDP buffs
through sendmsg()") for details.
For now, the vhost_net driver is the only one using TUN_MSG_PTR and
it already initializes the xdp_buffs with metadata support and
sufficient headroom. But the tun driver disables it again, so the
xdp_set_data_meta_invalid call has to be removed.
Signed-off-by: Marcus Wichelmann <marcus.wichelmann@hetzner-cloud.de>
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Acked-by: Jason Wang <jasowang@redhat.com>
Link: https://patch.msgid.link/20250305213438.3863922-2-marcus.wichelmann@hetzner-cloud.de
2025-03-05 21:34:33 +00:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
2020-04-03 16:13:21 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:05 +08:00
|
|
|
|
xdp: Use bulking for non-map XDP_REDIRECT and consolidate code paths
Since the bulk queue used by XDP_REDIRECT now lives in struct net_device,
we can re-use the bulking for the non-map version of the bpf_redirect()
helper. This is a simple matter of having xdp_do_redirect_slow() queue the
frame on the bulk queue instead of sending it out with __bpf_tx_xdp().
Unfortunately we can't make the bpf_redirect() helper return an error if
the ifindex doesn't exit (as bpf_redirect_map() does), because we don't
have a reference to the network namespace of the ingress device at the time
the helper is called. So we have to leave it as-is and keep the device
lookup in xdp_do_redirect_slow().
Since this leaves less reason to have the non-map redirect code in a
separate function, so we get rid of the xdp_do_redirect_slow() function
entirely. This does lose us the tracepoint disambiguation, but fortunately
the xdp_redirect and xdp_redirect_map tracepoints use the same tracepoint
entry structures. This means both can contain a map index, so we can just
amend the tracepoint definitions so we always emit the xdp_redirect(_err)
tracepoints, but with the map ID only populated if a map is present. This
means we retire the xdp_redirect_map(_err) tracepoints entirely, but keep
the definitions around in case someone is still listening for them.
With this change, the performance of the xdp_redirect sample program goes
from 5Mpps to 8.4Mpps (a 68% increase).
Since the flush functions are no longer map-specific, rename the flush()
functions to drop _map from their names. One of the renamed functions is
the xdp_do_flush_map() callback used in all the xdp-enabled drivers. To
keep from having to update all drivers, use a #define to keep the old name
working, and only update the virtual drivers in this patch.
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: John Fastabend <john.fastabend@gmail.com>
Link: https://lore.kernel.org/bpf/157918768505.1458396.17518057312953572912.stgit@toke.dk
2020-01-16 16:14:45 +01:00
|
|
|
|
2018-09-12 11:17:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2024-07-04 16:48:15 +02:00
|
|
|
|
2018-09-12 11:17:01 +08:00
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2018-09-12 11:17:02 +08:00
|
|
|
|
2024-07-04 16:48:15 +02:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2018-05-28 19:37:49 +09:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2014-06-19 15:36:49 -04:00
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-02-01 00:45:17 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-06-19 15:36:49 -04:00
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
2013-11-13 14:00:39 +08:00
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
2025-02-07 15:10:52 +09:00
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2017-09-04 11:36:09 +08:00
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2022-11-10 15:31:25 +08:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2013-08-15 15:52:57 +03:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-08-15 15:52:57 +03:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2016-11-01 22:09:04 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2017-02-03 18:20:48 -05:00
|
|
|
|
|
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-07 15:10:54 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-02-03 18:20:48 -05:00
|
|
|
|
2008-07-03 03:48:02 -07:00
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2011-06-08 14:33:07 +00:00
|
|
|
|
2025-02-07 15:10:52 +09:00
|
|
|
|
2008-04-12 18:49:30 -07:00
|
|
|
|
|
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2013-11-13 14:00:39 +08:00
|
|
|
|
|
|
|
|
|
2013-07-18 10:55:15 +08:00
|
|
|
|
2014-06-19 15:36:49 -04:00
|
|
|
|
|
|
|
|
|
2013-07-18 10:55:15 +08:00
|
|
|
|
|
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
|
|
|
|
|
2025-02-07 15:10:52 +09:00
|
|
|
|
2013-07-10 13:43:27 +08:00
|
|
|
|
2014-06-19 15:36:49 -04:00
|
|
|
|
|
|
|
|
|
2013-07-18 10:55:15 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2017-09-04 11:36:09 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
2022-11-10 15:31:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-07 15:10:52 +09:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-09 09:47:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-10 15:31:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-14 20:19:47 -07:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2022-11-10 15:31:25 +08:00
|
|
|
|
2017-08-11 19:41:16 +08:00
|
|
|
|
2006-03-11 18:49:13 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
net: always dissect GSO packets in __virtio_net_hdr_to_skb()
Commit 9e8db5913264 ("net: avoid false positives in untrusted gso
validation") added a '&& skb->network_header' check before flow-dissecting
GSO packets without VIRTIO_NET_HDR_F_NEEDS_CSUM in
__virtio_net_hdr_to_skb(), because some callers (such as tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header()) called
virtio_net_hdr_*_to_skb() before initializing skb->network_header and
skb->dev.
Because __alloc_skb() and __build_skb_around() zero-initialize
skb->network_header to 0 (unlike mac_header and transport_header which
are initialized to ~0U), those four callers always had
skb->network_header == 0 and bypassed flow dissection in
__virtio_net_hdr_to_skb(). More generally, skb->network_header is an
offset from skb->head (where 0 is also a valid offset whenever
skb_headroom(skb) is 0), not a boolean flag.
Whenever the 'if (gso_type && skb->network_header)' branch was skipped,
the fallback 'else if (gso_type)' only pulled nh_min_len + thlen (40 bytes
for TCPv4) without dissecting the packet, without validating ip_proto or
n_proto, and without setting skb->transport_header.
If the packet has a malformed network header, it is not rejected and a
subsequent skb_probe_transport_header() also fails, leaving
skb->transport_header at ~0U (0xffff). Similarly, if an IPv4 packet
carries IP options (ihl > 5) or an IPv6 packet carries extension headers,
pulling only nh_min_len + thlen can leave the TCP header outside
skb->head. In both cases, tcp_hdrlen(skb) in skb_gso_transport_seglen()
reads out-of-bounds:
BUG: KASAN: slab-out-of-bounds in skb_gso_transport_seglen
Read of size 2 by task poc/133
skb_gso_transport_seglen (net/core/gso.c:155)
skb_gso_validate_mac_len (net/core/gso.c:270)
tbf_enqueue (net/sched/sch_tbf.c:260)
dev_qdisc_enqueue (net/core/dev.c:4227)
__dev_queue_xmit (net/core/dev.c:4884)
In addition, checking virtio_net_hdr_match_proto() only inside
'if (!skb->protocol)' before flow dissection both skipped validation when
skb->protocol was pre-set by the caller and rejected VLAN-tagged frames
whose outer L2 protocol is ETH_P_8021Q or ETH_P_8021AD.
Fix this by:
1. Initializing skb->dev and skb->network_header (plus skb->protocol for
IFF_TUN) before virtio_net_hdr_*_to_skb() in tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header(). In
tun_get_user(), drop the redundant skb_reset_mac_header(skb) in the
IFF_TUN case since __virtio_net_hdr_to_skb() unconditionally resets
mac_header.
2. Removing '&& skb->network_header' and the unvalidated
'else if (gso_type)' fallback in __virtio_net_hdr_to_skb() so all GSO
packets without VIRTIO_NET_HDR_F_NEEDS_CSUM are flow-dissected, have
their transport header pulled into linear data, and have
skb->transport_header set.
3. Moving the virtio_net_hdr_match_proto() check to after
skb_flow_dissect_flow_keys_basic(), validating keys.basic.n_proto
against hdr_gso_type.
Fixes: 9e8db5913264 ("net: avoid false positives in untrusted gso validation")
Fixes: d5be7f632bad ("net: validate untrusted gso packets without csum offload")
Fixes: 924a9bc362a5 ("net: check if protocol extracted by virtio_net_hdr_set_proto is correct")
Reported-by: Weiming Shi <bestswngs@gmail.com>
Closes: https://lore.kernel.org/netdev/20260927163117.746432-2-bestswngs@gmail.com/
Signed-off-by: Eric Dumazet <edumazet@kernel.org>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Link: https://patch.msgid.link/20261001191140.2818991-3-edumazet@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-10-01 19:11:39 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
|
|
|
|
|
2026-08-12 01:21:53 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun: bail out from tun_get_user() if the skb is empty
KMSAN (https://github.com/google/kmsan) reported accessing uninitialized
skb->data[0] in the case the skb is empty (i.e. skb->len is 0):
================================================
BUG: KMSAN: use of uninitialized memory in tun_get_user+0x19ba/0x3770
CPU: 0 PID: 3051 Comm: probe Not tainted 4.13.0+ #3140
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
...
__msan_warning_32+0x66/0xb0 mm/kmsan/kmsan_instr.c:477
tun_get_user+0x19ba/0x3770 drivers/net/tun.c:1301
tun_chr_write_iter+0x19f/0x300 drivers/net/tun.c:1365
call_write_iter ./include/linux/fs.h:1743
new_sync_write fs/read_write.c:457
__vfs_write+0x6c3/0x7f0 fs/read_write.c:470
vfs_write+0x3e4/0x770 fs/read_write.c:518
SYSC_write+0x12f/0x2b0 fs/read_write.c:565
SyS_write+0x55/0x80 fs/read_write.c:557
do_syscall_64+0x242/0x330 arch/x86/entry/common.c:284
entry_SYSCALL64_slow_path+0x25/0x25 arch/x86/entry/entry_64.S:245
...
origin:
...
kmsan_poison_shadow+0x6e/0xc0 mm/kmsan/kmsan.c:211
slab_alloc_node mm/slub.c:2732
__kmalloc_node_track_caller+0x351/0x370 mm/slub.c:4351
__kmalloc_reserve net/core/skbuff.c:138
__alloc_skb+0x26a/0x810 net/core/skbuff.c:231
alloc_skb ./include/linux/skbuff.h:903
alloc_skb_with_frags+0x1d7/0xc80 net/core/skbuff.c:4756
sock_alloc_send_pskb+0xabf/0xfe0 net/core/sock.c:2037
tun_alloc_skb drivers/net/tun.c:1144
tun_get_user+0x9a8/0x3770 drivers/net/tun.c:1274
tun_chr_write_iter+0x19f/0x300 drivers/net/tun.c:1365
call_write_iter ./include/linux/fs.h:1743
new_sync_write fs/read_write.c:457
__vfs_write+0x6c3/0x7f0 fs/read_write.c:470
vfs_write+0x3e4/0x770 fs/read_write.c:518
SYSC_write+0x12f/0x2b0 fs/read_write.c:565
SyS_write+0x55/0x80 fs/read_write.c:557
do_syscall_64+0x242/0x330 arch/x86/entry/common.c:284
return_from_SYSCALL_64+0x0/0x6a arch/x86/entry/entry_64.S:245
================================================
Make sure tun_get_user() doesn't touch skb->data[0] unless there is
actual data.
C reproducer below:
==========================
// autogenerated by syzkaller (http://github.com/google/syzkaller)
#define _GNU_SOURCE
#include <fcntl.h>
#include <linux/if_tun.h>
#include <netinet/ip.h>
#include <net/if.h>
#include <string.h>
#include <sys/ioctl.h>
int main()
{
int sock = socket(PF_INET, SOCK_STREAM, IPPROTO_IP);
int tun_fd = open("/dev/net/tun", O_RDWR);
struct ifreq req;
memset(&req, 0, sizeof(struct ifreq));
strcpy((char*)&req.ifr_name, "gre0");
req.ifr_flags = IFF_UP | IFF_MULTICAST;
ioctl(tun_fd, TUNSETIFF, &req);
ioctl(sock, SIOCSIFFLAGS, "gre0");
write(tun_fd, "hi", 0);
return 0;
}
==========================
Signed-off-by: Alexander Potapenko <glider@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-09-28 11:32:37 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-06-17 21:10:33 -07:00
|
|
|
|
|
|
|
|
|
tun: bail out from tun_get_user() if the skb is empty
KMSAN (https://github.com/google/kmsan) reported accessing uninitialized
skb->data[0] in the case the skb is empty (i.e. skb->len is 0):
================================================
BUG: KMSAN: use of uninitialized memory in tun_get_user+0x19ba/0x3770
CPU: 0 PID: 3051 Comm: probe Not tainted 4.13.0+ #3140
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
...
__msan_warning_32+0x66/0xb0 mm/kmsan/kmsan_instr.c:477
tun_get_user+0x19ba/0x3770 drivers/net/tun.c:1301
tun_chr_write_iter+0x19f/0x300 drivers/net/tun.c:1365
call_write_iter ./include/linux/fs.h:1743
new_sync_write fs/read_write.c:457
__vfs_write+0x6c3/0x7f0 fs/read_write.c:470
vfs_write+0x3e4/0x770 fs/read_write.c:518
SYSC_write+0x12f/0x2b0 fs/read_write.c:565
SyS_write+0x55/0x80 fs/read_write.c:557
do_syscall_64+0x242/0x330 arch/x86/entry/common.c:284
entry_SYSCALL64_slow_path+0x25/0x25 arch/x86/entry/entry_64.S:245
...
origin:
...
kmsan_poison_shadow+0x6e/0xc0 mm/kmsan/kmsan.c:211
slab_alloc_node mm/slub.c:2732
__kmalloc_node_track_caller+0x351/0x370 mm/slub.c:4351
__kmalloc_reserve net/core/skbuff.c:138
__alloc_skb+0x26a/0x810 net/core/skbuff.c:231
alloc_skb ./include/linux/skbuff.h:903
alloc_skb_with_frags+0x1d7/0xc80 net/core/skbuff.c:4756
sock_alloc_send_pskb+0xabf/0xfe0 net/core/sock.c:2037
tun_alloc_skb drivers/net/tun.c:1144
tun_get_user+0x9a8/0x3770 drivers/net/tun.c:1274
tun_chr_write_iter+0x19f/0x300 drivers/net/tun.c:1365
call_write_iter ./include/linux/fs.h:1743
new_sync_write fs/read_write.c:457
__vfs_write+0x6c3/0x7f0 fs/read_write.c:470
vfs_write+0x3e4/0x770 fs/read_write.c:518
SYSC_write+0x12f/0x2b0 fs/read_write.c:565
SyS_write+0x55/0x80 fs/read_write.c:557
do_syscall_64+0x242/0x330 arch/x86/entry/common.c:284
return_from_SYSCALL_64+0x0/0x6a arch/x86/entry/entry_64.S:245
================================================
Make sure tun_get_user() doesn't touch skb->data[0] unless there is
actual data.
C reproducer below:
==========================
// autogenerated by syzkaller (http://github.com/google/syzkaller)
#define _GNU_SOURCE
#include <fcntl.h>
#include <linux/if_tun.h>
#include <netinet/ip.h>
#include <net/if.h>
#include <string.h>
#include <sys/ioctl.h>
int main()
{
int sock = socket(PF_INET, SOCK_STREAM, IPPROTO_IP);
int tun_fd = open("/dev/net/tun", O_RDWR);
struct ifreq req;
memset(&req, 0, sizeof(struct ifreq));
strcpy((char*)&req.ifr_name, "gre0");
req.ifr_flags = IFF_UP | IFF_MULTICAST;
ioctl(tun_fd, TUNSETIFF, &req);
ioctl(sock, SIOCSIFFLAGS, "gre0");
write(tun_fd, "hi", 0);
return 0;
}
==========================
Signed-off-by: Alexander Potapenko <glider@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-09-28 11:32:37 +02:00
|
|
|
|
2008-06-17 21:10:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-10 15:31:25 +08:00
|
|
|
|
|
|
|
|
|
2008-06-17 21:10:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: always dissect GSO packets in __virtio_net_hdr_to_skb()
Commit 9e8db5913264 ("net: avoid false positives in untrusted gso
validation") added a '&& skb->network_header' check before flow-dissecting
GSO packets without VIRTIO_NET_HDR_F_NEEDS_CSUM in
__virtio_net_hdr_to_skb(), because some callers (such as tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header()) called
virtio_net_hdr_*_to_skb() before initializing skb->network_header and
skb->dev.
Because __alloc_skb() and __build_skb_around() zero-initialize
skb->network_header to 0 (unlike mac_header and transport_header which
are initialized to ~0U), those four callers always had
skb->network_header == 0 and bypassed flow dissection in
__virtio_net_hdr_to_skb(). More generally, skb->network_header is an
offset from skb->head (where 0 is also a valid offset whenever
skb_headroom(skb) is 0), not a boolean flag.
Whenever the 'if (gso_type && skb->network_header)' branch was skipped,
the fallback 'else if (gso_type)' only pulled nh_min_len + thlen (40 bytes
for TCPv4) without dissecting the packet, without validating ip_proto or
n_proto, and without setting skb->transport_header.
If the packet has a malformed network header, it is not rejected and a
subsequent skb_probe_transport_header() also fails, leaving
skb->transport_header at ~0U (0xffff). Similarly, if an IPv4 packet
carries IP options (ihl > 5) or an IPv6 packet carries extension headers,
pulling only nh_min_len + thlen can leave the TCP header outside
skb->head. In both cases, tcp_hdrlen(skb) in skb_gso_transport_seglen()
reads out-of-bounds:
BUG: KASAN: slab-out-of-bounds in skb_gso_transport_seglen
Read of size 2 by task poc/133
skb_gso_transport_seglen (net/core/gso.c:155)
skb_gso_validate_mac_len (net/core/gso.c:270)
tbf_enqueue (net/sched/sch_tbf.c:260)
dev_qdisc_enqueue (net/core/dev.c:4227)
__dev_queue_xmit (net/core/dev.c:4884)
In addition, checking virtio_net_hdr_match_proto() only inside
'if (!skb->protocol)' before flow dissection both skipped validation when
skb->protocol was pre-set by the caller and rejected VLAN-tagged frames
whose outer L2 protocol is ETH_P_8021Q or ETH_P_8021AD.
Fix this by:
1. Initializing skb->dev and skb->network_header (plus skb->protocol for
IFF_TUN) before virtio_net_hdr_*_to_skb() in tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header(). In
tun_get_user(), drop the redundant skb_reset_mac_header(skb) in the
IFF_TUN case since __virtio_net_hdr_to_skb() unconditionally resets
mac_header.
2. Removing '&& skb->network_header' and the unvalidated
'else if (gso_type)' fallback in __virtio_net_hdr_to_skb() so all GSO
packets without VIRTIO_NET_HDR_F_NEEDS_CSUM are flow-dissected, have
their transport header pulled into linear data, and have
skb->transport_header set.
3. Moving the virtio_net_hdr_match_proto() check to after
skb_flow_dissect_flow_keys_basic(), validating keys.basic.n_proto
against hdr_gso_type.
Fixes: 9e8db5913264 ("net: avoid false positives in untrusted gso validation")
Fixes: d5be7f632bad ("net: validate untrusted gso packets without csum offload")
Fixes: 924a9bc362a5 ("net: check if protocol extracted by virtio_net_hdr_set_proto is correct")
Reported-by: Weiming Shi <bestswngs@gmail.com>
Closes: https://lore.kernel.org/netdev/20260927163117.746432-2-bestswngs@gmail.com/
Signed-off-by: Eric Dumazet <edumazet@kernel.org>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Link: https://patch.msgid.link/20261001191140.2818991-3-edumazet@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-10-01 19:11:39 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2026-08-12 01:21:53 +00:00
|
|
|
|
2020-05-30 15:41:31 -04:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2020-05-30 15:41:31 -04:00
|
|
|
|
|
|
|
|
|
net: always dissect GSO packets in __virtio_net_hdr_to_skb()
Commit 9e8db5913264 ("net: avoid false positives in untrusted gso
validation") added a '&& skb->network_header' check before flow-dissecting
GSO packets without VIRTIO_NET_HDR_F_NEEDS_CSUM in
__virtio_net_hdr_to_skb(), because some callers (such as tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header()) called
virtio_net_hdr_*_to_skb() before initializing skb->network_header and
skb->dev.
Because __alloc_skb() and __build_skb_around() zero-initialize
skb->network_header to 0 (unlike mac_header and transport_header which
are initialized to ~0U), those four callers always had
skb->network_header == 0 and bypassed flow dissection in
__virtio_net_hdr_to_skb(). More generally, skb->network_header is an
offset from skb->head (where 0 is also a valid offset whenever
skb_headroom(skb) is 0), not a boolean flag.
Whenever the 'if (gso_type && skb->network_header)' branch was skipped,
the fallback 'else if (gso_type)' only pulled nh_min_len + thlen (40 bytes
for TCPv4) without dissecting the packet, without validating ip_proto or
n_proto, and without setting skb->transport_header.
If the packet has a malformed network header, it is not rejected and a
subsequent skb_probe_transport_header() also fails, leaving
skb->transport_header at ~0U (0xffff). Similarly, if an IPv4 packet
carries IP options (ihl > 5) or an IPv6 packet carries extension headers,
pulling only nh_min_len + thlen can leave the TCP header outside
skb->head. In both cases, tcp_hdrlen(skb) in skb_gso_transport_seglen()
reads out-of-bounds:
BUG: KASAN: slab-out-of-bounds in skb_gso_transport_seglen
Read of size 2 by task poc/133
skb_gso_transport_seglen (net/core/gso.c:155)
skb_gso_validate_mac_len (net/core/gso.c:270)
tbf_enqueue (net/sched/sch_tbf.c:260)
dev_qdisc_enqueue (net/core/dev.c:4227)
__dev_queue_xmit (net/core/dev.c:4884)
In addition, checking virtio_net_hdr_match_proto() only inside
'if (!skb->protocol)' before flow dissection both skipped validation when
skb->protocol was pre-set by the caller and rejected VLAN-tagged frames
whose outer L2 protocol is ETH_P_8021Q or ETH_P_8021AD.
Fix this by:
1. Initializing skb->dev and skb->network_header (plus skb->protocol for
IFF_TUN) before virtio_net_hdr_*_to_skb() in tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header(). In
tun_get_user(), drop the redundant skb_reset_mac_header(skb) in the
IFF_TUN case since __virtio_net_hdr_to_skb() unconditionally resets
mac_header.
2. Removing '&& skb->network_header' and the unvalidated
'else if (gso_type)' fallback in __virtio_net_hdr_to_skb() so all GSO
packets without VIRTIO_NET_HDR_F_NEEDS_CSUM are flow-dissected, have
their transport header pulled into linear data, and have
skb->transport_header set.
3. Moving the virtio_net_hdr_match_proto() check to after
skb_flow_dissect_flow_keys_basic(), validating keys.basic.n_proto
against hdr_gso_type.
Fixes: 9e8db5913264 ("net: avoid false positives in untrusted gso validation")
Fixes: d5be7f632bad ("net: validate untrusted gso packets without csum offload")
Fixes: 924a9bc362a5 ("net: check if protocol extracted by virtio_net_hdr_set_proto is correct")
Reported-by: Weiming Shi <bestswngs@gmail.com>
Closes: https://lore.kernel.org/netdev/20260927163117.746432-2-bestswngs@gmail.com/
Signed-off-by: Eric Dumazet <edumazet@kernel.org>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Link: https://patch.msgid.link/20261001191140.2818991-3-edumazet@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-10-01 19:11:39 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-06-03 11:51:20 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
net: always dissect GSO packets in __virtio_net_hdr_to_skb()
Commit 9e8db5913264 ("net: avoid false positives in untrusted gso
validation") added a '&& skb->network_header' check before flow-dissecting
GSO packets without VIRTIO_NET_HDR_F_NEEDS_CSUM in
__virtio_net_hdr_to_skb(), because some callers (such as tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header()) called
virtio_net_hdr_*_to_skb() before initializing skb->network_header and
skb->dev.
Because __alloc_skb() and __build_skb_around() zero-initialize
skb->network_header to 0 (unlike mac_header and transport_header which
are initialized to ~0U), those four callers always had
skb->network_header == 0 and bypassed flow dissection in
__virtio_net_hdr_to_skb(). More generally, skb->network_header is an
offset from skb->head (where 0 is also a valid offset whenever
skb_headroom(skb) is 0), not a boolean flag.
Whenever the 'if (gso_type && skb->network_header)' branch was skipped,
the fallback 'else if (gso_type)' only pulled nh_min_len + thlen (40 bytes
for TCPv4) without dissecting the packet, without validating ip_proto or
n_proto, and without setting skb->transport_header.
If the packet has a malformed network header, it is not rejected and a
subsequent skb_probe_transport_header() also fails, leaving
skb->transport_header at ~0U (0xffff). Similarly, if an IPv4 packet
carries IP options (ihl > 5) or an IPv6 packet carries extension headers,
pulling only nh_min_len + thlen can leave the TCP header outside
skb->head. In both cases, tcp_hdrlen(skb) in skb_gso_transport_seglen()
reads out-of-bounds:
BUG: KASAN: slab-out-of-bounds in skb_gso_transport_seglen
Read of size 2 by task poc/133
skb_gso_transport_seglen (net/core/gso.c:155)
skb_gso_validate_mac_len (net/core/gso.c:270)
tbf_enqueue (net/sched/sch_tbf.c:260)
dev_qdisc_enqueue (net/core/dev.c:4227)
__dev_queue_xmit (net/core/dev.c:4884)
In addition, checking virtio_net_hdr_match_proto() only inside
'if (!skb->protocol)' before flow dissection both skipped validation when
skb->protocol was pre-set by the caller and rejected VLAN-tagged frames
whose outer L2 protocol is ETH_P_8021Q or ETH_P_8021AD.
Fix this by:
1. Initializing skb->dev and skb->network_header (plus skb->protocol for
IFF_TUN) before virtio_net_hdr_*_to_skb() in tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header(). In
tun_get_user(), drop the redundant skb_reset_mac_header(skb) in the
IFF_TUN case since __virtio_net_hdr_to_skb() unconditionally resets
mac_header.
2. Removing '&& skb->network_header' and the unvalidated
'else if (gso_type)' fallback in __virtio_net_hdr_to_skb() so all GSO
packets without VIRTIO_NET_HDR_F_NEEDS_CSUM are flow-dissected, have
their transport header pulled into linear data, and have
skb->transport_header set.
3. Moving the virtio_net_hdr_match_proto() check to after
skb_flow_dissect_flow_keys_basic(), validating keys.basic.n_proto
against hdr_gso_type.
Fixes: 9e8db5913264 ("net: avoid false positives in untrusted gso validation")
Fixes: d5be7f632bad ("net: validate untrusted gso packets without csum offload")
Fixes: 924a9bc362a5 ("net: check if protocol extracted by virtio_net_hdr_set_proto is correct")
Reported-by: Weiming Shi <bestswngs@gmail.com>
Closes: https://lore.kernel.org/netdev/20260927163117.746432-2-bestswngs@gmail.com/
Signed-off-by: Eric Dumazet <edumazet@kernel.org>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Link: https://patch.msgid.link/20261001191140.2818991-3-edumazet@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-10-01 19:11:39 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
|
|
|
|
|
2021-01-06 14:18:40 -08:00
|
|
|
|
2016-11-30 13:17:51 +08:00
|
|
|
|
|
|
|
|
|
2024-04-19 12:08:39 +01:00
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
|
|
|
|
|
2015-02-03 16:36:16 -05:00
|
|
|
|
net: Don't set transport offset to invalid value
If the socket was created with socket(AF_PACKET, SOCK_RAW, 0),
skb->protocol will be unset, __skb_flow_dissect() will fail, and
skb_probe_transport_header() will fall back to the offset_hint, making
the resulting skb_transport_offset incorrect.
If, however, there is no transport header in the packet,
transport_header shouldn't be set to an arbitrary value.
Fix it by leaving the transport offset unset if it couldn't be found, to
be explicit rather than to fill it with some wrong value. It changes the
behavior, but if some code relied on the old behavior, it would be
broken anyway, as the old one is incorrect.
Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-02-21 12:39:57 +00:00
|
|
|
|
2020-04-10 18:20:59 +02:00
|
|
|
|
2013-03-25 20:19:56 +00:00
|
|
|
|
2017-09-04 11:36:09 +08:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-05-28 19:37:49 +09:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-02-12 10:50:55 +01:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
2018-05-28 19:37:49 +09:00
|
|
|
|
2022-11-10 15:31:25 +08:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
2025-09-17 19:39:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
2018-05-28 19:37:49 +09:00
|
|
|
|
2017-08-11 19:41:18 +08:00
|
|
|
|
|
|
|
|
|
2018-04-20 13:18:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2019-03-14 20:19:47 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-16 13:09:53 -07:00
|
|
|
|
2022-03-04 06:55:07 -08:00
|
|
|
|
2019-03-14 20:19:47 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2020-05-30 15:41:31 -04:00
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2020-05-30 15:41:31 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2017-10-17 10:07:44 -07:00
|
|
|
|
2022-11-07 18:00:11 +00:00
|
|
|
|
|
|
|
|
|
2022-03-10 21:14:20 -08:00
|
|
|
|
2022-11-07 18:00:11 +00:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2019-03-14 20:19:47 -07:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2022-11-07 18:00:11 +00:00
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
2022-11-07 18:00:11 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
2017-10-18 12:12:09 -07:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun: Fix memory leak for detached NAPI queue.
syzkaller reported [0] memory leaks of sk and skb related to the TUN
device with no repro, but we can reproduce it easily with:
struct ifreq ifr = {}
int fd_tun, fd_tmp;
char buf[4] = {};
fd_tun = openat(AT_FDCWD, "/dev/net/tun", O_WRONLY, 0);
ifr.ifr_flags = IFF_TUN | IFF_NAPI | IFF_MULTI_QUEUE;
ioctl(fd_tun, TUNSETIFF, &ifr);
ifr.ifr_flags = IFF_DETACH_QUEUE;
ioctl(fd_tun, TUNSETQUEUE, &ifr);
fd_tmp = socket(AF_PACKET, SOCK_PACKET, 0);
ifr.ifr_flags = IFF_UP;
ioctl(fd_tmp, SIOCSIFFLAGS, &ifr);
write(fd_tun, buf, sizeof(buf));
close(fd_tun);
If we enable NAPI and multi-queue on a TUN device, we can put skb into
tfile->sk.sk_write_queue after the queue is detached. We should prevent
it by checking tfile->detached before queuing skb.
Note this must be done under tfile->sk.sk_write_queue.lock because write()
and ioctl(IFF_DETACH_QUEUE) can run concurrently. Otherwise, there would
be a small race window:
write() ioctl(IFF_DETACH_QUEUE)
`- tun_get_user `- __tun_detach
|- if (tfile->detached) |- tun_disable_queue
| `-> false | `- tfile->detached = tun
| `- tun_queue_purge
|- spin_lock_bh(&queue->lock)
`- __skb_queue_tail(queue, skb)
Another solution is to call tun_queue_purge() when closing and
reattaching the detached queue, but it could paper over another
problems. Also, we do the same kind of test for IFF_NAPI_FRAGS.
[0]:
unreferenced object 0xffff88801edbc800 (size 2048):
comm "syz-executor.1", pid 33269, jiffies 4295743834 (age 18.756s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 07 40 00 00 00 00 00 00 00 00 00 00 00 00 ...@............
backtrace:
[<000000008c16ea3d>] __do_kmalloc_node mm/slab_common.c:965 [inline]
[<000000008c16ea3d>] __kmalloc+0x4a/0x130 mm/slab_common.c:979
[<000000003addde56>] kmalloc include/linux/slab.h:563 [inline]
[<000000003addde56>] sk_prot_alloc+0xef/0x1b0 net/core/sock.c:2035
[<000000003e20621f>] sk_alloc+0x36/0x2f0 net/core/sock.c:2088
[<0000000028e43843>] tun_chr_open+0x3d/0x190 drivers/net/tun.c:3438
[<000000001b0f1f28>] misc_open+0x1a6/0x1f0 drivers/char/misc.c:165
[<000000004376f706>] chrdev_open+0x111/0x300 fs/char_dev.c:414
[<00000000614d379f>] do_dentry_open+0x2f9/0x750 fs/open.c:920
[<000000008eb24774>] do_open fs/namei.c:3636 [inline]
[<000000008eb24774>] path_openat+0x143f/0x1a30 fs/namei.c:3791
[<00000000955077b5>] do_filp_open+0xce/0x1c0 fs/namei.c:3818
[<00000000b78973b0>] do_sys_openat2+0xf0/0x260 fs/open.c:1356
[<00000000057be699>] do_sys_open fs/open.c:1372 [inline]
[<00000000057be699>] __do_sys_openat fs/open.c:1388 [inline]
[<00000000057be699>] __se_sys_openat fs/open.c:1383 [inline]
[<00000000057be699>] __x64_sys_openat+0x83/0xf0 fs/open.c:1383
[<00000000a7d2182d>] do_syscall_x64 arch/x86/entry/common.c:50 [inline]
[<00000000a7d2182d>] do_syscall_64+0x3c/0x90 arch/x86/entry/common.c:80
[<000000004cc4e8c4>] entry_SYSCALL_64_after_hwframe+0x72/0xdc
unreferenced object 0xffff88802f671700 (size 240):
comm "syz-executor.1", pid 33269, jiffies 4295743854 (age 18.736s)
hex dump (first 32 bytes):
68 c9 db 1e 80 88 ff ff 68 c9 db 1e 80 88 ff ff h.......h.......
00 c0 7b 2f 80 88 ff ff 00 c8 db 1e 80 88 ff ff ..{/............
backtrace:
[<00000000e9d9fdb6>] __alloc_skb+0x223/0x250 net/core/skbuff.c:644
[<000000002c3e4e0b>] alloc_skb include/linux/skbuff.h:1288 [inline]
[<000000002c3e4e0b>] alloc_skb_with_frags+0x6f/0x350 net/core/skbuff.c:6378
[<00000000825f98d7>] sock_alloc_send_pskb+0x3ac/0x3e0 net/core/sock.c:2729
[<00000000e9eb3df3>] tun_alloc_skb drivers/net/tun.c:1529 [inline]
[<00000000e9eb3df3>] tun_get_user+0x5e1/0x1f90 drivers/net/tun.c:1841
[<0000000053096912>] tun_chr_write_iter+0xac/0x120 drivers/net/tun.c:2035
[<00000000b9282ae0>] call_write_iter include/linux/fs.h:1868 [inline]
[<00000000b9282ae0>] new_sync_write fs/read_write.c:491 [inline]
[<00000000b9282ae0>] vfs_write+0x40f/0x530 fs/read_write.c:584
[<00000000524566e4>] ksys_write+0xa1/0x170 fs/read_write.c:637
[<00000000a7d2182d>] do_syscall_x64 arch/x86/entry/common.c:50 [inline]
[<00000000a7d2182d>] do_syscall_64+0x3c/0x90 arch/x86/entry/common.c:80
[<000000004cc4e8c4>] entry_SYSCALL_64_after_hwframe+0x72/0xdc
Fixes: cde8b15f1aab ("tuntap: add ioctl to attach or detach a file form tuntap device")
Reported-by: syzkaller <syzkaller@googlegroups.com>
Signed-off-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-05-15 11:42:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-03-06 22:57:46 +01:00
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2019-03-14 20:19:47 -07:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-20 09:23:23 +00:00
|
|
|
|
2022-11-10 15:31:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-06-19 15:36:49 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2020-11-20 07:59:54 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-11-20 07:59:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
2025-02-07 15:10:54 +09:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-07 15:10:54 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-06-08 14:33:08 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2011-06-08 14:33:08 +00:00
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2014-11-13 16:54:14 +08:00
|
|
|
|
2014-11-03 04:30:13 +08:00
|
|
|
|
2014-11-03 04:30:14 +08:00
|
|
|
|
2025-02-07 15:10:54 +09:00
|
|
|
|
2014-11-03 04:30:13 +08:00
|
|
|
|
2015-01-13 17:13:44 +01:00
|
|
|
|
2014-11-03 04:30:13 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2017-02-03 18:20:48 -05:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-03 04:30:14 +08:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
2016-06-08 16:09:20 +03:00
|
|
|
|
2026-06-06 22:44:28 -07:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
2025-02-07 15:10:54 +09:00
|
|
|
|
|
|
|
|
|
2008-07-03 03:48:02 -07:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-07 15:10:54 +09:00
|
|
|
|
|
|
|
|
|
2008-07-03 03:48:02 -07:00
|
|
|
|
|
|
|
|
|
2014-11-03 04:30:13 +08:00
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-25 13:00:33 +08:00
|
|
|
|
2015-01-13 17:13:44 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-25 13:00:33 +08:00
|
|
|
|
|
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
|
|
|
|
|
2013-07-25 13:00:33 +08:00
|
|
|
|
|
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
|
|
|
|
|
2013-07-25 13:00:33 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2013-07-25 13:00:33 +08:00
|
|
|
|
|
|
|
|
|
2016-04-13 10:52:20 +02:00
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2016-07-04 13:53:38 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2016-07-04 13:53:38 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-07-05 20:14:16 +01:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2019-02-23 12:53:13 +01:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2016-07-04 13:53:38 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-07-04 13:53:38 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-02-25 21:13:13 +01:00
|
|
|
|
2019-07-05 20:14:16 +01:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2016-07-04 13:53:38 +08:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2014-11-07 13:52:07 -05:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-07 13:52:07 -05:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-12-01 05:10:37 -05:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2014-11-07 13:52:07 -05:00
|
|
|
|
2017-12-01 05:10:37 -05:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2017-05-17 12:14:43 +08:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2017-05-17 12:14:43 +08:00
|
|
|
|
|
|
|
|
|
2014-11-07 21:22:23 +08:00
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
xdp: transition into using xdp_frame for return API
Changing API xdp_return_frame() to take struct xdp_frame as argument,
seems like a natural choice. But there are some subtle performance
details here that needs extra care, which is a deliberate choice.
When de-referencing xdp_frame on a remote CPU during DMA-TX
completion, result in the cache-line is change to "Shared"
state. Later when the page is reused for RX, then this xdp_frame
cache-line is written, which change the state to "Modified".
This situation already happens (naturally) for, virtio_net, tun and
cpumap as the xdp_frame pointer is the queued object. In tun and
cpumap, the ptr_ring is used for efficiently transferring cache-lines
(with pointers) between CPUs. Thus, the only option is to
de-referencing xdp_frame.
It is only the ixgbe driver that had an optimization, in which it can
avoid doing the de-reference of xdp_frame. The driver already have
TX-ring queue, which (in case of remote DMA-TX completion) have to be
transferred between CPUs anyhow. In this data area, we stored a
struct xdp_mem_info and a data pointer, which allowed us to avoid
de-referencing xdp_frame.
To compensate for this, a prefetchw is used for telling the cache
coherency protocol about our access pattern. My benchmarks show that
this prefetchw is enough to compensate the ixgbe driver.
V7: Adjust for commit d9314c474d4f ("i40e: add support for XDP_REDIRECT")
V8: Adjust for commit bd658dda4237 ("net/mlx5e: Separate dma base address
and offset in dma_sync call")
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 16:46:32 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-07 13:52:07 -05:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2014-11-07 13:52:07 -05:00
|
|
|
|
2020-11-20 07:59:54 -07:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-11-20 07:59:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-12-10 22:05:45 -05:00
|
|
|
|
2013-12-06 14:16:51 +08:00
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-22 10:55:38 +08:00
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2026-02-21 16:37:42 -08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-08 12:02:30 +08:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-08 12:02:30 +08:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-08 12:02:30 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2019-10-07 12:21:05 -07:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
2013-01-14 07:12:19 +00:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
2012-10-31 19:46:02 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-02-07 16:48:55 -08:00
|
|
|
|
|
|
|
|
|
2018-06-02 17:49:53 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
net: Fix inconsistent teardown and release of private netdev state.
Network devices can allocate reasources and private memory using
netdev_ops->ndo_init(). However, the release of these resources
can occur in one of two different places.
Either netdev_ops->ndo_uninit() or netdev->destructor().
The decision of which operation frees the resources depends upon
whether it is necessary for all netdev refs to be released before it
is safe to perform the freeing.
netdev_ops->ndo_uninit() presumably can occur right after the
NETDEV_UNREGISTER notifier completes and the unicast and multicast
address lists are flushed.
netdev->destructor(), on the other hand, does not run until the
netdev references all go away.
Further complicating the situation is that netdev->destructor()
almost universally does also a free_netdev().
This creates a problem for the logic in register_netdevice().
Because all callers of register_netdevice() manage the freeing
of the netdev, and invoke free_netdev(dev) if register_netdevice()
fails.
If netdev_ops->ndo_init() succeeds, but something else fails inside
of register_netdevice(), it does call ndo_ops->ndo_uninit(). But
it is not able to invoke netdev->destructor().
This is because netdev->destructor() will do a free_netdev() and
then the caller of register_netdevice() will do the same.
However, this means that the resources that would normally be released
by netdev->destructor() will not be.
Over the years drivers have added local hacks to deal with this, by
invoking their destructor parts by hand when register_netdevice()
fails.
Many drivers do not try to deal with this, and instead we have leaks.
Let's close this hole by formalizing the distinction between what
private things need to be freed up by netdev->destructor() and whether
the driver needs unregister_netdevice() to perform the free_netdev().
netdev->priv_destructor() performs all actions to free up the private
resources that used to be freed by netdev->destructor(), except for
free_netdev().
netdev->needs_free_netdev is a boolean that indicates whether
free_netdev() should be done at the end of unregister_netdevice().
Now, register_netdevice() can sanely release all resources after
ndo_ops->ndo_init() succeeds, by invoking both ndo_ops->ndo_uninit()
and netdev->priv_destructor().
And at the end of unregister_netdevice(), we invoke
netdev->priv_destructor() and optionally call free_netdev().
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-05-08 12:52:56 -04:00
|
|
|
|
|
|
|
|
|
2016-04-08 13:26:48 +08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-06-25 23:56:01 +02:00
|
|
|
|
|
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
2018-11-29 14:45:39 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
|
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-16 11:03:07 +01:00
|
|
|
|
|
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
|
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2010-04-29 11:01:49 +00:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-11-29 20:03:10 -08:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
2010-04-29 11:01:49 +00:00
|
|
|
|
|
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
|
|
|
|
|
2009-06-03 21:45:55 -07:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
2018-11-15 17:43:10 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2018-11-15 17:43:10 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2018-12-03 18:09:24 +09:00
|
|
|
|
2025-07-01 09:03:51 +08:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2025-07-01 09:03:51 +08:00
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2018-11-15 17:43:10 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2026-05-20 09:00:21 -07:00
|
|
|
|
|
|
|
|
|
2024-07-24 10:04:52 -07:00
|
|
|
|
2026-05-20 09:00:21 -07:00
|
|
|
|
2024-07-24 10:04:52 -07:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-12-22 22:09:28 +01:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2020-08-23 17:36:59 -05:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-15 17:43:10 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-05-21 09:33:13 -07:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-05 21:34:34 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
net: always dissect GSO packets in __virtio_net_hdr_to_skb()
Commit 9e8db5913264 ("net: avoid false positives in untrusted gso
validation") added a '&& skb->network_header' check before flow-dissecting
GSO packets without VIRTIO_NET_HDR_F_NEEDS_CSUM in
__virtio_net_hdr_to_skb(), because some callers (such as tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header()) called
virtio_net_hdr_*_to_skb() before initializing skb->network_header and
skb->dev.
Because __alloc_skb() and __build_skb_around() zero-initialize
skb->network_header to 0 (unlike mac_header and transport_header which
are initialized to ~0U), those four callers always had
skb->network_header == 0 and bypassed flow dissection in
__virtio_net_hdr_to_skb(). More generally, skb->network_header is an
offset from skb->head (where 0 is also a valid offset whenever
skb_headroom(skb) is 0), not a boolean flag.
Whenever the 'if (gso_type && skb->network_header)' branch was skipped,
the fallback 'else if (gso_type)' only pulled nh_min_len + thlen (40 bytes
for TCPv4) without dissecting the packet, without validating ip_proto or
n_proto, and without setting skb->transport_header.
If the packet has a malformed network header, it is not rejected and a
subsequent skb_probe_transport_header() also fails, leaving
skb->transport_header at ~0U (0xffff). Similarly, if an IPv4 packet
carries IP options (ihl > 5) or an IPv6 packet carries extension headers,
pulling only nh_min_len + thlen can leave the TCP header outside
skb->head. In both cases, tcp_hdrlen(skb) in skb_gso_transport_seglen()
reads out-of-bounds:
BUG: KASAN: slab-out-of-bounds in skb_gso_transport_seglen
Read of size 2 by task poc/133
skb_gso_transport_seglen (net/core/gso.c:155)
skb_gso_validate_mac_len (net/core/gso.c:270)
tbf_enqueue (net/sched/sch_tbf.c:260)
dev_qdisc_enqueue (net/core/dev.c:4227)
__dev_queue_xmit (net/core/dev.c:4884)
In addition, checking virtio_net_hdr_match_proto() only inside
'if (!skb->protocol)' before flow dissection both skipped validation when
skb->protocol was pre-set by the caller and rejected VLAN-tagged frames
whose outer L2 protocol is ETH_P_8021Q or ETH_P_8021AD.
Fix this by:
1. Initializing skb->dev and skb->network_header (plus skb->protocol for
IFF_TUN) before virtio_net_hdr_*_to_skb() in tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header(). In
tun_get_user(), drop the redundant skb_reset_mac_header(skb) in the
IFF_TUN case since __virtio_net_hdr_to_skb() unconditionally resets
mac_header.
2. Removing '&& skb->network_header' and the unvalidated
'else if (gso_type)' fallback in __virtio_net_hdr_to_skb() so all GSO
packets without VIRTIO_NET_HDR_F_NEEDS_CSUM are flow-dissected, have
their transport header pulled into linear data, and have
skb->transport_header set.
3. Moving the virtio_net_hdr_match_proto() check to after
skb_flow_dissect_flow_keys_basic(), validating keys.basic.n_proto
against hdr_gso_type.
Fixes: 9e8db5913264 ("net: avoid false positives in untrusted gso validation")
Fixes: d5be7f632bad ("net: validate untrusted gso packets without csum offload")
Fixes: 924a9bc362a5 ("net: check if protocol extracted by virtio_net_hdr_set_proto is correct")
Reported-by: Weiming Shi <bestswngs@gmail.com>
Closes: https://lore.kernel.org/netdev/20260927163117.746432-2-bestswngs@gmail.com/
Signed-off-by: Eric Dumazet <edumazet@kernel.org>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Link: https://patch.msgid.link/20261001191140.2818991-3-edumazet@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-10-01 19:11:39 +00:00
|
|
|
|
|
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: Don't set transport offset to invalid value
If the socket was created with socket(AF_PACKET, SOCK_RAW, 0),
skb->protocol will be unset, __skb_flow_dissect() will fail, and
skb_probe_transport_header() will fall back to the offset_hint, making
the resulting skb_transport_offset incorrect.
If, however, there is no transport header in the packet,
transport_header shouldn't be set to an arbitrary value.
Fix it by leaving the transport offset unset if it couldn't be found, to
be explicit rather than to fill it with some wrong value. It changes the
behavior, but if some code relied on the old behavior, it would be
broken anyway, as the old one is incorrect.
Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-02-21 12:39:57 +00:00
|
|
|
|
2020-04-10 18:20:59 +02:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2024-02-12 10:50:55 +01:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2018-11-07 10:34:36 +01:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun: Fix memory leak for detached NAPI queue.
syzkaller reported [0] memory leaks of sk and skb related to the TUN
device with no repro, but we can reproduce it easily with:
struct ifreq ifr = {}
int fd_tun, fd_tmp;
char buf[4] = {};
fd_tun = openat(AT_FDCWD, "/dev/net/tun", O_WRONLY, 0);
ifr.ifr_flags = IFF_TUN | IFF_NAPI | IFF_MULTI_QUEUE;
ioctl(fd_tun, TUNSETIFF, &ifr);
ifr.ifr_flags = IFF_DETACH_QUEUE;
ioctl(fd_tun, TUNSETQUEUE, &ifr);
fd_tmp = socket(AF_PACKET, SOCK_PACKET, 0);
ifr.ifr_flags = IFF_UP;
ioctl(fd_tmp, SIOCSIFFLAGS, &ifr);
write(fd_tun, buf, sizeof(buf));
close(fd_tun);
If we enable NAPI and multi-queue on a TUN device, we can put skb into
tfile->sk.sk_write_queue after the queue is detached. We should prevent
it by checking tfile->detached before queuing skb.
Note this must be done under tfile->sk.sk_write_queue.lock because write()
and ioctl(IFF_DETACH_QUEUE) can run concurrently. Otherwise, there would
be a small race window:
write() ioctl(IFF_DETACH_QUEUE)
`- tun_get_user `- __tun_detach
|- if (tfile->detached) |- tun_disable_queue
| `-> false | `- tfile->detached = tun
| `- tun_queue_purge
|- spin_lock_bh(&queue->lock)
`- __skb_queue_tail(queue, skb)
Another solution is to call tun_queue_purge() when closing and
reattaching the detached queue, but it could paper over another
problems. Also, we do the same kind of test for IFF_NAPI_FRAGS.
[0]:
unreferenced object 0xffff88801edbc800 (size 2048):
comm "syz-executor.1", pid 33269, jiffies 4295743834 (age 18.756s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 07 40 00 00 00 00 00 00 00 00 00 00 00 00 ...@............
backtrace:
[<000000008c16ea3d>] __do_kmalloc_node mm/slab_common.c:965 [inline]
[<000000008c16ea3d>] __kmalloc+0x4a/0x130 mm/slab_common.c:979
[<000000003addde56>] kmalloc include/linux/slab.h:563 [inline]
[<000000003addde56>] sk_prot_alloc+0xef/0x1b0 net/core/sock.c:2035
[<000000003e20621f>] sk_alloc+0x36/0x2f0 net/core/sock.c:2088
[<0000000028e43843>] tun_chr_open+0x3d/0x190 drivers/net/tun.c:3438
[<000000001b0f1f28>] misc_open+0x1a6/0x1f0 drivers/char/misc.c:165
[<000000004376f706>] chrdev_open+0x111/0x300 fs/char_dev.c:414
[<00000000614d379f>] do_dentry_open+0x2f9/0x750 fs/open.c:920
[<000000008eb24774>] do_open fs/namei.c:3636 [inline]
[<000000008eb24774>] path_openat+0x143f/0x1a30 fs/namei.c:3791
[<00000000955077b5>] do_filp_open+0xce/0x1c0 fs/namei.c:3818
[<00000000b78973b0>] do_sys_openat2+0xf0/0x260 fs/open.c:1356
[<00000000057be699>] do_sys_open fs/open.c:1372 [inline]
[<00000000057be699>] __do_sys_openat fs/open.c:1388 [inline]
[<00000000057be699>] __se_sys_openat fs/open.c:1383 [inline]
[<00000000057be699>] __x64_sys_openat+0x83/0xf0 fs/open.c:1383
[<00000000a7d2182d>] do_syscall_x64 arch/x86/entry/common.c:50 [inline]
[<00000000a7d2182d>] do_syscall_64+0x3c/0x90 arch/x86/entry/common.c:80
[<000000004cc4e8c4>] entry_SYSCALL_64_after_hwframe+0x72/0xdc
unreferenced object 0xffff88802f671700 (size 240):
comm "syz-executor.1", pid 33269, jiffies 4295743854 (age 18.736s)
hex dump (first 32 bytes):
68 c9 db 1e 80 88 ff ff 68 c9 db 1e 80 88 ff ff h.......h.......
00 c0 7b 2f 80 88 ff ff 00 c8 db 1e 80 88 ff ff ..{/............
backtrace:
[<00000000e9d9fdb6>] __alloc_skb+0x223/0x250 net/core/skbuff.c:644
[<000000002c3e4e0b>] alloc_skb include/linux/skbuff.h:1288 [inline]
[<000000002c3e4e0b>] alloc_skb_with_frags+0x6f/0x350 net/core/skbuff.c:6378
[<00000000825f98d7>] sock_alloc_send_pskb+0x3ac/0x3e0 net/core/sock.c:2729
[<00000000e9eb3df3>] tun_alloc_skb drivers/net/tun.c:1529 [inline]
[<00000000e9eb3df3>] tun_get_user+0x5e1/0x1f90 drivers/net/tun.c:1841
[<0000000053096912>] tun_chr_write_iter+0xac/0x120 drivers/net/tun.c:2035
[<00000000b9282ae0>] call_write_iter include/linux/fs.h:1868 [inline]
[<00000000b9282ae0>] new_sync_write fs/read_write.c:491 [inline]
[<00000000b9282ae0>] vfs_write+0x40f/0x530 fs/read_write.c:584
[<00000000524566e4>] ksys_write+0xa1/0x170 fs/read_write.c:637
[<00000000a7d2182d>] do_syscall_x64 arch/x86/entry/common.c:50 [inline]
[<00000000a7d2182d>] do_syscall_64+0x3c/0x90 arch/x86/entry/common.c:80
[<000000004cc4e8c4>] entry_SYSCALL_64_after_hwframe+0x72/0xdc
Fixes: cde8b15f1aab ("tuntap: add ioctl to attach or detach a file form tuntap device")
Reported-by: syzkaller <syzkaller@googlegroups.com>
Signed-off-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-05-15 11:42:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
2018-12-11 11:43:07 +09:00
|
|
|
|
|
|
|
|
|
2020-11-07 21:50:56 +01:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2015-03-02 15:37:48 +08:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2018-09-12 11:17:06 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-06-19 15:36:49 -04:00
|
|
|
|
2022-03-03 10:24:40 +08:00
|
|
|
|
|
|
|
|
|
2024-07-04 16:48:15 +02:00
|
|
|
|
2018-11-17 16:53:46 -08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2018-11-17 16:53:46 -08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
2024-07-04 16:48:15 +02:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
xdp: Use bulking for non-map XDP_REDIRECT and consolidate code paths
Since the bulk queue used by XDP_REDIRECT now lives in struct net_device,
we can re-use the bulking for the non-map version of the bpf_redirect()
helper. This is a simple matter of having xdp_do_redirect_slow() queue the
frame on the bulk queue instead of sending it out with __bpf_tx_xdp().
Unfortunately we can't make the bpf_redirect() helper return an error if
the ifindex doesn't exit (as bpf_redirect_map() does), because we don't
have a reference to the network namespace of the ingress device at the time
the helper is called. So we have to leave it as-is and keep the device
lookup in xdp_do_redirect_slow().
Since this leaves less reason to have the non-map redirect code in a
separate function, so we get rid of the xdp_do_redirect_slow() function
entirely. This does lose us the tracepoint disambiguation, but fortunately
the xdp_redirect and xdp_redirect_map tracepoints use the same tracepoint
entry structures. This means both can contain a map index, so we can just
amend the tracepoint definitions so we always emit the xdp_redirect(_err)
tracepoints, but with the map ID only populated if a map is present. This
means we retire the xdp_redirect_map(_err) tracepoints entirely, but keep
the definitions around in case someone is still listening for them.
With this change, the performance of the xdp_redirect sample program goes
from 5Mpps to 8.4Mpps (a 68% increase).
Since the flush functions are no longer map-specific, rename the flush()
functions to drop _map from their names. One of the renamed functions is
the xdp_do_flush_map() callback used in all the xdp-enabled drivers. To
keep from having to update all drivers, use a #define to keep the old name
working, and only update the virtual drivers in this patch.
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: John Fastabend <john.fastabend@gmail.com>
Link: https://lore.kernel.org/bpf/157918768505.1458396.17518057312953572912.stgit@toke.dk
2020-01-16 16:14:45 +01:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2022-02-28 11:38:05 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-07-04 16:48:15 +02:00
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-15 17:43:10 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:06 +08:00
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
|
|
|
|
|
2018-09-12 11:17:07 +08:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
2015-03-02 15:37:48 +08:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2017-12-01 05:10:37 -05:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2017-12-01 05:10:37 -05:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2013-07-19 19:40:10 +02:00
|
|
|
|
2013-04-24 21:59:23 +00:00
|
|
|
|
2017-12-01 05:10:37 -05:00
|
|
|
|
2013-04-24 21:59:23 +00:00
|
|
|
|
2013-07-19 19:40:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2014-12-25 23:05:03 -08:00
|
|
|
|
2013-12-10 22:05:45 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-24 21:59:23 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
2017-12-01 05:10:37 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
2017-12-01 05:10:37 -05:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2018-04-17 16:45:47 +02:00
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:28 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
2014-11-19 14:44:40 +02:00
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
2021-05-19 10:38:50 +08:00
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:43 +08:00
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
2021-05-19 10:38:50 +08:00
|
|
|
|
|
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
2012-02-07 16:48:55 -08:00
|
|
|
|
2022-09-28 19:49:43 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
2021-05-19 10:38:50 +08:00
|
|
|
|
|
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
2012-02-07 16:48:55 -08:00
|
|
|
|
2022-09-28 19:49:43 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
|
|
|
|
|
2021-05-19 10:38:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
2015-02-04 14:37:34 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-04-16 00:41:16 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set
This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.
IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.
The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.
A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.
The documentation in tuntap.rst is updated accordingly.
Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:41 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set
This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.
IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.
The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.
A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.
The documentation in tuntap.rst is updated accordingly.
Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:41 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-01-11 16:59:33 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:15 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 10:56:20 +00:00
|
|
|
|
|
|
|
|
|
2009-04-27 03:23:54 -07:00
|
|
|
|
|
|
|
|
|
2009-01-20 10:56:20 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-05-28 18:32:11 +00:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2013-05-28 18:32:11 +00:00
|
|
|
|
|
|
|
|
|
2025-02-04 11:10:06 -05:00
|
|
|
|
2009-08-28 18:12:43 -04:00
|
|
|
|
2013-01-14 07:12:19 +00:00
|
|
|
|
2009-08-28 18:12:43 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-22 13:49:14 -07:00
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2019-09-10 18:56:57 +08:00
|
|
|
|
2009-01-20 10:57:48 +00:00
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2013-04-22 20:40:39 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
2013-04-22 20:40:39 +00:00
|
|
|
|
|
|
|
|
|
2018-04-10 16:28:55 +02:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
|
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-01-23 03:59:12 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-11-18 21:34:11 +00:00
|
|
|
|
2006-06-22 16:07:52 -07:00
|
|
|
|
2009-08-28 18:12:43 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-06-22 16:07:52 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2009-09-16 21:36:13 +00:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
net: set name_assign_type in alloc_netdev()
Extend alloc_netdev{,_mq{,s}}() to take name_assign_type as argument, and convert
all users to pass NET_NAME_UNKNOWN.
Coccinelle patch:
@@
expression sizeof_priv, name, setup, txqs, rxqs, count;
@@
(
-alloc_netdev_mqs(sizeof_priv, name, setup, txqs, rxqs)
+alloc_netdev_mqs(sizeof_priv, name, NET_NAME_UNKNOWN, setup, txqs, rxqs)
|
-alloc_netdev_mq(sizeof_priv, name, setup, count)
+alloc_netdev_mq(sizeof_priv, name, NET_NAME_UNKNOWN, setup, count)
|
-alloc_netdev(sizeof_priv, name, setup)
+alloc_netdev(sizeof_priv, name, NET_NAME_UNKNOWN, setup)
)
v9: move comments here from the wrong commit
Signed-off-by: Tom Gundersen <teg@jklm.no>
Reviewed-by: David Herrmann <dh.herrmann@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-07-14 16:37:24 +02:00
|
|
|
|
|
|
|
|
|
2013-01-23 03:59:12 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-04-16 00:41:53 -07:00
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
tun: Add ability to create tun device with given index
Tun devices cannot be created with ifidex user wants, but it's
required by checkpoint-restore project.
Long time ago such ability was implemented for rtnl_ops-based
interface for creating links (9c7dafbf net: Allow to create links
with given ifindex), but the only API for creating and managing
tuntap devices is ioctl-based and is evolving with adding new ones
(cde8b15f tuntap: add ioctl to attach or detach a file form tuntap
device).
Following that trend, here's how a new ioctl that sets the ifindex
for device, that _will_ be created by TUNSETIFF ioctl looks like.
So those who want a tuntap device with the ifindex N, should open
the tun device, call ioctl(fd, TUNSETIFINDEX, &N), then call TUNSETIFF.
If the index N is busy, then the register_netdev will find this out
and the ioctl would be failed with -EBUSY.
If setifindex is not called, then it will be generated as before.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-21 14:31:38 +04:00
|
|
|
|
2015-02-04 14:37:34 +01:00
|
|
|
|
2008-11-19 22:10:37 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2010-03-17 17:45:01 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2016-02-26 10:45:40 +01:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
2016-04-13 10:52:20 +02:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
2012-12-02 17:19:45 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2021-12-16 13:25:32 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-01-18 03:15:39 -08:00
|
|
|
|
2019-09-10 18:56:57 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2022-09-20 12:48:25 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-28 00:38:02 +00:00
|
|
|
|
2008-07-10 16:59:11 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set
This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.
IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.
The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.
A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.
The documentation in tuntap.rst is updated accordingly.
Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:41 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-07-10 16:59:11 -07:00
|
|
|
|
2025-08-12 10:22:44 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-20 12:16:53 +03:00
|
|
|
|
2008-08-15 15:09:56 -07:00
|
|
|
|
2025-08-12 10:22:44 +02:00
|
|
|
|
2008-08-15 15:09:56 -07:00
|
|
|
|
2009-05-09 22:54:21 -07:00
|
|
|
|
2008-08-15 15:09:56 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
|
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
2011-11-15 15:29:55 +00:00
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
|
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: accept UFO datagrams from tuntap and packet
Tuntap and similar devices can inject GSO packets. Accept type
VIRTIO_NET_HDR_GSO_UDP, even though not generating UFO natively.
Processes are expected to use feature negotiation such as TUNSETOFFLOAD
to detect supported offload types and refrain from injecting other
packets. This process breaks down with live migration: guest kernels
do not renegotiate flags, so destination hosts need to expose all
features that the source host does.
Partially revert the UFO removal from 182e0b6b5846~1..d9d30adf5677.
This patch introduces nearly(*) no new code to simplify verification.
It brings back verbatim tuntap UFO negotiation, VIRTIO_NET_HDR_GSO_UDP
insertion and software UFO segmentation.
It does not reinstate protocol stack support, hardware offload
(NETIF_F_UFO), SKB_GSO_UDP tunneling in SKB_GSO_SOFTWARE or reception
of VIRTIO_NET_HDR_GSO_UDP packets in tuntap.
To support SKB_GSO_UDP reappearing in the stack, also reinstate
logic in act_csum and openvswitch. Achieve equivalence with v4.13 HEAD
by squashing in commit 939912216fa8 ("net: skb_needs_check() removes
CHECKSUM_UNNECESSARY check for tx.") and reverting commit 8d63bee643f1
("net: avoid skb_warn_bad_offload false positives on UFO").
(*) To avoid having to bring back skb_shinfo(skb)->ip6_frag_id,
ipv6_proxy_select_ident is changed to return a __be32 and this is
assigned directly to the frag_hdr. Also, SKB_GSO_UDP is inserted
at the end of the enum to minimize code churn.
Tested
Booted a v4.13 guest kernel with QEMU. On a host kernel before this
patch `ethtool -k eth0` shows UFO disabled. After the patch, it is
enabled, same as on a v4.13 host kernel.
A UFO packet sent from the guest appears on the tap device:
host:
nc -l -p -u 8000 &
tcpdump -n -i tap0
guest:
dd if=/dev/zero of=payload.txt bs=1 count=2000
nc -u 192.16.1.1 8000 < payload.txt
Direct tap to tap transmission of VIRTIO_NET_HDR_GSO_UDP succeeds,
packets arriving fragmented:
./with_tap_pair.sh ./tap_send_ufo tap0 tap1
(from https://github.com/wdebruij/kerneltools/tree/master/tests)
Changes
v1 -> v2
- simplified set_offload change (review comment)
- documented test procedure
Link: http://lkml.kernel.org/r/<CAF=yD-LuUeDuL9YWPJD9ykOZ0QCjNeznPDr6whqZ9NGMNF12Mw@mail.gmail.com>
Fixes: fb652fdfe837 ("macvlan/macvtap: Remove NETIF_F_UFO advertisement.")
Reported-by: Michal Kubecek <mkubecek@suse.cz>
Signed-off-by: Willem de Bruijn <willemb@google.com>
Acked-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-11-21 10:22:25 -05:00
|
|
|
|
|
|
|
|
|
2022-12-07 13:35:55 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-08 17:55:30 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2017-03-16 22:44:10 +03:00
|
|
|
|
|
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
2016-04-05 17:10:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
2016-04-05 17:10:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2013-01-14 07:12:19 +00:00
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
2013-01-14 07:12:19 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-28 14:51:49 -07:00
|
|
|
|
2019-09-10 18:56:57 +08:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2013-01-11 16:59:32 +00:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
|
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-14 07:12:19 +00:00
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-07-31 00:17:20 -04:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2021-04-06 18:45:54 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-01-20 11:01:48 +00:00
|
|
|
|
2018-05-08 19:21:34 +03:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2023-10-16 18:08:51 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-02-07 16:48:55 -08:00
|
|
|
|
|
|
|
|
|
2023-10-16 18:08:51 +00:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-02-14 16:40:14 +03:00
|
|
|
|
|
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-07-30 14:52:48 -07:00
|
|
|
|
2012-07-29 19:45:14 +00:00
|
|
|
|
2012-07-30 14:52:48 -07:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 14:44:40 +02:00
|
|
|
|
|
|
|
|
|
2022-09-20 12:48:25 -07:00
|
|
|
|
|
|
|
|
|
2018-05-08 19:21:34 +03:00
|
|
|
|
2012-10-31 19:46:01 +00:00
|
|
|
|
2018-05-08 19:21:34 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2009-08-06 14:22:44 +00:00
|
|
|
|
|
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2016-10-25 22:26:09 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2018-02-14 16:40:14 +03:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-08-06 14:22:44 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
2009-08-06 14:22:44 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
tun: Add ability to create tun device with given index
Tun devices cannot be created with ifidex user wants, but it's
required by checkpoint-restore project.
Long time ago such ability was implemented for rtnl_ops-based
interface for creating links (9c7dafbf net: Allow to create links
with given ifindex), but the only API for creating and managing
tuntap devices is ioctl-based and is evolving with adding new ones
(cde8b15f tuntap: add ioctl to attach or detach a file form tuntap
device).
Following that trend, here's how a new ioctl that sets the ifindex
for device, that _will_ be created by TUNSETIFF ioctl looks like.
So those who want a tuntap device with the ifindex N, should open
the tun device, call ioctl(fd, TUNSETIFINDEX, &N), then call TUNSETIFF.
If the index N is busy, then the register_netdev will find this out
and the ioctl would be failed with -EBUSY.
If setifindex is not called, then it will be generated as before.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-21 14:31:38 +04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-10-16 18:08:51 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tun: Add ability to create tun device with given index
Tun devices cannot be created with ifidex user wants, but it's
required by checkpoint-restore project.
Long time ago such ability was implemented for rtnl_ops-based
interface for creating links (9c7dafbf net: Allow to create links
with given ifindex), but the only API for creating and managing
tuntap devices is ioctl-based and is evolving with adding new ones
(cde8b15f tuntap: add ioctl to attach or detach a file form tuntap
device).
Following that trend, here's how a new ioctl that sets the ifindex
for device, that _will_ be created by TUNSETIFF ioctl looks like.
So those who want a tuntap device with the ifindex N, should open
the tun device, call ioctl(fd, TUNSETIFINDEX, &N), then call TUNSETIFF.
If the index N is busy, then the register_netdev will find this out
and the ioctl would be failed with -EBUSY.
If setifindex is not called, then it will be generated as before.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-21 14:31:38 +04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-08-06 14:22:44 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-08-06 14:22:44 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2019-03-20 12:16:42 +03:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-08-15 15:09:56 -07:00
|
|
|
|
2019-03-20 12:16:53 +03:00
|
|
|
|
2008-08-15 15:09:56 -07:00
|
|
|
|
2013-08-21 14:32:00 +04:00
|
|
|
|
|
|
|
|
|
tun: Allow to skip filter on attach
There's a small problem with sk-filters on tun devices. Consider
an application doing this sequence of steps:
fd = open("/dev/net/tun");
ioctl(fd, TUNSETIFF, { .ifr_name = "tun0" });
ioctl(fd, TUNATTACHFILTER, &my_filter);
ioctl(fd, TUNSETPERSIST, 1);
close(fd);
At that point the tun0 will remain in the system and will keep in
mind that there should be a socket filter at address '&my_filter'.
If after that we do
fd = open("/dev/net/tun");
ioctl(fd, TUNSETIFF, { .ifr_name = "tun0" });
we most likely receive the -EFAULT error, since tun_attach() would
try to connect the filter back. But (!) if we provide a filter at
address &my_filter, then tun0 will be created and the "new" filter
would be attached, but application may not know about that.
This may create certain problems to anyone using tun-s, but it's
critical problem for c/r -- if we meet a persistent tun device
with a filter in mind, we will not be able to attach to it to dump
its state (flags, owner, address, vnethdr size, etc.).
The proposal is to allow to attach to tun device (with TUNSETIFF)
w/o attaching the filter to the tun-file's socket. After this
attach app may e.g clean the device by dropping the filter, it
doesn't want to have one, or (in case of c/r) get information
about the device with tun ioctls.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-21 14:32:21 +04:00
|
|
|
|
|
|
|
|
|
2013-08-21 14:32:00 +04:00
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2008-08-15 15:09:56 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
2013-01-11 16:59:34 +00:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-02-07 16:48:55 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2007-07-02 22:50:25 -07:00
|
|
|
|
|
|
|
|
|
2012-02-07 16:48:55 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
2026-07-06 16:35:17 +00:00
|
|
|
|
2007-07-02 22:50:25 -07:00
|
|
|
|
|
|
|
|
|
2005-09-01 17:40:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
2008-04-23 19:37:58 -07:00
|
|
|
|
2005-09-01 17:40:05 -07:00
|
|
|
|
2020-11-18 07:39:19 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-01 17:40:05 -07:00
|
|
|
|
2021-04-06 18:45:54 +01:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
2020-11-18 07:39:19 +01:00
|
|
|
|
|
|
|
|
|
2005-09-01 17:40:05 -07:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-09-01 17:40:05 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
2011-04-19 06:13:10 +00:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2008-07-03 03:46:16 -07:00
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2008-07-16 12:45:34 -07:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
tree-wide: fix comment/printk typos
"gadget", "through", "command", "maintain", "maintain", "controller", "address",
"between", "initiali[zs]e", "instead", "function", "select", "already",
"equal", "access", "management", "hierarchy", "registration", "interest",
"relative", "memory", "offset", "already",
Signed-off-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2010-11-01 15:38:34 -04:00
|
|
|
|
2025-07-17 10:23:28 -07:00
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2008-07-14 22:18:19 -07:00
|
|
|
|
2025-05-21 13:46:16 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-10-30 18:50:11 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
2009-02-05 21:25:32 -08:00
|
|
|
|
|
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
|
|
|
|
|
2010-02-14 01:01:10 +00:00
|
|
|
|
|
|
|
|
|
2013-08-21 14:32:39 +04:00
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2013-08-21 14:32:39 +04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
2018-01-16 16:31:01 +08:00
|
|
|
|
2017-12-04 17:31:23 +08:00
|
|
|
|
|
|
|
|
|
2018-01-16 16:31:02 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-28 19:12:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-20 12:16:42 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2025-02-07 15:10:54 +09:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2010-05-17 22:47:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-04-10 16:28:56 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-08-06 14:22:44 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2024-08-09 18:00:01 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2008-06-19 15:50:37 -06:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-07-16 22:05:57 -05:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2008-06-19 15:50:37 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-05-08 21:07:08 -05:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2009-10-14 01:19:46 -07:00
|
|
|
|
2015-05-08 21:07:08 -05:00
|
|
|
|
2015-05-08 21:09:13 -05:00
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
|
|
|
|
|
2018-05-11 10:49:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-09-28 14:51:48 -07:00
|
|
|
|
2014-03-24 00:02:32 +05:30
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
tun: Add ability to create tun device with given index
Tun devices cannot be created with ifidex user wants, but it's
required by checkpoint-restore project.
Long time ago such ability was implemented for rtnl_ops-based
interface for creating links (9c7dafbf net: Allow to create links
with given ifindex), but the only API for creating and managing
tuntap devices is ioctl-based and is evolving with adding new ones
(cde8b15f tuntap: add ioctl to attach or detach a file form tuntap
device).
Following that trend, here's how a new ioctl that sets the ifindex
for device, that _will_ be created by TUNSETIFF ioctl looks like.
So those who want a tuntap device with the ifindex N, should open
the tun device, call ioctl(fd, TUNSETIFINDEX, &N), then call TUNSETIFF.
If the index N is busy, then the register_netdev will find this out
and the ioctl would be failed with -EBUSY.
If setifindex is not called, then it will be generated as before.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-21 14:31:38 +04:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2019-07-05 20:14:16 +01:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: tun_chr_open(): set sk_uid from current_fsuid()
Commit a096ccca6e50 initializes the "sk_uid" field in the protocol socket
(struct sock) from the "/dev/net/tun" device node's owner UID. Per
original commit 86741ec25462 ("net: core: Add a UID field to struct
sock.", 2016-11-04), that's wrong: the idea is to cache the UID of the
userspace process that creates the socket. Commit 86741ec25462 mentions
socket() and accept(); with "tun", the action that creates the socket is
open("/dev/net/tun").
Therefore the device node's owner UID is irrelevant. In most cases,
"/dev/net/tun" will be owned by root, so in practice, commit a096ccca6e50
has no observable effect:
- before, "sk_uid" would be zero, due to undefined behavior
(CVE-2023-1076),
- after, "sk_uid" would be zero, due to "/dev/net/tun" being owned by root.
What matters is the (fs)UID of the process performing the open(), so cache
that in "sk_uid".
Cc: Eric Dumazet <edumazet@google.com>
Cc: Lorenzo Colitti <lorenzo@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>
Cc: Pietro Borrello <borrello@diag.uniroma1.it>
Cc: netdev@vger.kernel.org
Cc: stable@vger.kernel.org
Fixes: a096ccca6e50 ("tun: tun_chr_open(): correctly initialize socket uid")
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2173435
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-07-31 18:42:36 +02:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2012-12-13 23:53:30 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2013-06-08 14:17:41 +08:00
|
|
|
|
|
|
|
|
|
2023-03-07 20:45:56 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-20 11:00:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-10-31 19:46:00 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-09-23 22:36:52 +08:00
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
2019-03-20 12:16:53 +03:00
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-09-29 16:08:25 -07:00
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-02-12 00:55:34 -08:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2014-11-07 13:52:07 -05:00
|
|
|
|
2014-06-19 15:36:49 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-11-06 22:52:32 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-01-29 16:43:31 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-09-18 23:01:12 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-02 17:49:53 -04:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-03-11 22:03:50 +01:00
|
|
|
|
|
|
|
|
|
2022-10-31 18:39:53 +01:00
|
|
|
|
2017-03-11 22:03:50 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-02 17:49:53 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-08-30 22:14:52 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2022-08-30 22:14:52 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-11-19 15:17:31 +02:00
|
|
|
|
2022-08-30 22:14:52 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-03-04 17:24:14 +01:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
2021-08-20 15:35:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-08-20 15:35:18 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-02-02 15:53:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-09-13 14:30:00 -04:00
|
|
|
|
2020-03-05 17:05:58 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-07-27 13:53:43 +00:00
|
|
|
|
2024-02-02 15:53:20 +08:00
|
|
|
|
2013-07-19 19:40:10 +02:00
|
|
|
|
2017-01-18 15:02:03 +08:00
|
|
|
|
|
|
|
|
|
2017-03-11 22:03:50 +01:00
|
|
|
|
2018-06-02 17:49:53 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-21 16:37:42 -08:00
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2024-12-17 13:51:21 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
tun/tap: add ptr_ring consume helper with netdev queue wakeup
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".
__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.
Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
stop from tun_net_close() is not mistaken for backpressure, and it only
wakes if the tfile still owns its slot in tun->tfiles[]. A detached
tfile keeps its queue_index, which __tun_detach() may already have
handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index, and that of tfile across the hand-over
of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
the ntfile taking it over, to avoid a possible stall. The queue is only
woken if the ring of the ntfile is empty, as otherwise the consumer
wakes it after consuming the remaining entries. This does not matter
for tun_detach_all(), as it is called during device teardown and no
tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
tun_force_wake_queue() helper from tun_attach(), and reuse it across
the existing wake paths. Unlike __tun_wake_queue() it ignores
IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
cleared. It does honour netif_running(), but it always clears cons_cnt,
so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
before netif_tx_stop_all_queues(), so that its stop is the last write
to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-17 21:26:36 +08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2016-07-06 18:44:20 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-17 21:26:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-04-16 00:40:46 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-02 07:18:10 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
2008-04-16 00:40:46 -07:00
|
|
|
|
2011-03-02 07:18:10 +00:00
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
2008-04-16 00:40:46 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-04-16 00:40:46 -07:00
|
|
|
|
2011-03-02 07:18:10 +00:00
|
|
|
|
2008-04-16 00:40:46 -07:00
|
|
|
|
|
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2017-07-20 02:41:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
2017-07-20 02:41:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-04-16 00:40:46 -07:00
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-14 16:30:00 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-09-13 13:24:59 -04:00
|
|
|
|
2009-01-21 16:02:16 -08:00
|
|
|
|
2016-06-30 14:45:36 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
2012-10-31 19:45:58 +00:00
|
|
|
|
|
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
2012-10-31 19:45:57 +00:00
|
|
|
|
2010-01-14 06:17:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2017-05-17 12:14:41 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2017-05-17 12:14:41 +08:00
|
|
|
|
2018-01-04 11:14:27 +08:00
|
|
|
|
2017-05-17 12:14:41 +08:00
|
|
|
|
vhost-net: wake queue of tun/tap after ptr_ring consume
Add tun_wake_queue() to tun.c and export it for use by vhost-net. The
function validates that the file belongs to a device implemented by
drivers/net/tun.c, in IFF_TUN as well as in IFF_TAP mode, and that the
tfile exists, dereferences the tun_struct under RCU, and delegates to
__tun_wake_queue().
vhost_net_buf_produce() now calls tun_wake_queue() after a successful
batched consume of the ring to allow the netdev subqueue to be woken up.
The point is to allow the queue to be stopped when it gets full, which is
required for traffic shaping, implemented by the following
"stop tail-drop when IFF_BACKPRESSURE is set".
As __tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, a
tun/tap device that does not opt in only pays for the added check.
macvtap and ipvtap rings, which get_tap_ptr_ring() accepts too, are
unaffected: their producer is the tap_handle_frame() rx_handler and not
ndo_start_xmit, so stopping a netdev TX queue would not hold it back.
drivers/net/tap.c has no netdev_ops of its own either. No
tap_wake_queue() is needed.
cons_cnt and the wake decision are best-effort and are not reverted by
ptr_ring_unconsume(), so vhost_net_buf_unproduce() can leave the subqueue
woken over a full ring. The producer re-stops it on the next packet, and
that path only runs from vhost_net_stop_vq() and vhost_net_set_backend(),
when the consumer is going away, so a stopped queue is the correct end
state rather than a stall.
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://patch.msgid.link/20260803183641.96882-4-simon.schippers@tu-dortmund.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03 20:36:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
driver core: add devname module aliases to allow module on-demand auto-loading
This adds:
alias: devname:<name>
to some common kernel modules, which will allow the on-demand loading
of the kernel module when the device node is accessed.
Ideally all these modules would be compiled-in, but distros seems too
much in love with their modularization that we need to cover the common
cases with this new facility. It will allow us to remove a bunch of pretty
useless init scripts and modprobes from init scripts.
The static device node aliases will be carried in the module itself. The
program depmod will extract this information to a file in the module directory:
$ cat /lib/modules/2.6.34-00650-g537b60d-dirty/modules.devname
# Device nodes to trigger on-demand module loading.
microcode cpu/microcode c10:184
fuse fuse c10:229
ppp_generic ppp c108:0
tun net/tun c10:200
dm_mod mapper/control c10:235
Udev will pick up the depmod created file on startup and create all the
static device nodes which the kernel modules specify, so that these modules
get automatically loaded when the device node is accessed:
$ /sbin/udevd --debug
...
static_dev_create_from_modules: mknod '/dev/cpu/microcode' c10:184
static_dev_create_from_modules: mknod '/dev/fuse' c10:229
static_dev_create_from_modules: mknod '/dev/ppp' c108:0
static_dev_create_from_modules: mknod '/dev/net/tun' c10:200
static_dev_create_from_modules: mknod '/dev/mapper/control' c10:235
udev_rules_apply_static_dev_perms: chmod '/dev/net/tun' 0666
udev_rules_apply_static_dev_perms: chmod '/dev/fuse' 0666
A few device nodes are switched to statically allocated numbers, to allow
the static nodes to work. This might also useful for systems which still run
a plain static /dev, which is completely unsafe to use with any dynamic minor
numbers.
Note:
The devname aliases must be limited to the *common* and *single*instance*
device nodes, like the misc devices, and never be used for conceptually limited
systems like the loop devices, which should rather get fixed properly and get a
control node for losetup to talk to, instead of creating a random number of
device nodes in advance, regardless if they are ever used.
This facility is to hide the mess distros are creating with too modualized
kernels, and just to hide that these modules are not compiled-in, and not to
paper-over broken concepts. Thanks! :)
Cc: Greg Kroah-Hartman <gregkh@suse.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Chris Mason <chris.mason@oracle.com>
Cc: Alasdair G Kergon <agk@redhat.com>
Cc: Tigran Aivazian <tigran@aivazian.fsnet.co.uk>
Cc: Ian Kent <raven@themaw.net>
Signed-Off-By: Kay Sievers <kay.sievers@vrfy.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
2010-05-20 18:07:20 +02:00
|
|
|
|
2025-07-17 10:23:28 -07:00
|
|
|
|
