2019-05-27 08:55:01 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-01-11 12:17:47 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 17:04:11 +09:00
|
|
|
|
2017-02-02 19:15:33 +01:00
|
|
|
|
2020-06-25 18:34:43 -04:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2011-07-15 11:47:34 -04:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2013-02-22 16:34:16 -08:00
|
|
|
|
2015-03-09 14:31:20 -07:00
|
|
|
|
2016-06-07 19:27:51 +01:00
|
|
|
|
2018-08-08 20:07:35 -07:00
|
|
|
|
2025-03-07 10:30:06 -08:00
|
|
|
|
2023-08-02 18:02:29 -07:00
|
|
|
|
2024-03-06 16:00:30 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2022-04-06 14:37:54 -07:00
|
|
|
|
2007-10-23 21:14:45 -07:00
|
|
|
|
|
|
|
|
|
2007-09-26 22:02:53 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2024-09-04 15:34:30 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-08 07:19:54 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2025-01-14 19:53:16 -08:00
|
|
|
|
2007-02-09 23:24:36 +09:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2024-02-13 06:32:37 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2024-02-13 06:32:37 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
|
|
|
|
|
2024-02-13 06:32:37 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2024-02-13 06:32:37 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-11-16 03:03:04 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2020-04-09 14:41:26 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-11-16 03:03:04 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2012-04-12 09:28:13 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-02-26 06:49:24 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-14 19:53:16 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2014-02-25 18:17:50 +02:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-04-02 17:07:00 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2015-04-02 17:07:00 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
net: add name_assign_type netdev attribute
Based on a patch by David Herrmann.
The name_assign_type attribute gives hints where the interface name of a
given net-device comes from. These values are currently defined:
NET_NAME_ENUM:
The ifname is provided by the kernel with an enumerated
suffix, typically based on order of discovery. Names may
be reused and unpredictable.
NET_NAME_PREDICTABLE:
The ifname has been assigned by the kernel in a predictable way
that is guaranteed to avoid reuse and always be the same for a
given device. Examples include statically created devices like
the loopback device and names deduced from hardware properties
(including being given explicitly by the firmware). Names
depending on the order of discovery, or in any other way on the
existence of other devices, must not be marked as PREDICTABLE.
NET_NAME_USER:
The ifname was provided by user-space during net-device setup.
NET_NAME_RENAMED:
The net-device has been renamed from userspace. Once this type is set,
it cannot change again.
NET_NAME_UNKNOWN:
This is an internal placeholder to indicate that we yet haven't yet
categorized the name. It will not be exposed to userspace, rather
-EINVAL is returned.
The aim of these patches is to improve user-space renaming of interfaces. As
a general rule, userspace must rename interfaces to guarantee that names stay
the same every time a given piece of hardware appears (at boot, or when
attaching it). However, there are several situations where userspace should
not perform the renaming, and that depends on both the policy of the local
admin, but crucially also on the nature of the current interface name.
If an interface was created in repsonse to a userspace request, and userspace
already provided a name, we most probably want to leave that name alone. The
main instance of this is wifi-P2P devices created over nl80211, which currently
have a long-standing bug where they are getting renamed by udev. We label such
names NET_NAME_USER.
If an interface, unbeknown to us, has already been renamed from userspace, we
most probably want to leave also that alone. This will typically happen when
third-party plugins (for instance to udev, but the interface is generic so could
be from anywhere) renames the interface without informing udev about it. A
typical situation is when you switch root from an installer or an initrd to the
real system and the new instance of udev does not know what happened before
the switch. These types of problems have caused repeated issues in the past. To
solve this, once an interface has been renamed, its name is labelled
NET_NAME_RENAMED.
In many cases, the kernel is actually able to name interfaces in such a
way that there is no need for userspace to rename them. This is the case when
the enumeration order of devices, or in fact any other (non-parent) device on
the system, can not influence the name of the interface. Examples include
statically created devices, or any naming schemes based on hardware properties
of the interface. In this case the admin may prefer to use the kernel-provided
names, and to make that possible we label such names NET_NAME_PREDICTABLE.
We want the kernel to have tho possibilty of performing predictable interface
naming itself (and exposing to userspace that it has), as the information
necessary for a proper naming scheme for a certain class of devices may not
be exposed to userspace.
The case where renaming is almost certainly desired, is when the kernel has
given the interface a name using global device enumeration based on order of
discovery (ethX, wlanY, etc). These naming schemes are labelled NET_NAME_ENUM.
Lastly, a fallback is left as NET_NAME_UNKNOWN, to indicate that a driver has
not yet been ported. This is mostly useful as a transitionary measure, allowing
us to label the various naming schemes bit by bit.
v8: minor documentation fixes
v9: move comment to the right commit
Signed-off-by: Tom Gundersen <teg@jklm.no>
Reviewed-by: David Herrmann <dh.herrmann@gmail.com>
Reviewed-by: Kay Sievers <kay@vrfy.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-07-14 16:37:22 +02:00
|
|
|
|
2024-02-13 06:32:33 +00:00
|
|
|
|
net: add name_assign_type netdev attribute
Based on a patch by David Herrmann.
The name_assign_type attribute gives hints where the interface name of a
given net-device comes from. These values are currently defined:
NET_NAME_ENUM:
The ifname is provided by the kernel with an enumerated
suffix, typically based on order of discovery. Names may
be reused and unpredictable.
NET_NAME_PREDICTABLE:
The ifname has been assigned by the kernel in a predictable way
that is guaranteed to avoid reuse and always be the same for a
given device. Examples include statically created devices like
the loopback device and names deduced from hardware properties
(including being given explicitly by the firmware). Names
depending on the order of discovery, or in any other way on the
existence of other devices, must not be marked as PREDICTABLE.
NET_NAME_USER:
The ifname was provided by user-space during net-device setup.
NET_NAME_RENAMED:
The net-device has been renamed from userspace. Once this type is set,
it cannot change again.
NET_NAME_UNKNOWN:
This is an internal placeholder to indicate that we yet haven't yet
categorized the name. It will not be exposed to userspace, rather
-EINVAL is returned.
The aim of these patches is to improve user-space renaming of interfaces. As
a general rule, userspace must rename interfaces to guarantee that names stay
the same every time a given piece of hardware appears (at boot, or when
attaching it). However, there are several situations where userspace should
not perform the renaming, and that depends on both the policy of the local
admin, but crucially also on the nature of the current interface name.
If an interface was created in repsonse to a userspace request, and userspace
already provided a name, we most probably want to leave that name alone. The
main instance of this is wifi-P2P devices created over nl80211, which currently
have a long-standing bug where they are getting renamed by udev. We label such
names NET_NAME_USER.
If an interface, unbeknown to us, has already been renamed from userspace, we
most probably want to leave also that alone. This will typically happen when
third-party plugins (for instance to udev, but the interface is generic so could
be from anywhere) renames the interface without informing udev about it. A
typical situation is when you switch root from an installer or an initrd to the
real system and the new instance of udev does not know what happened before
the switch. These types of problems have caused repeated issues in the past. To
solve this, once an interface has been renamed, its name is labelled
NET_NAME_RENAMED.
In many cases, the kernel is actually able to name interfaces in such a
way that there is no need for userspace to rename them. This is the case when
the enumeration order of devices, or in fact any other (non-parent) device on
the system, can not influence the name of the interface. Examples include
statically created devices, or any naming schemes based on hardware properties
of the interface. In this case the admin may prefer to use the kernel-provided
names, and to make that possible we label such names NET_NAME_PREDICTABLE.
We want the kernel to have tho possibilty of performing predictable interface
naming itself (and exposing to userspace that it has), as the information
necessary for a proper naming scheme for a certain class of devices may not
be exposed to userspace.
The case where renaming is almost certainly desired, is when the kernel has
given the interface a name using global device enumeration based on order of
discovery (ethX, wlanY, etc). These naming schemes are labelled NET_NAME_ENUM.
Lastly, a fallback is left as NET_NAME_UNKNOWN, to indicate that a driver has
not yet been ported. This is mostly useful as a transitionary measure, allowing
us to label the various naming schemes bit by bit.
v8: minor documentation fixes
v9: move comment to the right commit
Signed-off-by: Tom Gundersen <teg@jklm.no>
Reviewed-by: David Herrmann <dh.herrmann@gmail.com>
Reviewed-by: Kay Sievers <kay@vrfy.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-07-14 16:37:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
net: add name_assign_type netdev attribute
Based on a patch by David Herrmann.
The name_assign_type attribute gives hints where the interface name of a
given net-device comes from. These values are currently defined:
NET_NAME_ENUM:
The ifname is provided by the kernel with an enumerated
suffix, typically based on order of discovery. Names may
be reused and unpredictable.
NET_NAME_PREDICTABLE:
The ifname has been assigned by the kernel in a predictable way
that is guaranteed to avoid reuse and always be the same for a
given device. Examples include statically created devices like
the loopback device and names deduced from hardware properties
(including being given explicitly by the firmware). Names
depending on the order of discovery, or in any other way on the
existence of other devices, must not be marked as PREDICTABLE.
NET_NAME_USER:
The ifname was provided by user-space during net-device setup.
NET_NAME_RENAMED:
The net-device has been renamed from userspace. Once this type is set,
it cannot change again.
NET_NAME_UNKNOWN:
This is an internal placeholder to indicate that we yet haven't yet
categorized the name. It will not be exposed to userspace, rather
-EINVAL is returned.
The aim of these patches is to improve user-space renaming of interfaces. As
a general rule, userspace must rename interfaces to guarantee that names stay
the same every time a given piece of hardware appears (at boot, or when
attaching it). However, there are several situations where userspace should
not perform the renaming, and that depends on both the policy of the local
admin, but crucially also on the nature of the current interface name.
If an interface was created in repsonse to a userspace request, and userspace
already provided a name, we most probably want to leave that name alone. The
main instance of this is wifi-P2P devices created over nl80211, which currently
have a long-standing bug where they are getting renamed by udev. We label such
names NET_NAME_USER.
If an interface, unbeknown to us, has already been renamed from userspace, we
most probably want to leave also that alone. This will typically happen when
third-party plugins (for instance to udev, but the interface is generic so could
be from anywhere) renames the interface without informing udev about it. A
typical situation is when you switch root from an installer or an initrd to the
real system and the new instance of udev does not know what happened before
the switch. These types of problems have caused repeated issues in the past. To
solve this, once an interface has been renamed, its name is labelled
NET_NAME_RENAMED.
In many cases, the kernel is actually able to name interfaces in such a
way that there is no need for userspace to rename them. This is the case when
the enumeration order of devices, or in fact any other (non-parent) device on
the system, can not influence the name of the interface. Examples include
statically created devices, or any naming schemes based on hardware properties
of the interface. In this case the admin may prefer to use the kernel-provided
names, and to make that possible we label such names NET_NAME_PREDICTABLE.
We want the kernel to have tho possibilty of performing predictable interface
naming itself (and exposing to userspace that it has), as the information
necessary for a proper naming scheme for a certain class of devices may not
be exposed to userspace.
The case where renaming is almost certainly desired, is when the kernel has
given the interface a name using global device enumeration based on order of
discovery (ethX, wlanY, etc). These naming schemes are labelled NET_NAME_ENUM.
Lastly, a fallback is left as NET_NAME_UNKNOWN, to indicate that a driver has
not yet been ported. This is mostly useful as a transitionary measure, allowing
us to label the various naming schemes bit by bit.
v8: minor documentation fixes
v9: move comment to the right commit
Signed-off-by: Tom Gundersen <teg@jklm.no>
Reviewed-by: David Herrmann <dh.herrmann@gmail.com>
Reviewed-by: Kay Sievers <kay@vrfy.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-07-14 16:37:22 +02:00
|
|
|
|
|
|
|
|
|
2024-02-13 06:32:33 +00:00
|
|
|
|
net: add name_assign_type netdev attribute
Based on a patch by David Herrmann.
The name_assign_type attribute gives hints where the interface name of a
given net-device comes from. These values are currently defined:
NET_NAME_ENUM:
The ifname is provided by the kernel with an enumerated
suffix, typically based on order of discovery. Names may
be reused and unpredictable.
NET_NAME_PREDICTABLE:
The ifname has been assigned by the kernel in a predictable way
that is guaranteed to avoid reuse and always be the same for a
given device. Examples include statically created devices like
the loopback device and names deduced from hardware properties
(including being given explicitly by the firmware). Names
depending on the order of discovery, or in any other way on the
existence of other devices, must not be marked as PREDICTABLE.
NET_NAME_USER:
The ifname was provided by user-space during net-device setup.
NET_NAME_RENAMED:
The net-device has been renamed from userspace. Once this type is set,
it cannot change again.
NET_NAME_UNKNOWN:
This is an internal placeholder to indicate that we yet haven't yet
categorized the name. It will not be exposed to userspace, rather
-EINVAL is returned.
The aim of these patches is to improve user-space renaming of interfaces. As
a general rule, userspace must rename interfaces to guarantee that names stay
the same every time a given piece of hardware appears (at boot, or when
attaching it). However, there are several situations where userspace should
not perform the renaming, and that depends on both the policy of the local
admin, but crucially also on the nature of the current interface name.
If an interface was created in repsonse to a userspace request, and userspace
already provided a name, we most probably want to leave that name alone. The
main instance of this is wifi-P2P devices created over nl80211, which currently
have a long-standing bug where they are getting renamed by udev. We label such
names NET_NAME_USER.
If an interface, unbeknown to us, has already been renamed from userspace, we
most probably want to leave also that alone. This will typically happen when
third-party plugins (for instance to udev, but the interface is generic so could
be from anywhere) renames the interface without informing udev about it. A
typical situation is when you switch root from an installer or an initrd to the
real system and the new instance of udev does not know what happened before
the switch. These types of problems have caused repeated issues in the past. To
solve this, once an interface has been renamed, its name is labelled
NET_NAME_RENAMED.
In many cases, the kernel is actually able to name interfaces in such a
way that there is no need for userspace to rename them. This is the case when
the enumeration order of devices, or in fact any other (non-parent) device on
the system, can not influence the name of the interface. Examples include
statically created devices, or any naming schemes based on hardware properties
of the interface. In this case the admin may prefer to use the kernel-provided
names, and to make that possible we label such names NET_NAME_PREDICTABLE.
We want the kernel to have tho possibilty of performing predictable interface
naming itself (and exposing to userspace that it has), as the information
necessary for a proper naming scheme for a certain class of devices may not
be exposed to userspace.
The case where renaming is almost certainly desired, is when the kernel has
given the interface a name using global device enumeration based on order of
discovery (ethX, wlanY, etc). These naming schemes are labelled NET_NAME_ENUM.
Lastly, a fallback is left as NET_NAME_UNKNOWN, to indicate that a driver has
not yet been ported. This is mostly useful as a transitionary measure, allowing
us to label the various naming schemes bit by bit.
v8: minor documentation fixes
v9: move comment to the right commit
Signed-off-by: Tom Gundersen <teg@jklm.no>
Reviewed-by: David Herrmann <dh.herrmann@gmail.com>
Reviewed-by: Kay Sievers <kay@vrfy.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-07-14 16:37:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-17 10:23:28 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2025-03-12 12:05:12 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
|
|
|
|
|
2025-03-12 12:05:12 -07:00
|
|
|
|
2024-02-13 06:32:38 +00:00
|
|
|
|
2025-03-12 12:05:12 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2024-02-13 06:32:37 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2024-02-13 06:32:37 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2024-02-13 06:32:37 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2012-12-27 23:49:38 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2012-12-27 23:49:38 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2012-12-27 23:49:38 +00:00
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
2012-12-27 23:49:38 +00:00
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-27 23:49:38 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2025-02-21 05:12:23 +00:00
|
|
|
|
2023-12-06 17:21:23 +01:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2025-02-21 05:12:23 +00:00
|
|
|
|
2023-12-04 21:47:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-12-06 17:21:23 +01:00
|
|
|
|
2023-12-04 21:47:07 +01:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
2023-12-06 17:21:23 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
2025-02-21 05:12:23 +00:00
|
|
|
|
2024-08-23 16:26:58 +10:00
|
|
|
|
2016-02-24 10:58:10 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
2025-02-21 05:12:23 +00:00
|
|
|
|
2011-04-27 18:32:38 +00:00
|
|
|
|
2016-02-24 10:58:10 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-09-05 04:11:28 +00:00
|
|
|
|
2016-02-24 10:58:10 -08:00
|
|
|
|
|
|
|
|
|
2012-09-05 04:11:28 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2012-09-05 04:11:28 +00:00
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2009-10-02 09:26:12 +00:00
|
|
|
|
2020-04-20 00:11:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2020-04-20 00:11:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
2009-08-05 10:42:58 -07:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-20 00:11:51 +02:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-02-13 06:32:39 +00:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-04-05 22:19:47 -07:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2006-03-20 17:09:11 -08:00
|
|
|
|
2014-03-29 09:48:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-03-29 09:48:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-18 09:59:13 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2018-01-18 09:59:13 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2018-01-18 09:59:13 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-12-06 17:05:42 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-11-16 03:03:04 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-01-25 18:26:22 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-05-17 13:30:44 +03:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
net: gro: add a per device gro flush timer
Tuning coalescing parameters on NIC can be really hard.
Servers can handle both bulk and RPC like traffic, with conflicting
goals : bulk flows want as big GRO packets as possible, RPC want minimal
latencies.
To reach big GRO packets on 10Gbe NIC, one can use :
ethtool -C eth0 rx-usecs 4 rx-frames 44
But this penalizes rpc sessions, with an increase of latencies, up to
50% in some cases, as NICs generally do not force an interrupt when
a packet with TCP Push flag is received.
Some NICs do not have an absolute timer, only a timer rearmed for every
incoming packet.
This patch uses a different strategy : Let GRO stack decides what do do,
based on traffic pattern.
Packets with Push flag wont be delayed.
Packets without Push flag might be held in GRO engine, if we keep
receiving data.
This new mechanism is off by default, and shall be enabled by setting
/sys/class/net/ethX/gro_flush_timeout to a value in nanosecond.
To fully enable this mechanism, drivers should use napi_complete_done()
instead of napi_complete().
Tested:
Ran 200 netperf TCP_STREAM from A to B (10Gbe mlx4 link, 8 RX queues)
Without this feature, we send back about 305,000 ACK per second.
GRO aggregation ratio is low (811/305 = 2.65 segments per GRO packet)
Setting a timer of 2000 nsec is enough to increase GRO packet sizes
and reduce number of ACK packets. (811/19.2 = 42)
Receiver performs less calls to upper stacks, less wakes up.
This also reduces cpu usage on the sender, as it receives less ACK
packets.
Note that reducing number of wakes up increases cpu efficiency, but can
decrease QPS, as applications wont have the chance to warmup cpu caches
doing a partial read of RPC requests/answers if they fit in one skb.
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811269.80 305732.30 1199462.57 19705.72 0.00
0.00 0.50
B:~# echo 2000 >/sys/class/net/eth0/gro_flush_timeout
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811577.30 19230.80 1199916.51 1239.80 0.00
0.00 0.50
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-06 21:09:44 -08:00
|
|
|
|
|
|
|
|
|
2024-10-11 18:44:58 +00:00
|
|
|
|
net: gro: add a per device gro flush timer
Tuning coalescing parameters on NIC can be really hard.
Servers can handle both bulk and RPC like traffic, with conflicting
goals : bulk flows want as big GRO packets as possible, RPC want minimal
latencies.
To reach big GRO packets on 10Gbe NIC, one can use :
ethtool -C eth0 rx-usecs 4 rx-frames 44
But this penalizes rpc sessions, with an increase of latencies, up to
50% in some cases, as NICs generally do not force an interrupt when
a packet with TCP Push flag is received.
Some NICs do not have an absolute timer, only a timer rearmed for every
incoming packet.
This patch uses a different strategy : Let GRO stack decides what do do,
based on traffic pattern.
Packets with Push flag wont be delayed.
Packets without Push flag might be held in GRO engine, if we keep
receiving data.
This new mechanism is off by default, and shall be enabled by setting
/sys/class/net/ethX/gro_flush_timeout to a value in nanosecond.
To fully enable this mechanism, drivers should use napi_complete_done()
instead of napi_complete().
Tested:
Ran 200 netperf TCP_STREAM from A to B (10Gbe mlx4 link, 8 RX queues)
Without this feature, we send back about 305,000 ACK per second.
GRO aggregation ratio is low (811/305 = 2.65 segments per GRO packet)
Setting a timer of 2000 nsec is enough to increase GRO packet sizes
and reduce number of ACK packets. (811/19.2 = 42)
Receiver performs less calls to upper stacks, less wakes up.
This also reduces cpu usage on the sender, as it receives less ACK
packets.
Note that reducing number of wakes up increases cpu efficiency, but can
decrease QPS, as applications wont have the chance to warmup cpu caches
doing a partial read of RPC requests/answers if they fit in one skb.
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811269.80 305732.30 1199462.57 19705.72 0.00
0.00 0.50
B:~# echo 2000 >/sys/class/net/eth0/gro_flush_timeout
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811577.30 19230.80 1199916.51 1239.80 0.00
0.00 0.50
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-06 21:09:44 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
|
|
|
|
|
net: gro: add a per device gro flush timer
Tuning coalescing parameters on NIC can be really hard.
Servers can handle both bulk and RPC like traffic, with conflicting
goals : bulk flows want as big GRO packets as possible, RPC want minimal
latencies.
To reach big GRO packets on 10Gbe NIC, one can use :
ethtool -C eth0 rx-usecs 4 rx-frames 44
But this penalizes rpc sessions, with an increase of latencies, up to
50% in some cases, as NICs generally do not force an interrupt when
a packet with TCP Push flag is received.
Some NICs do not have an absolute timer, only a timer rearmed for every
incoming packet.
This patch uses a different strategy : Let GRO stack decides what do do,
based on traffic pattern.
Packets with Push flag wont be delayed.
Packets without Push flag might be held in GRO engine, if we keep
receiving data.
This new mechanism is off by default, and shall be enabled by setting
/sys/class/net/ethX/gro_flush_timeout to a value in nanosecond.
To fully enable this mechanism, drivers should use napi_complete_done()
instead of napi_complete().
Tested:
Ran 200 netperf TCP_STREAM from A to B (10Gbe mlx4 link, 8 RX queues)
Without this feature, we send back about 305,000 ACK per second.
GRO aggregation ratio is low (811/305 = 2.65 segments per GRO packet)
Setting a timer of 2000 nsec is enough to increase GRO packet sizes
and reduce number of ACK packets. (811/19.2 = 42)
Receiver performs less calls to upper stacks, less wakes up.
This also reduces cpu usage on the sender, as it receives less ACK
packets.
Note that reducing number of wakes up increases cpu efficiency, but can
decrease QPS, as applications wont have the chance to warmup cpu caches
doing a partial read of RPC requests/answers if they fit in one skb.
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811269.80 305732.30 1199462.57 19705.72 0.00
0.00 0.50
B:~# echo 2000 >/sys/class/net/eth0/gro_flush_timeout
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811577.30 19230.80 1199916.51 1239.80 0.00
0.00 0.50
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-06 21:09:44 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-14 19:53:18 -08:00
|
|
|
|
net: gro: add a per device gro flush timer
Tuning coalescing parameters on NIC can be really hard.
Servers can handle both bulk and RPC like traffic, with conflicting
goals : bulk flows want as big GRO packets as possible, RPC want minimal
latencies.
To reach big GRO packets on 10Gbe NIC, one can use :
ethtool -C eth0 rx-usecs 4 rx-frames 44
But this penalizes rpc sessions, with an increase of latencies, up to
50% in some cases, as NICs generally do not force an interrupt when
a packet with TCP Push flag is received.
Some NICs do not have an absolute timer, only a timer rearmed for every
incoming packet.
This patch uses a different strategy : Let GRO stack decides what do do,
based on traffic pattern.
Packets with Push flag wont be delayed.
Packets without Push flag might be held in GRO engine, if we keep
receiving data.
This new mechanism is off by default, and shall be enabled by setting
/sys/class/net/ethX/gro_flush_timeout to a value in nanosecond.
To fully enable this mechanism, drivers should use napi_complete_done()
instead of napi_complete().
Tested:
Ran 200 netperf TCP_STREAM from A to B (10Gbe mlx4 link, 8 RX queues)
Without this feature, we send back about 305,000 ACK per second.
GRO aggregation ratio is low (811/305 = 2.65 segments per GRO packet)
Setting a timer of 2000 nsec is enough to increase GRO packet sizes
and reduce number of ACK packets. (811/19.2 = 42)
Receiver performs less calls to upper stacks, less wakes up.
This also reduces cpu usage on the sender, as it receives less ACK
packets.
Note that reducing number of wakes up increases cpu efficiency, but can
decrease QPS, as applications wont have the chance to warmup cpu caches
doing a partial read of RPC requests/answers if they fit in one skb.
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811269.80 305732.30 1199462.57 19705.72 0.00
0.00 0.50
B:~# echo 2000 >/sys/class/net/eth0/gro_flush_timeout
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811577.30 19230.80 1199916.51 1239.80 0.00
0.00 0.50
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-06 21:09:44 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: napi: add hard irqs deferral feature
Back in commit 3b47d30396ba ("net: gro: add a per device gro flush timer")
we added the ability to arm one high resolution timer, that we used
to keep not-complete packets in GRO engine a bit longer, hoping that further
frames might be added to them.
Since then, we added the napi_complete_done() interface, and commit
364b6055738b ("net: busy-poll: return busypolling status to drivers")
allowed drivers to avoid re-arming NIC interrupts if we made a promise
that their NAPI poll() handler would be called in the near future.
This infrastructure can be leveraged, thanks to a new device parameter,
which allows to arm the napi hrtimer, instead of re-arming the device
hard IRQ.
We have noticed that on some servers with 32 RX queues or more, the chit-chat
between the NIC and the host caused by IRQ delivery and re-arming could hurt
throughput by ~20% on 100Gbit NIC.
In contrast, hrtimers are using local (percpu) resources and might have lower
cost.
The new tunable, named napi_defer_hard_irqs, is placed in the same hierarchy
than gro_flush_timeout (/sys/class/net/ethX/)
By default, both gro_flush_timeout and napi_defer_hard_irqs are zero.
This patch does not change the prior behavior of gro_flush_timeout
if used alone : NIC hard irqs should be rearmed as before.
One concrete usage can be :
echo 20000 >/sys/class/net/eth1/gro_flush_timeout
echo 10 >/sys/class/net/eth1/napi_defer_hard_irqs
If at least one packet is retired, then we will reset napi counter
to 10 (napi_defer_hard_irqs), ensuring at least 10 periodic scans
of the queue.
On busy queues, this should avoid NIC hard IRQ, while before this patch IRQ
avoidance was only possible if napi->poll() was exhausting its budget
and not call napi_complete_done().
This feature also can be used to work around some non-optimal NIC irq
coalescing strategies.
Having the ability to insert XX usec delays between each napi->poll()
can increase cache efficiency, since we increase batch sizes.
It also keeps serving cpus not idle too long, reducing tail latencies.
Co-developed-by: Luigi Rizzo <lrizzo@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-22 09:13:27 -07:00
|
|
|
|
|
|
|
|
|
2024-09-04 15:34:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-10-11 18:44:56 +00:00
|
|
|
|
net: napi: add hard irqs deferral feature
Back in commit 3b47d30396ba ("net: gro: add a per device gro flush timer")
we added the ability to arm one high resolution timer, that we used
to keep not-complete packets in GRO engine a bit longer, hoping that further
frames might be added to them.
Since then, we added the napi_complete_done() interface, and commit
364b6055738b ("net: busy-poll: return busypolling status to drivers")
allowed drivers to avoid re-arming NIC interrupts if we made a promise
that their NAPI poll() handler would be called in the near future.
This infrastructure can be leveraged, thanks to a new device parameter,
which allows to arm the napi hrtimer, instead of re-arming the device
hard IRQ.
We have noticed that on some servers with 32 RX queues or more, the chit-chat
between the NIC and the host caused by IRQ delivery and re-arming could hurt
throughput by ~20% on 100Gbit NIC.
In contrast, hrtimers are using local (percpu) resources and might have lower
cost.
The new tunable, named napi_defer_hard_irqs, is placed in the same hierarchy
than gro_flush_timeout (/sys/class/net/ethX/)
By default, both gro_flush_timeout and napi_defer_hard_irqs are zero.
This patch does not change the prior behavior of gro_flush_timeout
if used alone : NIC hard irqs should be rearmed as before.
One concrete usage can be :
echo 20000 >/sys/class/net/eth1/gro_flush_timeout
echo 10 >/sys/class/net/eth1/napi_defer_hard_irqs
If at least one packet is retired, then we will reset napi counter
to 10 (napi_defer_hard_irqs), ensuring at least 10 periodic scans
of the queue.
On busy queues, this should avoid NIC hard IRQ, while before this patch IRQ
avoidance was only possible if napi->poll() was exhausting its budget
and not call napi_complete_done().
This feature also can be used to work around some non-optimal NIC irq
coalescing strategies.
Having the ability to insert XX usec delays between each napi->poll()
can increase cache efficiency, since we increase batch sizes.
It also keeps serving cpus not idle too long, reducing tail latencies.
Co-developed-by: Luigi Rizzo <lrizzo@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-22 09:13:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-14 19:53:18 -08:00
|
|
|
|
|
|
|
|
|
net: napi: add hard irqs deferral feature
Back in commit 3b47d30396ba ("net: gro: add a per device gro flush timer")
we added the ability to arm one high resolution timer, that we used
to keep not-complete packets in GRO engine a bit longer, hoping that further
frames might be added to them.
Since then, we added the napi_complete_done() interface, and commit
364b6055738b ("net: busy-poll: return busypolling status to drivers")
allowed drivers to avoid re-arming NIC interrupts if we made a promise
that their NAPI poll() handler would be called in the near future.
This infrastructure can be leveraged, thanks to a new device parameter,
which allows to arm the napi hrtimer, instead of re-arming the device
hard IRQ.
We have noticed that on some servers with 32 RX queues or more, the chit-chat
between the NIC and the host caused by IRQ delivery and re-arming could hurt
throughput by ~20% on 100Gbit NIC.
In contrast, hrtimers are using local (percpu) resources and might have lower
cost.
The new tunable, named napi_defer_hard_irqs, is placed in the same hierarchy
than gro_flush_timeout (/sys/class/net/ethX/)
By default, both gro_flush_timeout and napi_defer_hard_irqs are zero.
This patch does not change the prior behavior of gro_flush_timeout
if used alone : NIC hard irqs should be rearmed as before.
One concrete usage can be :
echo 20000 >/sys/class/net/eth1/gro_flush_timeout
echo 10 >/sys/class/net/eth1/napi_defer_hard_irqs
If at least one packet is retired, then we will reset napi counter
to 10 (napi_defer_hard_irqs), ensuring at least 10 periodic scans
of the queue.
On busy queues, this should avoid NIC hard IRQ, while before this patch IRQ
avoidance was only possible if napi->poll() was exhausting its budget
and not call napi_complete_done().
This feature also can be used to work around some non-optimal NIC irq
coalescing strategies.
Having the ability to insert XX usec delays between each napi->poll()
can increase cache efficiency, since we increase batch sizes.
It also keeps serving cpus not idle too long, reducing tail latencies.
Co-developed-by: Luigi Rizzo <lrizzo@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-22 09:13:27 -07:00
|
|
|
|
2024-09-04 15:34:30 +00:00
|
|
|
|
net: napi: add hard irqs deferral feature
Back in commit 3b47d30396ba ("net: gro: add a per device gro flush timer")
we added the ability to arm one high resolution timer, that we used
to keep not-complete packets in GRO engine a bit longer, hoping that further
frames might be added to them.
Since then, we added the napi_complete_done() interface, and commit
364b6055738b ("net: busy-poll: return busypolling status to drivers")
allowed drivers to avoid re-arming NIC interrupts if we made a promise
that their NAPI poll() handler would be called in the near future.
This infrastructure can be leveraged, thanks to a new device parameter,
which allows to arm the napi hrtimer, instead of re-arming the device
hard IRQ.
We have noticed that on some servers with 32 RX queues or more, the chit-chat
between the NIC and the host caused by IRQ delivery and re-arming could hurt
throughput by ~20% on 100Gbit NIC.
In contrast, hrtimers are using local (percpu) resources and might have lower
cost.
The new tunable, named napi_defer_hard_irqs, is placed in the same hierarchy
than gro_flush_timeout (/sys/class/net/ethX/)
By default, both gro_flush_timeout and napi_defer_hard_irqs are zero.
This patch does not change the prior behavior of gro_flush_timeout
if used alone : NIC hard irqs should be rearmed as before.
One concrete usage can be :
echo 20000 >/sys/class/net/eth1/gro_flush_timeout
echo 10 >/sys/class/net/eth1/napi_defer_hard_irqs
If at least one packet is retired, then we will reset napi counter
to 10 (napi_defer_hard_irqs), ensuring at least 10 periodic scans
of the queue.
On busy queues, this should avoid NIC hard IRQ, while before this patch IRQ
avoidance was only possible if napi->poll() was exhausting its budget
and not call napi_complete_done().
This feature also can be used to work around some non-optimal NIC irq
coalescing strategies.
Having the ability to insert XX usec delays between each napi->poll()
can increase cache efficiency, since we increase batch sizes.
It also keeps serving cpus not idle too long, reducing tail latencies.
Co-developed-by: Luigi Rizzo <lrizzo@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-22 09:13:27 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-11-16 03:03:04 +00:00
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
2025-02-26 18:46:43 +01:00
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
2012-11-16 03:03:04 +00:00
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-11-13 23:21:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-10-02 23:50:05 +02:00
|
|
|
|
2025-02-26 18:46:43 +01:00
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
2017-10-02 23:50:05 +02:00
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2008-09-22 21:28:11 -07:00
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2011-01-24 03:37:29 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2011-01-24 03:37:29 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2011-01-24 03:37:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
2011-01-24 03:37:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2018-03-23 15:54:38 -07:00
|
|
|
|
2011-01-24 03:37:29 +00:00
|
|
|
|
2015-07-14 13:43:19 -07:00
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2015-07-14 13:43:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
Pull networking changes from David Miller:
"Noteworthy changes this time around:
1) Multicast rejoin support for team driver, from Jiri Pirko.
2) Centralize and simplify TCP RTT measurement handling in order to
reduce the impact of bad RTO seeding from SYN/ACKs. Also, when
both timestamps and local RTT measurements are available prefer
the later because there are broken middleware devices which
scramble the timestamp.
From Yuchung Cheng.
3) Add TCP_NOTSENT_LOWAT socket option to limit the amount of kernel
memory consumed to queue up unsend user data. From Eric Dumazet.
4) Add a "physical port ID" abstraction for network devices, from
Jiri Pirko.
5) Add a "suppress" operation to influence fib_rules lookups, from
Stefan Tomanek.
6) Add a networking development FAQ, from Paul Gortmaker.
7) Extend the information provided by tcp_probe and add ipv6 support,
from Daniel Borkmann.
8) Use RCU locking more extensively in openvswitch data paths, from
Pravin B Shelar.
9) Add SCTP support to openvswitch, from Joe Stringer.
10) Add EF10 chip support to SFC driver, from Ben Hutchings.
11) Add new SYNPROXY netfilter target, from Patrick McHardy.
12) Compute a rate approximation for sending in TCP sockets, and use
this to more intelligently coalesce TSO frames. Furthermore, add
a new packet scheduler which takes advantage of this estimate when
available. From Eric Dumazet.
13) Allow AF_PACKET fanouts with random selection, from Daniel
Borkmann.
14) Add ipv6 support to vxlan driver, from Cong Wang"
Resolved conflicts as per discussion.
* git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next: (1218 commits)
openvswitch: Fix alignment of struct sw_flow_key.
netfilter: Fix build errors with xt_socket.c
tcp: Add missing braces to do_tcp_setsockopt
caif: Add missing braces to multiline if in cfctrl_linkup_request
bnx2x: Add missing braces in bnx2x:bnx2x_link_initialize
vxlan: Fix kernel panic on device delete.
net: mvneta: implement ->ndo_do_ioctl() to support PHY ioctls
net: mvneta: properly disable HW PHY polling and ensure adjust_link() works
icplus: Use netif_running to determine device state
ethernet/arc/arc_emac: Fix huge delays in large file copies
tuntap: orphan frags before trying to set tx timestamp
tuntap: purge socket error queue on detach
qlcnic: use standard NAPI weights
ipv6:introduce function to find route for redirect
bnx2x: VF RSS support - VF side
bnx2x: VF RSS support - PF side
vxlan: Notify drivers for listening UDP port changes
net: usbnet: update addr_assign_type if appropriate
driver/net: enic: update enic maintainers and driver
driver/net: enic: Exposing symbols for Cisco's low latency driver
...
2013-09-05 14:54:29 -07:00
|
|
|
|
2013-07-29 18:16:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
2025-02-26 18:46:43 +01:00
|
|
|
|
2013-07-29 18:16:51 +02:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-29 18:16:51 +02:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-29 18:16:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
Pull networking changes from David Miller:
"Noteworthy changes this time around:
1) Multicast rejoin support for team driver, from Jiri Pirko.
2) Centralize and simplify TCP RTT measurement handling in order to
reduce the impact of bad RTO seeding from SYN/ACKs. Also, when
both timestamps and local RTT measurements are available prefer
the later because there are broken middleware devices which
scramble the timestamp.
From Yuchung Cheng.
3) Add TCP_NOTSENT_LOWAT socket option to limit the amount of kernel
memory consumed to queue up unsend user data. From Eric Dumazet.
4) Add a "physical port ID" abstraction for network devices, from
Jiri Pirko.
5) Add a "suppress" operation to influence fib_rules lookups, from
Stefan Tomanek.
6) Add a networking development FAQ, from Paul Gortmaker.
7) Extend the information provided by tcp_probe and add ipv6 support,
from Daniel Borkmann.
8) Use RCU locking more extensively in openvswitch data paths, from
Pravin B Shelar.
9) Add SCTP support to openvswitch, from Joe Stringer.
10) Add EF10 chip support to SFC driver, from Ben Hutchings.
11) Add new SYNPROXY netfilter target, from Patrick McHardy.
12) Compute a rate approximation for sending in TCP sockets, and use
this to more intelligently coalesce TSO frames. Furthermore, add
a new packet scheduler which takes advantage of this estimate when
available. From Eric Dumazet.
13) Allow AF_PACKET fanouts with random selection, from Daniel
Borkmann.
14) Add ipv6 support to vxlan driver, from Cong Wang"
Resolved conflicts as per discussion.
* git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next: (1218 commits)
openvswitch: Fix alignment of struct sw_flow_key.
netfilter: Fix build errors with xt_socket.c
tcp: Add missing braces to do_tcp_setsockopt
caif: Add missing braces to multiline if in cfctrl_linkup_request
bnx2x: Add missing braces in bnx2x:bnx2x_link_initialize
vxlan: Fix kernel panic on device delete.
net: mvneta: implement ->ndo_do_ioctl() to support PHY ioctls
net: mvneta: properly disable HW PHY polling and ensure adjust_link() works
icplus: Use netif_running to determine device state
ethernet/arc/arc_emac: Fix huge delays in large file copies
tuntap: orphan frags before trying to set tx timestamp
tuntap: purge socket error queue on detach
qlcnic: use standard NAPI weights
ipv6:introduce function to find route for redirect
bnx2x: VF RSS support - VF side
bnx2x: VF RSS support - PF side
vxlan: Notify drivers for listening UDP port changes
net: usbnet: update addr_assign_type if appropriate
driver/net: enic: update enic maintainers and driver
driver/net: enic: Exposing symbols for Cisco's low latency driver
...
2013-09-05 14:54:29 -07:00
|
|
|
|
2013-07-29 18:16:51 +02:00
|
|
|
|
2015-03-17 20:23:15 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
2025-02-26 18:46:43 +01:00
|
|
|
|
2015-03-17 20:23:15 -06:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-17 20:23:15 -06:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-17 20:23:15 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-28 14:34:19 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
2025-02-26 18:46:43 +01:00
|
|
|
|
2014-11-28 14:34:19 +01:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-11-28 14:34:19 +01:00
|
|
|
|
2025-07-17 10:23:27 -07:00
|
|
|
|
2025-02-04 18:03:10 +01:00
|
|
|
|
|
|
|
|
|
2014-11-28 14:34:19 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-12 14:27:07 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-02-08 11:34:10 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-02 17:39:26 +00:00
|
|
|
|
2021-02-08 11:34:10 -08:00
|
|
|
|
|
|
|
|
|
2024-05-02 17:39:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-02-08 11:34:10 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-17 10:23:32 -07:00
|
|
|
|
2021-02-08 11:34:10 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-14 19:53:16 -08:00
|
|
|
|
2021-02-08 11:34:10 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:23 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-02-25 18:17:50 +02:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
net: add name_assign_type netdev attribute
Based on a patch by David Herrmann.
The name_assign_type attribute gives hints where the interface name of a
given net-device comes from. These values are currently defined:
NET_NAME_ENUM:
The ifname is provided by the kernel with an enumerated
suffix, typically based on order of discovery. Names may
be reused and unpredictable.
NET_NAME_PREDICTABLE:
The ifname has been assigned by the kernel in a predictable way
that is guaranteed to avoid reuse and always be the same for a
given device. Examples include statically created devices like
the loopback device and names deduced from hardware properties
(including being given explicitly by the firmware). Names
depending on the order of discovery, or in any other way on the
existence of other devices, must not be marked as PREDICTABLE.
NET_NAME_USER:
The ifname was provided by user-space during net-device setup.
NET_NAME_RENAMED:
The net-device has been renamed from userspace. Once this type is set,
it cannot change again.
NET_NAME_UNKNOWN:
This is an internal placeholder to indicate that we yet haven't yet
categorized the name. It will not be exposed to userspace, rather
-EINVAL is returned.
The aim of these patches is to improve user-space renaming of interfaces. As
a general rule, userspace must rename interfaces to guarantee that names stay
the same every time a given piece of hardware appears (at boot, or when
attaching it). However, there are several situations where userspace should
not perform the renaming, and that depends on both the policy of the local
admin, but crucially also on the nature of the current interface name.
If an interface was created in repsonse to a userspace request, and userspace
already provided a name, we most probably want to leave that name alone. The
main instance of this is wifi-P2P devices created over nl80211, which currently
have a long-standing bug where they are getting renamed by udev. We label such
names NET_NAME_USER.
If an interface, unbeknown to us, has already been renamed from userspace, we
most probably want to leave also that alone. This will typically happen when
third-party plugins (for instance to udev, but the interface is generic so could
be from anywhere) renames the interface without informing udev about it. A
typical situation is when you switch root from an installer or an initrd to the
real system and the new instance of udev does not know what happened before
the switch. These types of problems have caused repeated issues in the past. To
solve this, once an interface has been renamed, its name is labelled
NET_NAME_RENAMED.
In many cases, the kernel is actually able to name interfaces in such a
way that there is no need for userspace to rename them. This is the case when
the enumeration order of devices, or in fact any other (non-parent) device on
the system, can not influence the name of the interface. Examples include
statically created devices, or any naming schemes based on hardware properties
of the interface. In this case the admin may prefer to use the kernel-provided
names, and to make that possible we label such names NET_NAME_PREDICTABLE.
We want the kernel to have tho possibilty of performing predictable interface
naming itself (and exposing to userspace that it has), as the information
necessary for a proper naming scheme for a certain class of devices may not
be exposed to userspace.
The case where renaming is almost certainly desired, is when the kernel has
given the interface a name using global device enumeration based on order of
discovery (ethX, wlanY, etc). These naming schemes are labelled NET_NAME_ENUM.
Lastly, a fallback is left as NET_NAME_UNKNOWN, to indicate that a driver has
not yet been ported. This is mostly useful as a transitionary measure, allowing
us to label the various naming schemes bit by bit.
v8: minor documentation fixes
v9: move comment to the right commit
Signed-off-by: Tom Gundersen <teg@jklm.no>
Reviewed-by: David Herrmann <dh.herrmann@gmail.com>
Reviewed-by: Kay Sievers <kay@vrfy.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-07-14 16:37:22 +02:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-20 00:11:51 +02:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2014-03-29 09:48:35 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: gro: add a per device gro flush timer
Tuning coalescing parameters on NIC can be really hard.
Servers can handle both bulk and RPC like traffic, with conflicting
goals : bulk flows want as big GRO packets as possible, RPC want minimal
latencies.
To reach big GRO packets on 10Gbe NIC, one can use :
ethtool -C eth0 rx-usecs 4 rx-frames 44
But this penalizes rpc sessions, with an increase of latencies, up to
50% in some cases, as NICs generally do not force an interrupt when
a packet with TCP Push flag is received.
Some NICs do not have an absolute timer, only a timer rearmed for every
incoming packet.
This patch uses a different strategy : Let GRO stack decides what do do,
based on traffic pattern.
Packets with Push flag wont be delayed.
Packets without Push flag might be held in GRO engine, if we keep
receiving data.
This new mechanism is off by default, and shall be enabled by setting
/sys/class/net/ethX/gro_flush_timeout to a value in nanosecond.
To fully enable this mechanism, drivers should use napi_complete_done()
instead of napi_complete().
Tested:
Ran 200 netperf TCP_STREAM from A to B (10Gbe mlx4 link, 8 RX queues)
Without this feature, we send back about 305,000 ACK per second.
GRO aggregation ratio is low (811/305 = 2.65 segments per GRO packet)
Setting a timer of 2000 nsec is enough to increase GRO packet sizes
and reduce number of ACK packets. (811/19.2 = 42)
Receiver performs less calls to upper stacks, less wakes up.
This also reduces cpu usage on the sender, as it receives less ACK
packets.
Note that reducing number of wakes up increases cpu efficiency, but can
decrease QPS, as applications wont have the chance to warmup cpu caches
doing a partial read of RPC requests/answers if they fit in one skb.
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811269.80 305732.30 1199462.57 19705.72 0.00
0.00 0.50
B:~# echo 2000 >/sys/class/net/eth0/gro_flush_timeout
B:~# sar -n DEV 1 10 | grep eth0 | tail -1
Average: eth0 811577.30 19230.80 1199916.51 1239.80 0.00
0.00 0.50
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-06 21:09:44 -08:00
|
|
|
|
net: napi: add hard irqs deferral feature
Back in commit 3b47d30396ba ("net: gro: add a per device gro flush timer")
we added the ability to arm one high resolution timer, that we used
to keep not-complete packets in GRO engine a bit longer, hoping that further
frames might be added to them.
Since then, we added the napi_complete_done() interface, and commit
364b6055738b ("net: busy-poll: return busypolling status to drivers")
allowed drivers to avoid re-arming NIC interrupts if we made a promise
that their NAPI poll() handler would be called in the near future.
This infrastructure can be leveraged, thanks to a new device parameter,
which allows to arm the napi hrtimer, instead of re-arming the device
hard IRQ.
We have noticed that on some servers with 32 RX queues or more, the chit-chat
between the NIC and the host caused by IRQ delivery and re-arming could hurt
throughput by ~20% on 100Gbit NIC.
In contrast, hrtimers are using local (percpu) resources and might have lower
cost.
The new tunable, named napi_defer_hard_irqs, is placed in the same hierarchy
than gro_flush_timeout (/sys/class/net/ethX/)
By default, both gro_flush_timeout and napi_defer_hard_irqs are zero.
This patch does not change the prior behavior of gro_flush_timeout
if used alone : NIC hard irqs should be rearmed as before.
One concrete usage can be :
echo 20000 >/sys/class/net/eth1/gro_flush_timeout
echo 10 >/sys/class/net/eth1/napi_defer_hard_irqs
If at least one packet is retired, then we will reset napi counter
to 10 (napi_defer_hard_irqs), ensuring at least 10 periodic scans
of the queue.
On busy queues, this should avoid NIC hard IRQ, while before this patch IRQ
avoidance was only possible if napi->poll() was exhausting its budget
and not call napi_complete_done().
This feature also can be used to work around some non-optimal NIC irq
coalescing strategies.
Having the ability to insert XX usec delays between each napi->poll()
can increase cache efficiency, since we increase batch sizes.
It also keeps serving cpus not idle too long, reducing tail latencies.
Co-developed-by: Luigi Rizzo <lrizzo@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-04-22 09:13:27 -07:00
|
|
|
|
2015-07-14 13:43:19 -07:00
|
|
|
|
2018-01-18 09:59:13 -08:00
|
|
|
|
|
|
|
|
|
2021-02-08 11:34:10 -08:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2010-06-08 07:19:54 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2024-02-13 06:32:40 +00:00
|
|
|
|
2008-05-21 14:12:46 -07:00
|
|
|
|
2010-07-07 14:58:56 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2008-05-21 14:12:46 -07:00
|
|
|
|
2024-02-13 06:32:40 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2010-06-08 07:19:54 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-01 18:51:05 -05:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-08-18 13:46:23 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-01 18:51:05 -05:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-06-29 16:31:26 +05:30
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-11-16 20:46:19 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-06-29 16:31:26 +05:30
|
|
|
|
2012-11-16 20:46:19 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-05-19 13:20:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-11-16 20:46:19 +01:00
|
|
|
|
2022-05-19 13:20:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-05-16 21:59:45 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:24 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:24 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2010-08-31 12:14:13 +00:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2017-08-18 13:46:24 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-02-13 14:37:42 -08:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2015-02-13 14:37:42 -08:00
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2023-02-07 19:44:56 +01:00
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2015-08-13 14:03:16 -04:00
|
|
|
|
2023-02-07 19:44:56 +01:00
|
|
|
|
|
|
|
|
|
2020-06-25 18:34:43 -04:00
|
|
|
|
2012-04-15 05:58:06 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
|
|
|
|
|
2023-02-07 19:44:56 +01:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-08-13 14:03:16 -04:00
|
|
|
|
2010-10-25 03:02:02 +00:00
|
|
|
|
2015-08-13 14:03:16 -04:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2011-11-17 03:13:26 +00:00
|
|
|
|
2019-03-22 08:56:38 -07:00
|
|
|
|
2015-08-05 09:39:27 -07:00
|
|
|
|
2019-03-22 08:56:38 -07:00
|
|
|
|
2015-08-05 09:39:27 -07:00
|
|
|
|
2015-08-13 14:03:16 -04:00
|
|
|
|
2015-08-05 09:39:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-02-07 19:44:56 +01:00
|
|
|
|
|
|
|
|
|
2015-08-05 09:39:27 -07:00
|
|
|
|
2023-02-07 19:44:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-15 16:28:38 +02:00
|
|
|
|
2023-02-07 19:44:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2023-02-07 19:44:56 +01:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-24 06:56:49 +00:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
|
|
|
|
|
2010-04-19 14:40:57 -07:00
|
|
|
|
2017-08-18 13:46:24 -07:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
2011-12-24 06:56:49 +00:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
2011-12-24 06:56:49 +00:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-24 06:56:49 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
|
|
|
|
|
2011-12-24 06:56:49 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-22 13:35:22 +00:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
|
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: Prevent RPS table overwrite of active flows
This patch fixes an issue where two different flows on the same RXq
produce the same hash resulting in continuous flow overwrites.
Flow #1: A packet for Flow #1 comes in, kernel calls the steering
function. The driver gives back a filter id. The kernel saves
this filter id in the selected slot. Later, the driver's
service task checks if any filters have expired and then
installs the rule for Flow #1.
Flow #2: A packet for Flow #2 comes in. It goes through the same steps.
But this time, the chosen slot is being used by Flow #1. The
driver gives a new filter id and the kernel saves it in the
same slot. When the driver's service task runs, it runs through
all the flows, checks if Flow #1 should be expired, the kernel
returns True as the slot has a different filter id, and then
the driver installs the rule for Flow #2.
Flow #1: Another packet for Flow #1 comes in. The same thing repeats.
The slot is overwritten with a new filter id for Flow #1.
This causes a repeated cycle of flow programming for missed packets,
wasting CPU cycles while not improving performance. This problem happens
at higher rates when the RPS table is small, but tests show it still
happens even with 12,000 connections and an RPS size of 16K per queue
(global table size = 144x16K = 64K).
This patch prevents overwriting an rps_dev_flow entry if it is active.
The intention is that it is better to do aRFS for the first flow instead
of hurting all flows on the same hash. Without this, two (or more) flows
on one RX queue with the same hash can keep overwriting each other. This
causes the driver to reprogram the flow repeatedly.
Changes:
1. Add a new 'hash' field to struct rps_dev_flow.
2. Add rps_flow_is_active(): a helper function to check if a flow is
active or not, extracted from rps_may_expire_flow(). It is further
simplified as per reviewer feedback.
3. In set_rps_cpu():
- Avoid overwriting by programming a new filter if:
- The slot is not in use, or
- The slot is in use but the flow is not active, or
- The slot has an active flow with the same hash, but target CPU
differs.
- Save the hash in the rps_dev_flow entry.
4. rps_may_expire_flow(): Use earlier extracted rps_flow_is_active().
Testing & results:
- Driver: ice (E810 NIC), Kernel: net-next
- #CPUs = #RXq = 144 (1:1)
- Number of flows: 12K
- Eight RPS settings from 256 to 32768. Though RPS=256 is not ideal,
it is still sufficient to cover 12K flows (256*144 rx-queues = 64K
global table slots)
- Global Table Size = 144 * RPS (effectively equal to 256 * RPS)
- Each RPS test duration = 8 mins (org code) + 8 mins (new code).
- Metrics captured on client
Legend for following tables:
Steer-C: #times ndo_rx_flow_steer() was Called by set_rps_cpu()
Steer-L: #times ice_arfs_flow_steer() Looped over aRFS entries
Add: #times driver actually programmed aRFS (ice_arfs_build_entry())
Del: #times driver deleted the flow (ice_arfs_del_flow_rules())
Units: K = 1,000 times, M = 1 million times
|-------|---------|------| Org Code |---------|---------|
| RPS | Latency | CPU | Add | Del | Steer-C | Steer-L |
|-------|---------|------|--------|--------|---------|---------|
| 256 | 227.0 | 93.2 | 1.6M | 1.6M | 121.7M | 267.6M |
| 512 | 225.9 | 94.1 | 11.5M | 11.2M | 65.7M | 199.6M |
| 1024 | 223.5 | 95.6 | 16.5M | 16.5M | 27.1M | 187.3M |
| 2048 | 222.2 | 96.3 | 10.5M | 10.5M | 12.5M | 115.2M |
| 4096 | 223.9 | 94.1 | 5.5M | 5.5M | 7.2M | 65.9M |
| 8192 | 224.7 | 92.5 | 2.7M | 2.7M | 3.0M | 29.9M |
| 16384 | 223.5 | 92.5 | 1.3M | 1.3M | 1.4M | 13.9M |
| 32768 | 219.6 | 93.2 | 838.1K | 838.1K | 965.1K | 8.9M |
|-------|---------|------| New Code |---------|---------|
| 256 | 201.5 | 99.1 | 13.4K | 5.0K | 13.7K | 75.2K |
| 512 | 202.5 | 98.2 | 11.2K | 5.9K | 11.2K | 55.5K |
| 1024 | 207.3 | 93.9 | 11.5K | 9.7K | 11.5K | 59.6K |
| 2048 | 207.5 | 96.7 | 11.8K | 11.1K | 15.5K | 79.3K |
| 4096 | 206.9 | 96.6 | 11.8K | 11.7K | 11.8K | 63.2K |
| 8192 | 205.8 | 96.7 | 11.9K | 11.8K | 11.9K | 63.9K |
| 16384 | 200.9 | 98.2 | 11.9K | 11.9K | 11.9K | 64.2K |
| 32768 | 202.5 | 98.0 | 11.9K | 11.9K | 11.9K | 64.2K |
|-------|---------|------|--------|--------|---------|---------|
Some observations:
1. Overall Latency improved: (1790.19-1634.94)/1790.19*100 = 8.67%
2. Overall CPU increased: (777.32-751.49)/751.45*100 = 3.44%
3. Flow Management (add/delete) remained almost constant at ~11K
compared to values in millions.
Signed-off-by: Krishna Kumar <krikku@gmail.com>
Link: https://patch.msgid.link/20250825031005.3674864-2-krikku@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-08-25 08:40:04 +05:30
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
net: Prevent RPS table overwrite of active flows
This patch fixes an issue where two different flows on the same RXq
produce the same hash resulting in continuous flow overwrites.
Flow #1: A packet for Flow #1 comes in, kernel calls the steering
function. The driver gives back a filter id. The kernel saves
this filter id in the selected slot. Later, the driver's
service task checks if any filters have expired and then
installs the rule for Flow #1.
Flow #2: A packet for Flow #2 comes in. It goes through the same steps.
But this time, the chosen slot is being used by Flow #1. The
driver gives a new filter id and the kernel saves it in the
same slot. When the driver's service task runs, it runs through
all the flows, checks if Flow #1 should be expired, the kernel
returns True as the slot has a different filter id, and then
the driver installs the rule for Flow #2.
Flow #1: Another packet for Flow #1 comes in. The same thing repeats.
The slot is overwritten with a new filter id for Flow #1.
This causes a repeated cycle of flow programming for missed packets,
wasting CPU cycles while not improving performance. This problem happens
at higher rates when the RPS table is small, but tests show it still
happens even with 12,000 connections and an RPS size of 16K per queue
(global table size = 144x16K = 64K).
This patch prevents overwriting an rps_dev_flow entry if it is active.
The intention is that it is better to do aRFS for the first flow instead
of hurting all flows on the same hash. Without this, two (or more) flows
on one RX queue with the same hash can keep overwriting each other. This
causes the driver to reprogram the flow repeatedly.
Changes:
1. Add a new 'hash' field to struct rps_dev_flow.
2. Add rps_flow_is_active(): a helper function to check if a flow is
active or not, extracted from rps_may_expire_flow(). It is further
simplified as per reviewer feedback.
3. In set_rps_cpu():
- Avoid overwriting by programming a new filter if:
- The slot is not in use, or
- The slot is in use but the flow is not active, or
- The slot has an active flow with the same hash, but target CPU
differs.
- Save the hash in the rps_dev_flow entry.
4. rps_may_expire_flow(): Use earlier extracted rps_flow_is_active().
Testing & results:
- Driver: ice (E810 NIC), Kernel: net-next
- #CPUs = #RXq = 144 (1:1)
- Number of flows: 12K
- Eight RPS settings from 256 to 32768. Though RPS=256 is not ideal,
it is still sufficient to cover 12K flows (256*144 rx-queues = 64K
global table slots)
- Global Table Size = 144 * RPS (effectively equal to 256 * RPS)
- Each RPS test duration = 8 mins (org code) + 8 mins (new code).
- Metrics captured on client
Legend for following tables:
Steer-C: #times ndo_rx_flow_steer() was Called by set_rps_cpu()
Steer-L: #times ice_arfs_flow_steer() Looped over aRFS entries
Add: #times driver actually programmed aRFS (ice_arfs_build_entry())
Del: #times driver deleted the flow (ice_arfs_del_flow_rules())
Units: K = 1,000 times, M = 1 million times
|-------|---------|------| Org Code |---------|---------|
| RPS | Latency | CPU | Add | Del | Steer-C | Steer-L |
|-------|---------|------|--------|--------|---------|---------|
| 256 | 227.0 | 93.2 | 1.6M | 1.6M | 121.7M | 267.6M |
| 512 | 225.9 | 94.1 | 11.5M | 11.2M | 65.7M | 199.6M |
| 1024 | 223.5 | 95.6 | 16.5M | 16.5M | 27.1M | 187.3M |
| 2048 | 222.2 | 96.3 | 10.5M | 10.5M | 12.5M | 115.2M |
| 4096 | 223.9 | 94.1 | 5.5M | 5.5M | 7.2M | 65.9M |
| 8192 | 224.7 | 92.5 | 2.7M | 2.7M | 3.0M | 29.9M |
| 16384 | 223.5 | 92.5 | 1.3M | 1.3M | 1.4M | 13.9M |
| 32768 | 219.6 | 93.2 | 838.1K | 838.1K | 965.1K | 8.9M |
|-------|---------|------| New Code |---------|---------|
| 256 | 201.5 | 99.1 | 13.4K | 5.0K | 13.7K | 75.2K |
| 512 | 202.5 | 98.2 | 11.2K | 5.9K | 11.2K | 55.5K |
| 1024 | 207.3 | 93.9 | 11.5K | 9.7K | 11.5K | 59.6K |
| 2048 | 207.5 | 96.7 | 11.8K | 11.1K | 15.5K | 79.3K |
| 4096 | 206.9 | 96.6 | 11.8K | 11.7K | 11.8K | 63.2K |
| 8192 | 205.8 | 96.7 | 11.9K | 11.8K | 11.9K | 63.9K |
| 16384 | 200.9 | 98.2 | 11.9K | 11.9K | 11.9K | 64.2K |
| 32768 | 202.5 | 98.0 | 11.9K | 11.9K | 11.9K | 64.2K |
|-------|---------|------|--------|--------|---------|---------|
Some observations:
1. Overall Latency improved: (1790.19-1634.94)/1790.19*100 = 8.67%
2. Overall CPU increased: (777.32-751.49)/751.45*100 = 3.44%
3. Flow Management (add/delete) remained almost constant at ~11K
compared to values in millions.
Signed-off-by: Krishna Kumar <krikku@gmail.com>
Link: https://patch.msgid.link/20250825031005.3674864-2-krikku@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-08-25 08:40:04 +05:30
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
2018-03-23 15:54:38 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
2018-03-23 15:54:38 -07:00
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2019-04-01 22:51:35 -04:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
2010-10-25 03:02:02 +00:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2011-08-11 19:30:52 +00:00
|
|
|
|
2010-11-16 06:31:39 +00:00
|
|
|
|
|
|
|
|
|
2011-03-18 12:01:31 +08:00
|
|
|
|
2010-11-16 06:31:39 +00:00
|
|
|
|
2010-10-25 03:02:02 +00:00
|
|
|
|
2026-03-02 18:14:32 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2010-11-16 06:31:39 +00:00
|
|
|
|
2022-06-07 21:39:55 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2014-01-16 17:24:31 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-05 07:35:38 +08:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2014-01-16 17:24:31 +08:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2014-01-16 17:24:31 +08:00
|
|
|
|
|
|
|
|
|
2022-11-21 10:46:45 +01:00
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
|
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
|
|
|
|
|
2023-02-14 04:23:12 +00:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2023-02-17 13:28:49 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-02 06:15:58 +00:00
|
|
|
|
|
|
|
|
|
2023-02-17 13:28:49 +01:00
|
|
|
|
2025-07-02 06:15:58 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-02-17 13:28:49 +01:00
|
|
|
|
2025-07-02 06:15:58 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-02-17 13:28:49 +01:00
|
|
|
|
2025-07-02 06:15:58 +00:00
|
|
|
|
2023-02-17 13:28:49 +01:00
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:12 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-12-17 13:46:34 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-06-07 21:39:55 -07:00
|
|
|
|
2019-12-17 13:46:34 +02:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2019-11-20 09:08:16 +02:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
|
|
|
|
|
2019-11-20 09:08:16 +02:00
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2023-02-17 13:28:49 +01:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
2023-02-17 13:28:49 +01:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-11-20 09:08:16 +02:00
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
|
|
|
|
|
2019-11-20 09:08:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2020-02-27 04:37:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-02-09 14:07:11 +01:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2010-09-27 08:24:33 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
|
|
|
|
|
2010-09-27 08:24:33 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-09-27 08:24:33 +00:00
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2010-09-27 08:24:33 +00:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
|
|
|
|
|
2016-10-24 19:09:53 -07:00
|
|
|
|
2025-09-18 12:11:53 +02:00
|
|
|
|
2016-10-24 19:09:53 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2016-10-24 19:09:53 -07:00
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
2016-10-24 19:09:53 -07:00
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-09-27 08:24:33 +00:00
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:27 -07:00
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2021-11-16 19:29:21 -08:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
|
|
|
|
|
2015-09-15 18:28:00 -03:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2015-09-15 18:28:00 -03:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
2015-09-15 18:28:00 -03:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2016-10-28 11:43:49 -04:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2016-10-28 11:43:49 -04:00
|
|
|
|
2018-07-09 12:19:32 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-02-08 14:29:18 -08:00
|
|
|
|
2018-07-09 12:19:32 -04:00
|
|
|
|
2018-07-09 12:19:38 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-02-08 14:29:18 -08:00
|
|
|
|
2018-07-09 12:19:32 -04:00
|
|
|
|
2021-02-08 14:29:18 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-10-28 11:43:49 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-07-09 12:19:38 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
|
|
|
|
|
2016-10-28 11:43:49 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2018-07-20 21:56:51 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2021-10-07 16:00:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2025-03-05 08:37:26 -08:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2025-03-05 08:37:26 -08:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:26 -07:00
|
|
|
|
|
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:26 -07:00
|
|
|
|
|
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2017-08-18 13:46:26 -07:00
|
|
|
|
|
|
|
|
|
2016-10-28 11:43:49 -04:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
2012-04-15 05:58:06 +00:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:25 -07:00
|
|
|
|
2018-03-23 15:54:38 -07:00
|
|
|
|
2017-08-18 13:46:25 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-04-04 16:46:04 +00:00
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
2024-04-04 16:46:04 +00:00
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-04-04 16:46:04 +00:00
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-09-28 19:49:44 +08:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:25 -07:00
|
|
|
|
2018-03-23 15:54:38 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:25 -07:00
|
|
|
|
2018-03-23 15:54:38 -07:00
|
|
|
|
2017-08-18 13:46:25 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2017-08-18 13:46:25 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2017-08-18 13:46:25 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: dqs: add NIC stall detector based on BQL
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2024-03-04 06:08:47 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-06-29 16:31:26 +05:30
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-02-16 01:41:52 -08:00
|
|
|
|
|
|
|
|
|
2024-09-02 03:17:30 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:41 +01:00
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:49 +01:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:44 +01:00
|
|
|
|
2021-03-22 16:43:29 +01:00
|
|
|
|
2021-03-18 19:37:44 +01:00
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:44 +01:00
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:43 +01:00
|
|
|
|
2021-03-18 19:37:42 +01:00
|
|
|
|
2016-10-28 11:50:13 -04:00
|
|
|
|
2021-03-18 19:37:45 +01:00
|
|
|
|
2021-03-18 19:37:43 +01:00
|
|
|
|
2021-03-18 19:37:42 +01:00
|
|
|
|
2016-10-28 11:50:13 -04:00
|
|
|
|
2021-03-18 19:37:42 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-11-21 19:01:03 +01:00
|
|
|
|
2021-03-18 19:37:42 +01:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:42 +01:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2020-12-23 22:23:21 +01:00
|
|
|
|
2026-03-19 16:17:11 -04:00
|
|
|
|
2021-03-18 19:37:40 +01:00
|
|
|
|
2020-12-23 22:23:21 +01:00
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:41 +01:00
|
|
|
|
2013-01-10 08:57:02 +00:00
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2018-07-09 12:19:32 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2020-12-23 22:23:20 +01:00
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2020-12-23 22:23:20 +01:00
|
|
|
|
|
|
|
|
|
2013-01-10 08:57:02 +00:00
|
|
|
|
2020-12-23 22:23:20 +01:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-10 08:57:02 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:26 -07:00
|
|
|
|
|
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:50 +01:00
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-12-23 22:23:23 +01:00
|
|
|
|
2021-03-18 19:37:43 +01:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-03-18 19:37:41 +01:00
|
|
|
|
|
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-04 11:48:56 +02:00
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-04 11:48:56 +02:00
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
|
|
|
|
|
2020-12-23 22:23:22 +01:00
|
|
|
|
2025-02-04 18:03:13 +01:00
|
|
|
|
2020-12-23 22:23:22 +01:00
|
|
|
|
|
|
|
|
|
2018-08-08 20:07:35 -07:00
|
|
|
|
2021-03-18 19:37:46 +01:00
|
|
|
|
2018-08-08 20:07:35 -07:00
|
|
|
|
|
|
|
|
|
2020-12-23 22:23:22 +01:00
|
|
|
|
|
|
|
|
|
2019-03-04 11:48:56 +02:00
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2017-08-18 13:46:26 -07:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2016-10-28 11:43:49 -04:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2018-06-29 21:27:07 -07:00
|
|
|
|
2015-03-18 14:57:33 +02:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2019-04-01 22:51:35 -04:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-06-07 21:39:55 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2014-01-16 17:24:31 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-05 07:35:38 +08:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2014-01-16 17:24:31 +08:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2014-01-16 17:24:31 +08:00
|
|
|
|
|
|
|
|
|
2022-11-21 10:46:45 +01:00
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
|
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
|
|
|
|
|
2023-02-14 04:23:12 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2014-01-16 17:24:31 +08:00
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2024-02-16 01:41:52 -08:00
|
|
|
|
|
|
|
|
|
2024-08-29 14:33:37 +02:00
|
|
|
|
2024-02-16 01:41:52 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:12 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-12-05 15:57:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-06-07 21:39:55 -07:00
|
|
|
|
2019-12-05 15:57:07 +02:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2019-11-20 09:08:16 +02:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-02-16 01:41:52 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
2024-02-16 01:41:52 -08:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2019-11-20 19:19:07 -08:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
|
|
|
|
|
2019-11-20 09:08:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2020-02-27 04:37:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-02-16 01:41:52 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:18 +01:00
|
|
|
|
|
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-12-07 15:57:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
2025-09-18 12:11:53 +02:00
|
|
|
|
2016-10-24 19:09:53 -07:00
|
|
|
|
2024-02-16 01:41:52 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 18:03:11 +01:00
|
|
|
|
2011-11-28 16:33:09 +00:00
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
|
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-09-27 08:24:33 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
|
|
|
|
|
2019-03-02 10:34:55 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-11-21 13:17:27 +00:00
|
|
|
|
2010-09-27 08:24:33 +00:00
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
|
|
|
|
|
2014-01-16 22:23:28 -08:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
|
|
|
|
|
2021-12-07 15:57:24 +01:00
|
|
|
|
2025-03-24 15:45:31 -07:00
|
|
|
|
2021-12-07 15:57:24 +01:00
|
|
|
|
|
|
|
|
|
2025-03-24 15:45:31 -07:00
|
|
|
|
2011-11-16 12:15:10 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-11-26 08:36:09 +00:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2013-03-25 20:07:01 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2011-06-08 21:13:01 -04:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
|
|
|
|
|
2011-06-08 21:13:01 -04:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:22 -07:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2013-03-25 20:07:01 -07:00
|
|
|
|
2011-06-08 21:13:01 -04:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
|
|
|
|
|
2011-06-08 21:13:01 -04:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2010-08-05 17:45:15 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2022-11-23 13:25:19 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2022-11-23 13:25:19 +01:00
|
|
|
|
2007-08-14 15:15:12 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2005-11-16 09:00:00 +01:00
|
|
|
|
2007-08-14 15:15:12 +02:00
|
|
|
|
2007-03-30 22:23:12 -07:00
|
|
|
|
|
|
|
|
|
2007-03-07 10:49:30 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
|
|
|
|
|
2007-08-14 15:15:12 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2007-03-30 22:23:12 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-02-09 23:24:36 +09:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-10-02 23:50:05 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
netdevice: define and allocate &net_device _properly_
In fact, this structure contains a flexible array at the end, but
historically its size, alignment etc., is calculated manually.
There are several instances of the structure embedded into other
structures, but also there's ongoing effort to remove them and we
could in the meantime declare &net_device properly.
Declare the array explicitly, use struct_size() and store the array
size inside the structure, so that __counted_by() can be applied.
Don't use PTR_ALIGN(), as SLUB itself tries its best to ensure the
allocated buffer is aligned to what the user expects.
Also, change its alignment from %NETDEV_ALIGN to the cacheline size
as per several suggestions on the netdev ML.
bloat-o-meter for vmlinux:
free_netdev 445 440 -5
netdev_freemem 24 - -24
alloc_netdev_mqs 1481 1450 -31
On x86_64 with several NICs of different vendors, I was never able to
get a &net_device pointer not aligned to the cacheline size after the
change.
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Przemek Kitszel <przemyslaw.kitszel@intel.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Kees Cook <kees@kernel.org>
Link: https://patch.msgid.link/20240710113036.2125584-1-leitao@debian.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-07-10 04:30:28 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2022-10-01 18:54:26 +02:00
|
|
|
|
2015-12-22 23:11:49 +08:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
|
|
|
|
|
2022-10-01 18:54:26 +02:00
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
2022-10-01 18:54:26 +02:00
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-04-12 12:17:57 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2013-07-24 15:05:33 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
|
|
|
|
|
2018-07-20 21:56:52 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2021-10-06 18:06:54 -07:00
|
|
|
|
2015-03-09 14:31:20 -07:00
|
|
|
|
|
|
|
|
|
2020-05-15 11:52:52 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-09 14:31:20 -07:00
|
|
|
|
2020-05-15 11:52:52 +02:00
|
|
|
|
2015-03-09 14:31:20 -07:00
|
|
|
|
|
|
|
|
|
2015-09-24 20:36:33 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-09 14:31:20 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-05-19 15:39:25 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2007-05-19 15:39:25 -07:00
|
|
|
|
2025-09-18 12:11:53 +02:00
|
|
|
|
2016-10-24 19:09:53 -07:00
|
|
|
|
|
|
|
|
|
2007-05-19 15:39:25 -07:00
|
|
|
|
2008-10-27 17:51:47 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2013-02-22 16:34:16 -08:00
|
|
|
|
|
|
|
|
|
2007-05-19 15:39:25 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-08-18 13:46:28 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-05-04 17:36:49 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2002-04-09 12:14:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2007-09-26 22:02:53 -07:00
|
|
|
|
2009-10-29 14:18:21 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-10-29 14:18:21 +00:00
|
|
|
|
2025-06-12 14:27:07 +00:00
|
|
|
|
2012-11-16 20:46:19 +01:00
|
|
|
|
2022-05-19 13:20:54 -07:00
|
|
|
|
2012-11-16 20:46:19 +01:00
|
|
|
|
2007-09-26 22:02:53 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
|
|
|
|
|
2019-04-12 16:36:33 -04:00
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2014-07-23 16:09:10 -07:00
|
|
|
|
2019-04-12 16:36:33 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2013-02-22 16:34:16 -08:00
|
|
|
|
|
|
|
|
|
2010-03-16 08:03:29 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:17 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-10-25 02:31:48 -04:00
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:17 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-02-27 04:37:17 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-18 13:46:20 -07:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2008-06-13 18:12:04 -07:00
|
|
|
|
2013-09-11 22:29:04 -04:00
|
|
|
|
2008-06-13 18:12:04 -07:00
|
|
|
|
2013-09-11 22:29:04 -04:00
|
|
|
|
2008-06-13 18:12:04 -07:00
|
|
|
|
2017-08-18 13:46:20 -07:00
|
|
|
|
2026-04-01 12:15:58 +02:00
|
|
|
|
2008-06-13 18:12:04 -07:00
|
|
|
|
2013-09-11 22:29:04 -04:00
|
|
|
|
2008-06-13 18:12:04 -07:00
|
|
|
|
2013-09-11 22:29:04 -04:00
|
|
|
|
2008-06-13 18:12:04 -07:00
|
|
|
|
2014-01-06 01:20:11 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-05-04 17:36:45 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
