2019-05-27 08:55:01 +02:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-01-05 13:56:15 -08:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2007-03-23 11:40:27 -07:00
|
|
|
|
rfs: Receive Flow Steering
This patch implements receive flow steering (RFS). RFS steers
received packets for layer 3 and 4 processing to the CPU where
the application for the corresponding flow is running. RFS is an
extension of Receive Packet Steering (RPS).
The basic idea of RFS is that when an application calls recvmsg
(or sendmsg) the application's running CPU is stored in a hash
table that is indexed by the connection's rxhash which is stored in
the socket structure. The rxhash is passed in skb's received on
the connection from netif_receive_skb. For each received packet,
the associated rxhash is used to look up the CPU in the hash table,
if a valid CPU is set then the packet is steered to that CPU using
the RPS mechanisms.
The convolution of the simple approach is that it would potentially
allow OOO packets. If threads are thrashing around CPUs or multiple
threads are trying to read from the same sockets, a quickly changing
CPU value in the hash table could cause rampant OOO packets--
we consider this a non-starter.
To avoid OOO packets, this solution implements two types of hash
tables: rps_sock_flow_table and rps_dev_flow_table.
rps_sock_table is a global hash table. Each entry is just a CPU
number and it is populated in recvmsg and sendmsg as described above.
This table contains the "desired" CPUs for flows.
rps_dev_flow_table is specific to each device queue. Each entry
contains a CPU and a tail queue counter. The CPU is the "current"
CPU for a matching flow. The tail queue counter holds the value
of a tail queue counter for the associated CPU's backlog queue at
the time of last enqueue for a flow matching the entry.
Each backlog queue has a queue head counter which is incremented
on dequeue, and so a queue tail counter is computed as queue head
count + queue length. When a packet is enqueued on a backlog queue,
the current value of the queue tail counter is saved in the hash
entry of the rps_dev_flow_table.
And now the trick: when selecting the CPU for RPS (get_rps_cpu)
the rps_sock_flow table and the rps_dev_flow table for the RX queue
are consulted. When the desired CPU for the flow (found in the
rps_sock_flow table) does not match the current CPU (found in the
rps_dev_flow table), the current CPU is changed to the desired CPU
if one of the following is true:
- The current CPU is unset (equal to RPS_NO_CPU)
- Current CPU is offline
- The current CPU's queue head counter >= queue tail counter in the
rps_dev_flow table. This checks if the queue tail has advanced
beyond the last packet that was enqueued using this table entry.
This guarantees that all packets queued using this entry have been
dequeued, thus preserving in order delivery.
Making each queue have its own rps_dev_flow table has two advantages:
1) the tail queue counters will be written on each receive, so
keeping the table local to interrupting CPU s good for locality. 2)
this allows lockless access to the table-- the CPU number and queue
tail counter need to be accessed together under mutual exclusion
from netif_receive_skb, we assume that this is only called from
device napi_poll which is non-reentrant.
This patch implements RFS for TCP and connected UDP sockets.
It should be usable for other flow oriented protocols.
There are two configuration parameters for RFS. The
"rps_flow_entries" kernel init parameter sets the number of
entries in the rps_sock_flow_table, the per rxqueue sysfs entry
"rps_flow_cnt" contains the number of entries in the rps_dev_flow
table for the rxqueue. Both are rounded to power of two.
The obvious benefit of RFS (over just RPS) is that it achieves
CPU locality between the receive processing for a flow and the
applications processing; this can result in increased performance
(higher pps, lower latency).
The benefits of RFS are dependent on cache hierarchy, application
load, and other factors. On simple benchmarks, we don't necessarily
see improvement and sometimes see degradation. However, for more
complex benchmarks and for applications where cache pressure is
much higher this technique seems to perform very well.
Below are some benchmark results which show the potential benfit of
this patch. The netperf test has 500 instances of netperf TCP_RR
test with 1 byte req. and resp. The RPC test is an request/response
test similar in structure to netperf RR test ith 100 threads on
each host, but does more work in userspace that netperf.
e1000e on 8 core Intel
No RFS or RPS 104K tps at 30% CPU
No RFS (best RPS config): 290K tps at 63% CPU
RFS 303K tps at 61% CPU
RPC test tps CPU% 50/90/99% usec latency Latency StdDev
No RFS/RPS 103K 48% 757/900/3185 4472.35
RPS only: 174K 73% 415/993/2468 491.66
RFS 223K 73% 379/651/1382 315.61
Signed-off-by: Tom Herbert <therbert@google.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-04-16 16:01:27 -07:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2025-01-02 17:34:18 +01:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2008-06-16 17:14:11 -07:00
|
|
|
|
2015-03-12 16:44:05 -07:00
|
|
|
|
2015-12-16 13:20:44 -08:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2026-01-05 15:45:25 +09:00
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-11-22 23:33:10 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-09-27 18:28:07 -07:00
|
|
|
|
2011-11-22 23:33:10 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-03-22 16:35:29 -07:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-08-03 16:46:20 -07:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2020-03-02 06:07:42 -06:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2011-04-21 09:45:37 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-05 15:45:25 +09:00
|
|
|
|
|
|
|
|
|
2011-04-21 09:45:37 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-10-09 15:21:29 -07:00
|
|
|
|
|
|
|
|
|
2013-10-10 00:04:37 -07:00
|
|
|
|
2013-10-09 15:21:29 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-11 18:53:14 -07:00
|
|
|
|
|
|
|
|
|
2015-03-12 16:44:05 -07:00
|
|
|
|
2015-03-12 16:44:10 -07:00
|
|
|
|
2013-10-09 15:21:29 -07:00
|
|
|
|
2008-09-09 06:43:12 +02:00
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-10-01 07:41:00 -07:00
|
|
|
|
2017-10-25 11:01:45 +02:00
|
|
|
|
|
|
|
|
|
2015-03-16 21:06:18 -07:00
|
|
|
|
2014-06-25 17:09:52 +03:00
|
|
|
|
2017-10-20 09:04:13 -07:00
|
|
|
|
2016-06-27 15:05:28 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-06-25 17:09:52 +03:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2026-01-20 12:53:53 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2015-03-16 21:06:18 -07:00
|
|
|
|
net: support marking accepting TCP sockets
When using mark-based routing, sockets returned from accept()
may need to be marked differently depending on the incoming
connection request.
This is the case, for example, if different socket marks identify
different networks: a listening socket may want to accept
connections from all networks, but each connection should be
marked with the network that the request came in on, so that
subsequent packets are sent on the correct network.
This patch adds a sysctl to mark TCP sockets based on the fwmark
of the incoming SYN packet. If enabled, and an unmarked socket
receives a SYN, then the SYN packet's fwmark is written to the
connection's inet_request_sock, and later written back to the
accepted socket when the connection is established. If the
socket already has a nonzero mark, then the behaviour is the same
as it is today, i.e., the listening socket's fwmark is used.
Black-box tested using user-mode linux:
- IPv4/IPv6 SYN+ACK, FIN, etc. packets are routed based on the
mark of the incoming SYN packet.
- The socket returned by accept() is marked with the mark of the
incoming SYN packet.
- Tested with syncookies=1 and syncookies=2.
Signed-off-by: Lorenzo Colitti <lorenzo@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-05-13 10:17:35 -07:00
|
|
|
|
2023-07-28 15:03:15 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
net: support marking accepting TCP sockets
When using mark-based routing, sockets returned from accept()
may need to be marked differently depending on the incoming
connection request.
This is the case, for example, if different socket marks identify
different networks: a listening socket may want to accept
connections from all networks, but each connection should be
marked with the network that the request came in on, so that
subsequent packets are sent on the correct network.
This patch adds a sysctl to mark TCP sockets based on the fwmark
of the incoming SYN packet. If enabled, and an unmarked socket
receives a SYN, then the SYN packet's fwmark is written to the
connection's inet_request_sock, and later written back to the
accepted socket when the connection is established. If the
socket already has a nonzero mark, then the behaviour is the same
as it is today, i.e., the listening socket's fwmark is used.
Black-box tested using user-mode linux:
- IPv4/IPv6 SYN+ACK, FIN, etc. packets are routed based on the
mark of the incoming SYN packet.
- The socket returned by accept() is marked with the mark of the
incoming SYN packet.
- Tested with syncookies=1 and syncookies=2.
Signed-off-by: Lorenzo Colitti <lorenzo@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-05-13 10:17:35 -07:00
|
|
|
|
2015-03-16 21:06:18 -07:00
|
|
|
|
2023-07-28 15:03:15 +00:00
|
|
|
|
net: support marking accepting TCP sockets
When using mark-based routing, sockets returned from accept()
may need to be marked differently depending on the incoming
connection request.
This is the case, for example, if different socket marks identify
different networks: a listening socket may want to accept
connections from all networks, but each connection should be
marked with the network that the request came in on, so that
subsequent packets are sent on the correct network.
This patch adds a sysctl to mark TCP sockets based on the fwmark
of the incoming SYN packet. If enabled, and an unmarked socket
receives a SYN, then the SYN packet's fwmark is written to the
connection's inet_request_sock, and later written back to the
accepted socket when the connection is established. If the
socket already has a nonzero mark, then the behaviour is the same
as it is today, i.e., the listening socket's fwmark is used.
Black-box tested using user-mode linux:
- IPv4/IPv6 SYN+ACK, FIN, etc. packets are routed based on the
mark of the incoming SYN packet.
- The socket returned by accept() is marked with the mark of the
incoming SYN packet.
- Tested with syncookies=1 and syncookies=2.
Signed-off-by: Lorenzo Colitti <lorenzo@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-05-13 10:17:35 -07:00
|
|
|
|
|
|
|
|
|
2015-12-16 13:20:44 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-05-13 11:55:43 -07:00
|
|
|
|
2015-12-16 13:20:44 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-07-13 13:51:59 -07:00
|
|
|
|
2015-12-16 13:20:44 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-05-13 11:55:43 -07:00
|
|
|
|
2015-12-16 13:20:44 -08:00
|
|
|
|
|
|
|
|
|
2018-11-07 15:36:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-07-13 13:51:59 -07:00
|
|
|
|
2018-11-07 15:36:02 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-07 15:36:03 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-08-02 13:40:25 +00:00
|
|
|
|
|
|
|
|
|
2022-07-25 19:14:42 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-30 21:03:03 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
|
|
|
|
|
2011-05-06 15:02:07 -07:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2011-05-06 15:02:07 -07:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
net: use a per task frag allocator
We currently use a per socket order-0 page cache for tcp_sendmsg()
operations.
This page is used to build fragments for skbs.
Its done to increase probability of coalescing small write() into
single segments in skbs still in write queue (not yet sent)
But it wastes a lot of memory for applications handling many mostly
idle sockets, since each socket holds one page in sk->sk_sndmsg_page
Its also quite inefficient to build TSO 64KB packets, because we need
about 16 pages per skb on arches where PAGE_SIZE = 4096, so we hit
page allocator more than wanted.
This patch adds a per task frag allocator and uses bigger pages,
if available. An automatic fallback is done in case of memory pressure.
(up to 32768 bytes per frag, thats order-3 pages on x86)
This increases TCP stream performance by 20% on loopback device,
but also benefits on other network devices, since 8x less frags are
mapped on transmit and unmapped on tx completion. Alexander Duyck
mentioned a probable performance win on systems with IOMMU enabled.
Its possible some SG enabled hardware cant cope with bigger fragments,
but their ndo_start_xmit() should already handle this, splitting a
fragment in sub fragments, since some arches have PAGE_SIZE=65536
Successfully tested on various ethernet devices.
(ixgbe, igb, bnx2x, tg3, mellanox mlx4)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Vijay Subramanian <subramanian.vijay@gmail.com>
Cc: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Vijay Subramanian <subramanian.vijay@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-23 23:04:42 +00:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2013-09-24 15:43:09 +02:00
|
|
|
|
|
|
|
|
|
sock: support SO_PRIORITY cmsg
The Linux socket API currently allows setting SO_PRIORITY at the
socket level, applying a uniform priority to all packets sent through
that socket. The exception to this is IP_TOS, when the priority value
is calculated during the handling of
ancillary data, as implemented in commit f02db315b8d8 ("ipv4: IP_TOS
and IP_TTL can be specified as ancillary data").
However, this is a computed
value, and there is currently no mechanism to set a custom priority
via control messages prior to this patch.
According to this patch, if SO_PRIORITY is specified as ancillary data,
the packet is sent with the priority value set through
sockc->priority, overriding the socket-level values
set via the traditional setsockopt() method. This is analogous to
the existing support for SO_MARK, as implemented in
commit c6af0c227a22 ("ip: support SO_MARK cmsg").
If both cmsg SO_PRIORITY and IP_TOS are passed, then the one that
takes precedence is the last one in the cmsg list.
This patch has the side effect that raw_send_hdrinc now interprets cmsg
IP_TOS.
Reviewed-by: Willem de Bruijn <willemb@google.com>
Suggested-by: Ferenc Fejes <fejes@inf.elte.hu>
Signed-off-by: Anna Emese Nyiri <annaemesenyiri@gmail.com>
Link: https://patch.msgid.link/20241213084457.45120-3-annaemesenyiri@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-12-13 09:44:55 +01:00
|
|
|
|
udp: generate gso with UDP_SEGMENT
Support generic segmentation offload for udp datagrams. Callers can
concatenate and send at once the payload of multiple datagrams with
the same destination.
To set segment size, the caller sets socket option UDP_SEGMENT to the
length of each discrete payload. This value must be smaller than or
equal to the relevant MTU.
A follow-up patch adds cmsg UDP_SEGMENT to specify segment size on a
per send call basis.
Total byte length may then exceed MTU. If not an exact multiple of
segment size, the last segment will be shorter.
The implementation adds a gso_size field to the udp socket, ip(v6)
cmsg cookie and inet_cork structure to be able to set the value at
setsockopt or cmsg time and to work with both lockless and corked
paths.
Initial benchmark numbers show UDP GSO about as expensive as TCP GSO.
tcp tso
3197 MB/s 54232 msg/s 54232 calls/s
6,457,754,262 cycles
tcp gso
1765 MB/s 29939 msg/s 29939 calls/s
11,203,021,806 cycles
tcp without tso/gso *
739 MB/s 12548 msg/s 12548 calls/s
11,205,483,630 cycles
udp
876 MB/s 14873 msg/s 624666 calls/s
11,205,777,429 cycles
udp gso
2139 MB/s 36282 msg/s 36282 calls/s
11,204,374,561 cycles
[*] after reverting commit 0a6b2a1dc2a2
("tcp: switch to GSO being always on")
Measured total system cycles ('-a') for one core while pinning both
the network receive path and benchmark process to that core:
perf stat -a -C 12 -e cycles \
./udpgso_bench_tx -C 12 -4 -D "$DST" -l 4
Note the reduction in calls/s with GSO. Bytes per syscall drops
increases from 1470 to 61818.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-26 13:42:17 -04:00
|
|
|
|
2024-10-01 05:57:14 -07:00
|
|
|
|
2018-07-03 15:42:49 -07:00
|
|
|
|
2019-09-11 15:50:51 -04:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
|
|
|
|
|
2011-05-06 15:02:07 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-30 21:03:03 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-06 15:02:07 -07:00
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-10-15 06:30:45 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-16 08:15:33 +00:00
|
|
|
|
2009-10-15 06:30:45 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2009-10-15 06:30:45 +00:00
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2012-02-08 09:11:07 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-10 09:48:31 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
ipv6: Move ipv6_fl_list from ipv6_pinfo to inet_sock.
In {tcp6,udp6,raw6}_sock, struct ipv6_pinfo is always placed at
the beginning of a new cache line because
1. __alignof__(struct tcp_sock) is 64 due to ____cacheline_aligned
of __cacheline_group_begin(tcp_sock_write_tx)
2. __alignof__(struct udp_sock) is 64 due to ____cacheline_aligned
of struct numa_drop_counters
3. in raw6_sock, struct numa_drop_counters is placed before
struct ipv6_pinfo
. struct ipv6_pinfo is 136 bytes, but the last cache line is
only used by ipv6_fl_list:
$ pahole -C ipv6_pinfo vmlinux
struct ipv6_pinfo {
...
/* --- cacheline 2 boundary (128 bytes) --- */
struct ipv6_fl_socklist * ipv6_fl_list; /* 128 8 */
/* size: 136, cachelines: 3, members: 23 */
Let's move ipv6_fl_list from struct ipv6_pinfo to struct inet_sock
to save a full cache line for {tcp6,udp6,raw6}_sock.
Now, struct ipv6_pinfo is 128 bytes, and {tcp6,udp6,raw6}_sock have
64 bytes less, while {tcp,udp,raw}_sock retain the same size.
Before:
# grep -E "^(RAW|UDP[^L\-]|TCP)" /proc/slabinfo | awk '{print $1, "\t", $4}'
RAWv6 1408
UDPv6 1472
TCPv6 2560
RAW 1152
UDP 1280
TCP 2368
After:
# grep -E "^(RAW|UDP[^L\-]|TCP)" /proc/slabinfo | awk '{print $1, "\t", $4}'
RAWv6 1344
UDPv6 1408
TCPv6 2496
RAW 1152
UDP 1280
TCP 2368
Also, ipv6_fl_list and inet_flags (SNDFLOW bit) are placed in the
same cache line.
$ pahole -C inet_sock vmlinux
...
/* --- cacheline 11 boundary (704 bytes) was 56 bytes ago --- */
struct ipv6_pinfo * pinet6; /* 760 8 */
/* --- cacheline 12 boundary (768 bytes) --- */
struct ipv6_fl_socklist * ipv6_fl_list; /* 768 8 */
unsigned long inet_flags; /* 776 8 */
Doc churn is due to the insufficient Type column (only 1 space short).
Suggested-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20251014224210.2964778-1-kuniyu@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-10-14 22:42:07 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
net: optimize INET input path further
Followup of commit b178bb3dfc30 (net: reorder struct sock fields)
Optimize INET input path a bit further, by :
1) moving sk_refcnt close to sk_lock.
This reduces number of dirtied cache lines by one on 64bit arches (and
64 bytes cache line size).
2) moving inet_daddr & inet_rcv_saddr at the beginning of sk
(same cache line than hash / family / bound_dev_if / nulls_node)
This reduces number of accessed cache lines in lookups by one, and dont
increase size of inet and timewait socks.
inet and tw sockets now share same place-holder for these fields.
Before patch :
offsetof(struct sock, sk_refcnt) = 0x10
offsetof(struct sock, sk_lock) = 0x40
offsetof(struct sock, sk_receive_queue) = 0x60
offsetof(struct inet_sock, inet_daddr) = 0x270
offsetof(struct inet_sock, inet_rcv_saddr) = 0x274
After patch :
offsetof(struct sock, sk_refcnt) = 0x44
offsetof(struct sock, sk_lock) = 0x48
offsetof(struct sock, sk_receive_queue) = 0x68
offsetof(struct inet_sock, inet_daddr) = 0x0
offsetof(struct inet_sock, inet_rcv_saddr) = 0x4
compute_score() (udp or tcp) now use a single cache line per ignored
item, instead of two.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-11-30 19:04:07 +00:00
|
|
|
|
|
|
|
|
|
2012-11-30 09:49:27 +00:00
|
|
|
|
|
|
|
|
|
net: optimize INET input path further
Followup of commit b178bb3dfc30 (net: reorder struct sock fields)
Optimize INET input path a bit further, by :
1) moving sk_refcnt close to sk_lock.
This reduces number of dirtied cache lines by one on 64bit arches (and
64 bytes cache line size).
2) moving inet_daddr & inet_rcv_saddr at the beginning of sk
(same cache line than hash / family / bound_dev_if / nulls_node)
This reduces number of accessed cache lines in lookups by one, and dont
increase size of inet and timewait socks.
inet and tw sockets now share same place-holder for these fields.
Before patch :
offsetof(struct sock, sk_refcnt) = 0x10
offsetof(struct sock, sk_lock) = 0x40
offsetof(struct sock, sk_receive_queue) = 0x60
offsetof(struct inet_sock, inet_daddr) = 0x270
offsetof(struct inet_sock, inet_rcv_saddr) = 0x274
After patch :
offsetof(struct sock, sk_refcnt) = 0x44
offsetof(struct sock, sk_lock) = 0x48
offsetof(struct sock, sk_receive_queue) = 0x68
offsetof(struct inet_sock, inet_daddr) = 0x0
offsetof(struct inet_sock, inet_rcv_saddr) = 0x4
compute_score() (udp or tcp) now use a single cache line per ignored
item, instead of two.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-11-30 19:04:07 +00:00
|
|
|
|
2023-08-16 08:15:33 +00:00
|
|
|
|
2009-10-15 06:30:45 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2009-10-15 06:30:45 +00:00
|
|
|
|
2023-08-16 08:15:33 +00:00
|
|
|
|
2023-08-19 03:17:07 +00:00
|
|
|
|
2010-01-11 16:28:01 -08:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2010-01-11 16:28:01 -08:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2012-02-09 09:35:49 +00:00
|
|
|
|
2015-01-05 13:56:14 -08:00
|
|
|
|
2012-02-08 09:11:07 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2006-09-26 21:27:35 -07:00
|
|
|
|
2023-12-06 13:44:20 +00:00
|
|
|
|
2023-08-16 08:15:45 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2024-10-01 05:57:14 -07:00
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2023-08-16 08:15:33 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-16 08:15:35 +00:00
|
|
|
|
|
|
|
|
|
2023-08-16 08:15:36 +00:00
|
|
|
|
2023-08-16 08:15:37 +00:00
|
|
|
|
2023-08-16 08:15:38 +00:00
|
|
|
|
2023-08-16 08:15:39 +00:00
|
|
|
|
2023-08-16 08:15:40 +00:00
|
|
|
|
2023-08-16 08:15:41 +00:00
|
|
|
|
2023-08-16 08:15:42 +00:00
|
|
|
|
2023-08-16 08:15:43 +00:00
|
|
|
|
2023-08-16 08:15:44 +00:00
|
|
|
|
2023-08-16 08:15:45 +00:00
|
|
|
|
2023-09-12 16:02:00 +00:00
|
|
|
|
2023-09-12 16:02:04 +00:00
|
|
|
|
2023-09-12 16:02:05 +00:00
|
|
|
|
2023-09-12 16:02:06 +00:00
|
|
|
|
|
|
|
|
|
2023-09-12 16:02:07 +00:00
|
|
|
|
2023-09-12 16:02:08 +00:00
|
|
|
|
2023-09-12 16:02:09 +00:00
|
|
|
|
2023-09-12 16:02:10 +00:00
|
|
|
|
2023-09-12 16:02:12 +00:00
|
|
|
|
2024-03-04 11:32:08 +00:00
|
|
|
|
2023-08-16 08:15:33 +00:00
|
|
|
|
|
|
|
|
|
2015-01-05 13:56:15 -08:00
|
|
|
|
2023-08-16 08:15:33 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-02 17:34:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-16 08:15:33 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-01-05 13:56:15 -08:00
|
|
|
|
2015-12-07 08:53:17 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-11-08 10:54:07 -08:00
|
|
|
|
|
|
|
|
|
2015-12-07 08:53:17 -08:00
|
|
|
|
2015-11-08 10:54:07 -08:00
|
|
|
|
2015-12-07 08:53:17 -08:00
|
|
|
|
2024-10-10 17:48:13 +00:00
|
|
|
|
2015-11-08 10:54:07 -08:00
|
|
|
|
2024-10-10 17:48:13 +00:00
|
|
|
|
|
|
|
|
|
2015-12-07 08:53:17 -08:00
|
|
|
|
2015-11-08 10:54:07 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-12-07 08:53:17 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-10-10 17:48:13 +00:00
|
|
|
|
2015-12-07 08:53:17 -08:00
|
|
|
|
2024-10-10 17:48:13 +00:00
|
|
|
|
|
|
|
|
|
2015-12-07 08:53:17 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-03-16 15:31:55 +00:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2013-09-21 10:22:41 -07:00
|
|
|
|
2017-12-20 11:12:52 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-20 11:12:51 +08:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2017-12-20 11:12:52 +08:00
|
|
|
|
|
|
|
|
|
2013-10-19 21:48:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
2008-03-04 14:28:41 -08:00
|
|
|
|
|
|
|
|
|
2007-03-23 11:40:27 -07:00
|
|
|
|
2013-10-19 21:48:51 +02:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2015-03-17 18:32:27 -07:00
|
|
|
|
2015-10-04 21:08:11 -07:00
|
|
|
|
|
|
|
|
|
2008-06-10 12:39:35 -07:00
|
|
|
|
2008-10-01 07:41:00 -07:00
|
|
|
|
|
|
|
|
|
2011-01-27 22:01:53 -08:00
|
|
|
|
|
|
|
|
|
2023-08-16 08:15:41 +00:00
|
|
|
|
2011-01-27 22:01:53 -08:00
|
|
|
|
|
|
|
|
|
2008-10-01 07:41:00 -07:00
|
|
|
|
|
|
|
|
|
2015-01-05 13:56:14 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-07-31 21:18:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-07-13 13:51:55 -07:00
|
|
|
|
2023-08-16 08:15:37 +00:00
|
|
|
|
2023-08-16 08:15:41 +00:00
|
|
|
|
2018-07-31 21:18:11 +02:00
|
|
|
|
|
|
|
|
|
2021-11-17 10:00:11 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
