2019-05-27 08:55:01 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-05-05 16:16:16 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-03-12 20:09:15 -03:00
|
|
|
|
2017-12-01 12:52:30 -08:00
|
|
|
|
2020-07-23 08:08:57 +02:00
|
|
|
|
2021-10-25 09:48:24 -07:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2013-09-24 15:43:09 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-10-01 07:44:42 -07:00
|
|
|
|
2015-05-12 14:56:07 +02:00
|
|
|
|
2017-12-01 12:52:30 -08:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-04-12 12:34:03 +08:00
|
|
|
|
2017-12-11 07:17:39 -08:00
|
|
|
|
2017-04-12 12:34:03 +08:00
|
|
|
|
2019-03-20 09:18:59 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2016-05-10 11:19:51 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2016-10-16 20:02:52 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-01-23 12:01:26 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-05-22 16:32:51 +02:00
|
|
|
|
2016-11-02 16:36:17 -04:00
|
|
|
|
2022-05-13 23:34:02 +03:00
|
|
|
|
2012-08-26 19:13:55 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2016-10-16 20:02:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-03-12 20:09:15 -03:00
|
|
|
|
|
|
|
|
|
2007-04-20 22:47:35 -07:00
|
|
|
|
2007-03-12 20:09:15 -03:00
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2016-04-02 23:08:10 -04:00
|
|
|
|
2006-09-27 18:28:28 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-04-21 09:45:37 +00:00
|
|
|
|
2013-09-24 15:43:08 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
udp: generate gso with UDP_SEGMENT
Support generic segmentation offload for udp datagrams. Callers can
concatenate and send at once the payload of multiple datagrams with
the same destination.
To set segment size, the caller sets socket option UDP_SEGMENT to the
length of each discrete payload. This value must be smaller than or
equal to the relevant MTU.
A follow-up patch adds cmsg UDP_SEGMENT to specify segment size on a
per send call basis.
Total byte length may then exceed MTU. If not an exact multiple of
segment size, the last segment will be shorter.
The implementation adds a gso_size field to the udp socket, ip(v6)
cmsg cookie and inet_cork structure to be able to set the value at
setsockopt or cmsg time and to work with both lockless and corked
paths.
Initial benchmark numbers show UDP GSO about as expensive as TCP GSO.
tcp tso
3197 MB/s 54232 msg/s 54232 calls/s
6,457,754,262 cycles
tcp gso
1765 MB/s 29939 msg/s 29939 calls/s
11,203,021,806 cycles
tcp without tso/gso *
739 MB/s 12548 msg/s 12548 calls/s
11,205,483,630 cycles
udp
876 MB/s 14873 msg/s 624666 calls/s
11,205,777,429 cycles
udp gso
2139 MB/s 36282 msg/s 36282 calls/s
11,204,374,561 cycles
[*] after reverting commit 0a6b2a1dc2a2
("tcp: switch to GSO being always on")
Measured total system cycles ('-a') for one core while pinning both
the network receive path and benchmark process to that core:
perf stat -a -C 12 -e cycles \
./udpgso_bench_tx -C 12 -4 -D "$DST" -l 4
Note the reduction in calls/s with GSO. Bytes per syscall drops
increases from 1470 to 61818.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-26 13:42:17 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2018-07-06 10:12:54 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-11 15:50:51 -04:00
|
|
|
|
2018-07-06 10:12:54 -04:00
|
|
|
|
2022-05-13 11:55:41 -07:00
|
|
|
|
2018-07-06 10:12:54 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-01-20 03:43:08 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-08-07 08:44:16 -07:00
|
|
|
|
2020-11-09 15:13:48 -08:00
|
|
|
|
2017-08-07 08:44:16 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-22 12:45:32 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2010-10-25 03:32:44 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-09 16:21:07 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-07 03:12:08 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-10 16:09:45 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-09-25 07:39:16 -07:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2020-09-09 17:50:47 -07:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
2018-07-02 16:14:12 +01:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2018-11-07 12:38:31 +01:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2015-10-07 16:48:47 -05:00
|
|
|
|
|
|
|
|
|
2015-06-12 21:55:31 -05:00
|
|
|
|
|
|
|
|
|
2019-05-29 13:25:31 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-05-29 13:25:33 +02:00
|
|
|
|
2019-10-19 09:26:37 -07:00
|
|
|
|
2019-05-29 13:25:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-10-19 09:26:37 -07:00
|
|
|
|
2019-05-29 13:25:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2015-10-07 16:48:45 -05:00
|
|
|
|
2015-10-07 16:48:46 -05:00
|
|
|
|
2014-04-15 13:47:15 -04:00
|
|
|
|
2018-07-02 18:21:11 +08:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-26 13:42:15 -04:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2020-06-19 12:12:34 -07:00
|
|
|
|
2018-07-02 18:21:11 +08:00
|
|
|
|
2011-05-08 17:12:19 -07:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2011-05-08 17:12:19 -07:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-09-24 15:43:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-07-14 08:10:22 +02:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2013-01-21 02:00:03 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2008-10-01 07:41:00 -07:00
|
|
|
|
2006-11-14 21:26:08 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2007-06-04 21:32:46 -07:00
|
|
|
|
2011-10-24 03:06:21 -04:00
|
|
|
|
2016-11-04 02:23:43 +09:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-10-01 07:41:00 -07:00
|
|
|
|
|
|
|
|
|
2008-10-01 07:44:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-01-29 21:35:05 -08:00
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-13 21:22:35 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2008-07-18 04:03:08 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2014-03-06 15:03:17 -05:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2021-09-29 18:03:32 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-05 15:55:55 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2015-08-30 11:29:41 +05:30
|
|
|
|
|
|
|
|
|
2014-05-05 15:55:55 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2015-08-30 11:29:41 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-05 15:55:55 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-04-20 15:57:15 -07:00
|
|
|
|
2016-09-30 11:28:58 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
inet: Add IP_LOCAL_PORT_RANGE socket option
Users who want to share a single public IP address for outgoing connections
between several hosts traditionally reach for SNAT. However, SNAT requires
state keeping on the node(s) performing the NAT.
A stateless alternative exists, where a single IP address used for egress
can be shared between several hosts by partitioning the available ephemeral
port range. In such a setup:
1. Each host gets assigned a disjoint range of ephemeral ports.
2. Applications open connections from the host-assigned port range.
3. Return traffic gets routed to the host based on both, the destination IP
and the destination port.
An application which wants to open an outgoing connection (connect) from a
given port range today can choose between two solutions:
1. Manually pick the source port by bind()'ing to it before connect()'ing
the socket.
This approach has a couple of downsides:
a) Search for a free port has to be implemented in the user-space. If
the chosen 4-tuple happens to be busy, the application needs to retry
from a different local port number.
Detecting if 4-tuple is busy can be either easy (TCP) or hard
(UDP). In TCP case, the application simply has to check if connect()
returned an error (EADDRNOTAVAIL). That is assuming that the local
port sharing was enabled (REUSEADDR) by all the sockets.
# Assume desired local port range is 60_000-60_511
s = socket(AF_INET, SOCK_STREAM)
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
s.bind(("192.0.2.1", 60_000))
s.connect(("1.1.1.1", 53))
# Fails only if 192.0.2.1:60000 -> 1.1.1.1:53 is busy
# Application must retry with another local port
In case of UDP, the network stack allows binding more than one socket
to the same 4-tuple, when local port sharing is enabled
(REUSEADDR). Hence detecting the conflict is much harder and involves
querying sock_diag and toggling the REUSEADDR flag [1].
b) For TCP, bind()-ing to a port within the ephemeral port range means
that no connecting sockets, that is those which leave it to the
network stack to find a free local port at connect() time, can use
the this port.
IOW, the bind hash bucket tb->fastreuse will be 0 or 1, and the port
will be skipped during the free port search at connect() time.
2. Isolate the app in a dedicated netns and use the use the per-netns
ip_local_port_range sysctl to adjust the ephemeral port range bounds.
The per-netns setting affects all sockets, so this approach can be used
only if:
- there is just one egress IP address, or
- the desired egress port range is the same for all egress IP addresses
used by the application.
For TCP, this approach avoids the downsides of (1). Free port search and
4-tuple conflict detection is done by the network stack:
system("sysctl -w net.ipv4.ip_local_port_range='60000 60511'")
s = socket(AF_INET, SOCK_STREAM)
s.setsockopt(SOL_IP, IP_BIND_ADDRESS_NO_PORT, 1)
s.bind(("192.0.2.1", 0))
s.connect(("1.1.1.1", 53))
# Fails if all 4-tuples 192.0.2.1:60000-60511 -> 1.1.1.1:53 are busy
For UDP this approach has limited applicability. Setting the
IP_BIND_ADDRESS_NO_PORT socket option does not result in local source
port being shared with other connected UDP sockets.
Hence relying on the network stack to find a free source port, limits the
number of outgoing UDP flows from a single IP address down to the number
of available ephemeral ports.
To put it another way, partitioning the ephemeral port range between hosts
using the existing Linux networking API is cumbersome.
To address this use case, add a new socket option at the SOL_IP level,
named IP_LOCAL_PORT_RANGE. The new option can be used to clamp down the
ephemeral port range for each socket individually.
The option can be used only to narrow down the per-netns local port
range. If the per-socket range lies outside of the per-netns range, the
latter takes precedence.
UAPI-wise, the low and high range bounds are passed to the kernel as a pair
of u16 values in host byte order packed into a u32. This avoids pointer
passing.
PORT_LO = 40_000
PORT_HI = 40_511
s = socket(AF_INET, SOCK_STREAM)
v = struct.pack("I", PORT_HI << 16 | PORT_LO)
s.setsockopt(SOL_IP, IP_LOCAL_PORT_RANGE, v)
s.bind(("127.0.0.1", 0))
s.getsockname()
# Local address between ("127.0.0.1", 40_000) and ("127.0.0.1", 40_511),
# if there is a free port. EADDRINUSE otherwise.
[1] https://github.com/cloudflare/cloudflare-blog/blob/232b432c1d57/2022-02-connectx/connectx.py#L116
Reviewed-by: Marek Majkowski <marek@cloudflare.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-01-24 14:36:43 +01:00
|
|
|
|
|
|
|
|
|
2007-10-10 17:30:46 -07:00
|
|
|
|
2014-05-15 13:43:14 -04:00
|
|
|
|
2019-11-26 14:44:16 -08:00
|
|
|
|
2010-05-05 00:27:06 +00:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2019-11-22 13:50:52 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2010-05-05 00:27:06 +00:00
|
|
|
|
2014-07-25 15:25:08 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-11-25 15:37:04 -08:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2022-07-18 10:26:42 -07:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
|
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2019-11-26 14:44:16 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2019-11-22 13:50:52 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2019-11-25 15:37:04 -08:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2019-11-25 15:37:04 -08:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2010-05-05 00:27:06 +00:00
|
|
|
|
2016-02-27 00:32:15 -08:00
|
|
|
|
|
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2008-07-15 16:00:59 -04:00
|
|
|
|
2014-05-13 10:17:33 -07:00
|
|
|
|
2022-07-13 13:51:57 -07:00
|
|
|
|
2014-05-13 10:17:33 -07:00
|
|
|
|
2011-06-23 21:28:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-11-14 21:42:26 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-14 10:21:14 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-09-25 07:39:14 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-09-25 07:39:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-14 10:21:14 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2014-02-26 01:20:42 +01:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-04 16:39:18 -07:00
|
|
|
|
2014-02-26 01:20:42 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-07-20 23:06:28 +03:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2020-09-23 13:18:15 -07:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2022-07-13 13:51:53 -07:00
|
|
|
|
2018-03-14 10:21:14 +01:00
|
|
|
|
2021-07-20 23:06:28 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2020-09-23 13:18:15 -07:00
|
|
|
|
|
|
|
|
|
2021-07-20 23:06:28 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-09-23 13:18:15 -07:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2016-06-29 21:47:03 +03:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
|
|
|
|
|
2015-10-04 21:08:08 -07:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2015-10-04 21:08:08 -07:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2015-10-04 21:08:08 -07:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2018-10-04 20:07:51 -07:00
|
|
|
|
2018-11-06 12:51:15 -08:00
|
|
|
|
|
|
|
|
|
2018-10-04 20:07:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-17 17:33:07 -07:00
|
|
|
|
2018-10-04 20:07:53 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-10-04 20:07:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-09-18 15:29:53 -07:00
|
|
|
|
|
|
|
|
|
2022-01-26 17:10:22 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-04 16:39:18 -07:00
|
|
|
|
2022-01-26 17:10:22 -08:00
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
2022-01-26 17:10:22 -08:00
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
|
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-05-02 16:29:38 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-06-04 09:16:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-15 22:24:00 +08:00
|
|
|
|
2015-06-04 09:16:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-28 22:40:53 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-10 09:48:31 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-10-15 06:30:45 +00:00
|
|
|
|
2011-12-10 09:48:31 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-03 15:42:29 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-08-27 16:06:59 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-01 12:52:30 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2008-01-01 21:13:09 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-05-24 14:33:03 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-05-24 14:33:03 -07:00
|
|
|
|
2009-12-15 16:59:59 +01:00
|
|
|
|
2010-05-24 14:33:03 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2011-07-05 01:05:48 -07:00
|
|
|
|
|
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2015-05-15 14:15:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-10-09 13:44:54 -05:00
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
2015-10-09 13:44:54 -05:00
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
2015-10-09 13:44:54 -05:00
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2022-01-28 08:06:54 -08:00
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
2017-08-03 18:07:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
2017-08-03 18:07:06 +02:00
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2019-02-25 19:27:15 +03:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-07-23 08:08:57 +02:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
2019-04-01 09:17:32 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-10-07 11:01:40 -05:00
|
|
|
|
2016-11-04 11:28:58 +01:00
|
|
|
|
|
|
|
|
|
2016-04-02 23:08:10 -04:00
|
|
|
|
2014-02-18 21:38:08 +01:00
|
|
|
|
2021-10-25 09:48:24 -07:00
|
|
|
|
2022-08-16 23:18:26 -07:00
|
|
|
|
|
|
|
|
|
2020-07-23 08:09:07 +02:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2022-09-01 17:29:25 -07:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-23 00:46:12 +01:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-01-05 13:56:16 -08:00
|
|
|
|
|
|
|
|
|
2016-11-04 11:28:58 +01:00
|
|
|
|
2015-01-05 13:56:16 -08:00
|
|
|
|
|
|
|
|
|
2014-09-19 07:38:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
|
|
|
|
|
2019-02-27 16:15:29 +08:00
|
|
|
|
2018-05-22 14:03:27 -07:00
|
|
|
|
|
|
|
|
|
2019-12-05 20:43:46 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-28 07:12:27 +02:00
|
|
|
|
2020-05-28 07:12:29 +02:00
|
|
|
|
2020-05-28 07:12:30 +02:00
|
|
|
|
2020-05-28 07:12:28 +02:00
|
|
|
|
2020-05-28 07:12:26 +02:00
|
|
|
|
2021-11-19 12:41:34 -08:00
|
|
|
|
2020-05-28 07:12:26 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|