2019-05-27 08:55:01 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-05-05 16:16:16 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-03-12 20:09:15 -03:00
|
|
|
|
2017-12-01 12:52:30 -08:00
|
|
|
|
2020-07-23 08:08:57 +02:00
|
|
|
|
2021-10-25 09:48:24 -07:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2013-09-24 15:43:09 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-10-01 07:44:42 -07:00
|
|
|
|
2015-05-12 14:56:07 +02:00
|
|
|
|
2017-12-01 12:52:30 -08:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-04-12 12:34:03 +08:00
|
|
|
|
2017-12-11 07:17:39 -08:00
|
|
|
|
2017-04-12 12:34:03 +08:00
|
|
|
|
2019-03-20 09:18:59 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2016-05-10 11:19:51 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2016-10-16 20:02:52 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-01-23 12:01:26 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-05-22 16:32:51 +02:00
|
|
|
|
2016-11-02 16:36:17 -04:00
|
|
|
|
2022-05-13 23:34:02 +03:00
|
|
|
|
2012-08-26 19:13:55 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2016-10-16 20:02:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-03-12 20:09:15 -03:00
|
|
|
|
|
|
|
|
|
2007-04-20 22:47:35 -07:00
|
|
|
|
2007-03-12 20:09:15 -03:00
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2016-04-02 23:08:10 -04:00
|
|
|
|
2006-09-27 18:28:28 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-04-21 09:45:37 +00:00
|
|
|
|
2013-09-24 15:43:08 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
udp: generate gso with UDP_SEGMENT
Support generic segmentation offload for udp datagrams. Callers can
concatenate and send at once the payload of multiple datagrams with
the same destination.
To set segment size, the caller sets socket option UDP_SEGMENT to the
length of each discrete payload. This value must be smaller than or
equal to the relevant MTU.
A follow-up patch adds cmsg UDP_SEGMENT to specify segment size on a
per send call basis.
Total byte length may then exceed MTU. If not an exact multiple of
segment size, the last segment will be shorter.
The implementation adds a gso_size field to the udp socket, ip(v6)
cmsg cookie and inet_cork structure to be able to set the value at
setsockopt or cmsg time and to work with both lockless and corked
paths.
Initial benchmark numbers show UDP GSO about as expensive as TCP GSO.
tcp tso
3197 MB/s 54232 msg/s 54232 calls/s
6,457,754,262 cycles
tcp gso
1765 MB/s 29939 msg/s 29939 calls/s
11,203,021,806 cycles
tcp without tso/gso *
739 MB/s 12548 msg/s 12548 calls/s
11,205,483,630 cycles
udp
876 MB/s 14873 msg/s 624666 calls/s
11,205,777,429 cycles
udp gso
2139 MB/s 36282 msg/s 36282 calls/s
11,204,374,561 cycles
[*] after reverting commit 0a6b2a1dc2a2
("tcp: switch to GSO being always on")
Measured total system cycles ('-a') for one core while pinning both
the network receive path and benchmark process to that core:
perf stat -a -C 12 -e cycles \
./udpgso_bench_tx -C 12 -4 -D "$DST" -l 4
Note the reduction in calls/s with GSO. Bytes per syscall drops
increases from 1470 to 61818.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-26 13:42:17 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2018-07-06 10:12:54 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-11 15:50:51 -04:00
|
|
|
|
2018-07-06 10:12:54 -04:00
|
|
|
|
2022-05-13 11:55:41 -07:00
|
|
|
|
2018-07-06 10:12:54 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-01-20 03:43:08 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2017-08-07 08:44:16 -07:00
|
|
|
|
2020-11-09 15:13:48 -08:00
|
|
|
|
2017-08-07 08:44:16 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-22 12:45:32 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2010-10-25 03:32:44 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-09 16:21:07 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-07 03:12:08 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-10 16:09:45 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-09-25 07:39:16 -07:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2020-09-09 17:50:47 -07:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
2018-07-02 16:14:12 +01:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2018-11-07 12:38:31 +01:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2015-10-07 16:48:47 -05:00
|
|
|
|
|
|
|
|
|
2015-06-12 21:55:31 -05:00
|
|
|
|
|
|
|
|
|
2019-05-29 13:25:31 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-05-29 13:25:33 +02:00
|
|
|
|
2019-10-19 09:26:37 -07:00
|
|
|
|
2019-05-29 13:25:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-10-19 09:26:37 -07:00
|
|
|
|
2019-05-29 13:25:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2015-10-07 16:48:45 -05:00
|
|
|
|
2015-10-07 16:48:46 -05:00
|
|
|
|
2014-04-15 13:47:15 -04:00
|
|
|
|
2018-07-02 18:21:11 +08:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-26 13:42:15 -04:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2020-06-19 12:12:34 -07:00
|
|
|
|
2018-07-02 18:21:11 +08:00
|
|
|
|
2011-05-08 17:12:19 -07:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2011-05-08 17:12:19 -07:00
|
|
|
|
2011-03-01 02:36:47 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-09-24 15:43:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-07-14 08:10:22 +02:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2013-01-21 02:00:03 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2008-10-01 07:41:00 -07:00
|
|
|
|
2006-11-14 21:26:08 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2007-06-04 21:32:46 -07:00
|
|
|
|
2011-10-24 03:06:21 -04:00
|
|
|
|
2016-11-04 02:23:43 +09:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-10-01 07:41:00 -07:00
|
|
|
|
|
|
|
|
|
2008-10-01 07:44:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-01-29 21:35:05 -08:00
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-13 21:22:35 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2008-07-18 04:03:08 -07:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2014-03-06 15:03:17 -05:00
|
|
|
|
2016-04-27 16:44:43 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2021-09-29 18:03:32 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-05 15:55:55 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2015-08-30 11:29:41 +05:30
|
|
|
|
|
|
|
|
|
2014-05-05 15:55:55 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
2015-08-30 11:29:41 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-05 15:55:55 -07:00
|
|
|
|
2010-06-30 13:31:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-04-20 15:57:15 -07:00
|
|
|
|
2016-09-30 11:28:58 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-28 14:10:59 -07:00
|
|
|
|
2007-10-10 17:30:46 -07:00
|
|
|
|
2014-05-15 13:43:14 -04:00
|
|
|
|
2019-11-26 14:44:16 -08:00
|
|
|
|
2010-05-05 00:27:06 +00:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2019-11-22 13:50:52 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2010-05-05 00:27:06 +00:00
|
|
|
|
2014-07-25 15:25:08 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-11-25 15:37:04 -08:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2022-07-18 10:26:42 -07:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
|
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2019-11-26 14:44:16 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2019-11-22 13:50:52 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2019-11-25 15:37:04 -08:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2019-11-25 15:37:04 -08:00
|
|
|
|
2017-01-20 17:49:11 -08:00
|
|
|
|
2014-05-12 16:04:53 -07:00
|
|
|
|
2010-05-05 00:27:06 +00:00
|
|
|
|
2016-02-27 00:32:15 -08:00
|
|
|
|
|
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2008-07-15 16:00:59 -04:00
|
|
|
|
2014-05-13 10:17:33 -07:00
|
|
|
|
2022-07-13 13:51:57 -07:00
|
|
|
|
2014-05-13 10:17:33 -07:00
|
|
|
|
2011-06-23 21:28:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2005-12-27 02:43:12 -02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-11-14 21:42:26 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-14 10:21:14 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-09-25 07:39:14 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-09-25 07:39:14 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-03-14 10:21:14 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2014-02-26 01:20:42 +01:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-04 16:39:18 -07:00
|
|
|
|
2014-02-26 01:20:42 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-07-20 23:06:28 +03:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2020-09-23 13:18:15 -07:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2022-07-13 13:51:53 -07:00
|
|
|
|
2018-03-14 10:21:14 +01:00
|
|
|
|
2021-07-20 23:06:28 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2020-09-23 13:18:15 -07:00
|
|
|
|
|
|
|
|
|
2021-07-20 23:06:28 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-09-23 13:18:15 -07:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2016-06-29 21:47:03 +03:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
|
|
|
|
|
2015-10-04 21:08:08 -07:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
2015-10-04 21:08:08 -07:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2015-10-04 21:08:08 -07:00
|
|
|
|
2021-06-25 19:21:39 +03:00
|
|
|
|
|
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
2018-10-04 20:07:51 -07:00
|
|
|
|
2018-11-06 12:51:15 -08:00
|
|
|
|
|
|
|
|
|
2018-10-04 20:07:52 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-04-17 17:33:07 -07:00
|
|
|
|
2018-10-04 20:07:53 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-10-04 20:07:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-09-18 15:29:53 -07:00
|
|
|
|
|
|
|
|
|
2022-01-26 17:10:22 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-05-04 16:39:18 -07:00
|
|
|
|
2022-01-26 17:10:22 -08:00
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
2022-01-26 17:10:22 -08:00
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
|
|
|
|
|
inetpeer: get rid of ip_id_count
Ideally, we would need to generate IP ID using a per destination IP
generator.
linux kernels used inet_peer cache for this purpose, but this had a huge
cost on servers disabling MTU discovery.
1) each inet_peer struct consumes 192 bytes
2) inetpeer cache uses a binary tree of inet_peer structs,
with a nominal size of ~66000 elements under load.
3) lookups in this tree are hitting a lot of cache lines, as tree depth
is about 20.
4) If server deals with many tcp flows, we have a high probability of
not finding the inet_peer, allocating a fresh one, inserting it in
the tree with same initial ip_id_count, (cf secure_ip_id())
5) We garbage collect inet_peer aggressively.
IP ID generation do not have to be 'perfect'
Goal is trying to avoid duplicates in a short period of time,
so that reassembly units have a chance to complete reassembly of
fragments belonging to one message before receiving other fragments
with a recycled ID.
We simply use an array of generators, and a Jenkin hash using the dst IP
as a key.
ipv6_select_ident() is put back into net/ipv6/ip6_output.c where it
belongs (it is only used from this file)
secure_ip_id() and secure_ipv6_id() no longer are needed.
Rename ip_select_ident_more() to ip_select_ident_segs() to avoid
unnecessary decrement/increment of the number of segments.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-06-02 05:26:03 -07:00
|
|
|
|
2015-03-25 17:07:44 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-05-02 16:29:38 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-06-04 09:16:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-15 22:24:00 +08:00
|
|
|
|
2015-06-04 09:16:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-11-14 20:51:49 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2007-12-10 13:38:41 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-28 22:40:53 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-10 09:48:31 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-10-15 06:30:45 +00:00
|
|
|
|
2011-12-10 09:48:31 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-03 15:42:29 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-08-27 16:06:59 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-12-01 12:52:30 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2008-01-01 21:13:09 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-05-24 14:33:03 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-05-24 14:33:03 -07:00
|
|
|
|
2009-12-15 16:59:59 +01:00
|
|
|
|
2010-05-24 14:33:03 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2011-07-05 01:05:48 -07:00
|
|
|
|
|
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2015-05-15 14:15:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-10-09 13:44:54 -05:00
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
2015-10-09 13:44:54 -05:00
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
2015-10-09 13:44:54 -05:00
|
|
|
|
2011-10-06 10:28:31 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-02-27 15:48:21 -08:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2022-01-28 08:06:54 -08:00
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
2017-08-03 18:07:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
2017-08-03 18:07:06 +02:00
|
|
|
|
2014-09-27 09:50:55 -07:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2019-02-25 19:27:15 +03:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-07-23 08:08:57 +02:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
2019-04-01 09:17:32 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-10-07 11:01:40 -05:00
|
|
|
|
2016-11-04 11:28:58 +01:00
|
|
|
|
|
|
|
|
|
2016-04-02 23:08:10 -04:00
|
|
|
|
2014-02-18 21:38:08 +01:00
|
|
|
|
2021-10-25 09:48:24 -07:00
|
|
|
|
2022-08-16 23:18:26 -07:00
|
|
|
|
|
|
|
|
|
2020-07-23 08:09:07 +02:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2022-09-01 17:29:25 -07:00
|
|
|
|
|
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-23 00:46:12 +01:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2015-01-05 13:56:16 -08:00
|
|
|
|
|
|
|
|
|
2016-11-04 11:28:58 +01:00
|
|
|
|
2015-01-05 13:56:16 -08:00
|
|
|
|
|
|
|
|
|
2014-09-19 07:38:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
2013-09-21 10:22:42 -07:00
|
|
|
|
2005-08-16 02:18:02 -03:00
|
|
|
|
|
|
|
|
|
2019-02-27 16:15:29 +08:00
|
|
|
|
2018-05-22 14:03:27 -07:00
|
|
|
|
|
|
|
|
|
2019-12-05 20:43:46 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-05-28 07:12:27 +02:00
|
|
|
|
2020-05-28 07:12:29 +02:00
|
|
|
|
2020-05-28 07:12:30 +02:00
|
|
|
|
2020-05-28 07:12:28 +02:00
|
|
|
|
2020-05-28 07:12:26 +02:00
|
|
|
|
2021-11-19 12:41:34 -08:00
|
|
|
|
2020-05-28 07:12:26 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|