2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-05-05 16:16:16 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-10-01 07:35:39 -07:00
|
|
|
|
2015-09-30 10:12:22 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2012-07-26 11:14:38 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-08-04 23:12:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-01-09 10:01:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-24 15:43:09 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
net: Implement read-only protection and COW'ing of metrics.
Routing metrics are now copy-on-write.
Initially a route entry points it's metrics at a read-only location.
If a routing table entry exists, it will point there. Else it will
point at the all zero metric place-holder called 'dst_default_metrics'.
The writeability state of the metrics is stored in the low bits of the
metrics pointer, we have two bits left to spare if we want to store
more states.
For the initial implementation, COW is implemented simply via kmalloc.
However future enhancements will change this to place the writable
metrics somewhere else, in order to increase sharing. Very likely
this "somewhere else" will be the inetpeer cache.
Note also that this means that metrics updates may transiently fail
if we cannot COW the metrics successfully.
But even by itself, this patch should decrease memory usage and
increase cache locality especially for routing workloads. In those
cases the read-only metric copies stay in place and never get written
to.
TCP workloads where metrics get updated, and those rare cases where
PMTU triggers occur, will take a very slight performance hit. But
that hit will be alleviated when the long-term writable metrics
move to a more sharable location.
Since the metrics storage went from a u32 array of RTAX_MAX entries to
what is essentially a pointer, some retooling of the dst_entry layout
was necessary.
Most importantly, we need to preserve the alignment of the reference
count so that it doesn't share cache lines with the read-mostly state,
as per Eric Dumazet's alignment assertion checks.
The only non-trivial bit here is the move of the 'flags' member into
the writeable cacheline. This is OK since we are always accessing the
flags around the same moment when we made a modification to the
reference count.
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-26 20:51:05 -08:00
|
|
|
|
2015-01-14 15:17:06 -08:00
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2010-06-10 23:31:35 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
[IPV4] route cache: Introduce rt_genid for smooth cache invalidation
Current ip route cache implementation is not suited to large caches.
We can consume a lot of CPU when cache must be invalidated, since we
currently need to evict all cache entries, and this eviction is
sometimes asynchronous. min_delay & max_delay can somewhat control this
asynchronism behavior, but whole thing is a kludge, regularly triggering
infamous soft lockup messages. When entries are still in use, this also
consumes a lot of ram, filling dst_garbage.list.
A better scheme is to use a generation identifier on each entry,
so that cache invalidation can be performed by changing the table
identifier, without having to scan all entries.
No more delayed flushing, no more stalling when secret_interval expires.
Invalidated entries will then be freed at GC time (controled by
ip_rt_gc_timeout or stress), or when an invalidated entry is found
in a chain when an insert is done.
Thus we keep a normal equilibrium.
This patch :
- renames rt_hash_rnd to rt_genid (and makes it an atomic_t)
- Adds a new rt_genid field to 'struct rtable' (filling a hole on 64bit)
- Checks entry->rt_genid at appropriate places :
2008-01-31 17:05:09 -08:00
|
|
|
|
2012-04-15 05:58:06 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-10-08 11:41:18 +00:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-09-26 21:26:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2018-03-14 10:21:14 +01:00
|
|
|
|
|
|
|
|
|
2012-07-31 15:06:50 -07:00
|
|
|
|
2015-09-02 13:58:35 -07:00
|
|
|
|
|
|
|
|
|
2012-07-31 15:06:50 -07:00
|
|
|
|
2015-01-14 15:17:06 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2011-11-23 02:14:15 +00:00
|
|
|
|
2010-11-11 17:07:48 -08:00
|
|
|
|
2012-07-17 14:44:26 -07:00
|
|
|
|
2010-11-11 17:07:48 -08:00
|
|
|
|
|
|
|
|
|
2011-11-23 02:14:15 +00:00
|
|
|
|
2010-11-11 17:07:48 -08:00
|
|
|
|
2012-07-17 14:44:26 -07:00
|
|
|
|
2010-11-11 17:07:48 -08:00
|
|
|
|
|
|
|
|
|
2012-07-13 05:03:45 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-03 03:26:03 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-02-16 15:20:26 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-05-25 10:42:33 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-09-30 10:12:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-05-25 10:42:33 -07:00
|
|
|
|
2015-09-30 10:12:22 +02:00
|
|
|
|
|
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
2015-09-25 07:39:10 -07:00
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
|
|
|
|
|
2010-05-10 11:32:55 +00:00
|
|
|
|
2011-03-12 01:12:47 -05:00
|
|
|
|
2011-03-02 14:56:30 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-12 01:12:47 -05:00
|
|
|
|
|
|
|
|
|
2012-06-10 20:05:24 +00:00
|
|
|
|
2011-03-12 01:12:47 -05:00
|
|
|
|
|
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
2011-03-12 01:12:47 -05:00
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
|
|
|
|
|
2011-05-03 20:25:42 -07:00
|
|
|
|
|
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-03 20:25:42 -07:00
|
|
|
|
2011-03-31 04:52:59 -07:00
|
|
|
|
|
|
|
|
|
2016-11-04 02:23:43 +09:00
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
2011-05-03 20:25:42 -07:00
|
|
|
|
|
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
|
|
|
|
|
2011-05-04 12:33:34 -07:00
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-04 12:33:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-12 00:00:52 -05:00
|
|
|
|
2017-09-28 15:51:37 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
|
|
|
|
|
2017-05-25 10:42:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-26 11:14:38 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-08-31 18:11:41 +02:00
|
|
|
|
2017-09-21 09:15:46 -07:00
|
|
|
|
2017-08-31 18:11:41 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-26 11:14:38 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-05-10 11:32:55 +00:00
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-09-01 14:26:35 -06:00
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
|
|
|
|
|
2015-08-13 14:59:05 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
2017-07-01 08:03:10 -04:00
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
2016-04-07 11:10:06 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2005-11-22 14:47:37 -08:00
|
|
|
|
2013-09-22 10:32:22 -07:00
|
|
|
|
|
|
|
|
|
2005-11-22 14:47:37 -08:00
|
|
|
|
xfrm: reuse uncached_list to track xdsts
In early time, when freeing a xdst, it would be inserted into
dst_garbage.list first. Then if it's refcnt was still held
somewhere, later it would be put into dst_busy_list in
dst_gc_task().
When one dev was being unregistered, the dev of these dsts in
dst_busy_list would be set with loopback_dev and put this dev.
So that this dev's removal wouldn't get blocked, and avoid the
kmsg warning:
kernel:unregister_netdevice: waiting for veth0 to become \
free. Usage count = 2
However after Commit 52df157f17e5 ("xfrm: take refcnt of dst
when creating struct xfrm_dst bundle"), the xdst will not be
freed with dst gc, and this warning happens.
To fix it, we need to find these xdsts that are still held by
others when removing the dev, and free xdst's dev and set it
with loopback_dev.
But unfortunately after flow_cache for xfrm was deleted, no
list tracks them anymore. So we need to save these xdsts
somewhere to release the xdst's dev later.
To make this easier, this patch is to reuse uncached_list to
track xdsts, so that the dev refcnt can be released in the
event NETDEV_UNREGISTER process of fib_netdev_notifier.
Thanks to Florian, we could move forward this fix quickly.
Fixes: 52df157f17e5 ("xfrm: take refcnt of dst when creating struct xfrm_dst bundle")
Reported-by: Jianlin Shi <jishi@redhat.com>
Reported-by: Hangbin Liu <liuhangbin@gmail.com>
Tested-by: Eyal Birger <eyal.birger@gmail.com>
Signed-off-by: Xin Long <lucien.xin@gmail.com>
Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
2018-02-14 19:06:02 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-28 22:33:23 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-10-28 22:33:23 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-07-09 15:32:57 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-28 08:04:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
2008-10-01 07:35:39 -07:00
|
|
|
|
|
|
|
|
|
2011-03-31 04:52:59 -07:00
|
|
|
|
|
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
2016-11-04 02:23:43 +09:00
|
|
|
|
|
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-28 08:04:14 +02:00
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-28 08:04:14 +02:00
|
|
|
|
2008-10-01 07:35:39 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
2011-03-02 14:31:35 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-02-04 13:04:46 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-02-24 13:38:12 -08:00
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
|
|
|
|
|
2011-03-02 14:31:35 -08:00
|
|
|
|
2012-02-04 13:04:46 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-04-26 13:28:44 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-03-02 14:31:35 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2008-10-01 07:33:10 -07:00
|
|
|
|
|
|
|
|
|
2016-04-05 08:22:49 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-23 13:57:45 -07:00
|
|
|
|
|
|
|
|
|
2008-10-01 07:33:10 -07:00
|
|
|
|
|
|
|
|
|
2010-12-12 21:55:08 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-15 12:11:27 +02:00
|
|
|
|
2010-12-12 21:55:08 -08:00
|
|
|
|
|
|
|
|
|
2016-02-15 12:11:27 +02:00
|
|
|
|
2010-12-12 21:55:08 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|