2005-04-16 15:20:36 -07:00
|
|
|
|
2010-09-10 16:51:36 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-09-10 16:51:36 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-09-10 16:51:36 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-09-10 16:51:36 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-09-10 16:51:36 +02:00
|
|
|
|
|
|
|
|
|
2005-10-30 15:01:59 -08:00
|
|
|
|
2010-09-10 16:51:36 +02:00
|
|
|
|
|
|
|
|
|
2013-08-21 08:50:39 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-09-10 16:51:36 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2011-05-23 14:51:41 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-02-23 12:43:43 -06:00
|
|
|
|
2006-10-11 01:21:26 -07:00
|
|
|
|
2006-12-06 20:34:49 -08:00
|
|
|
|
2006-12-06 20:37:26 -08:00
|
|
|
|
|
|
|
|
|
2007-10-18 23:39:55 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2012-12-17 10:01:23 -05:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-04-30 15:27:22 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-18 14:05:55 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:15 +02:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-02-16 18:10:19 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-13 22:16:45 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2010-08-24 14:22:47 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-01-18 14:05:55 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: reimplement idle worker rebinding
Currently rebind_workers() uses rebinds idle workers synchronously
before proceeding to requesting busy workers to rebind. This is
necessary because all workers on @worker_pool->idle_list must be bound
before concurrency management local wake-ups from the busy workers
take place.
Unfortunately, the synchronous idle rebinding is quite complicated.
This patch reimplements idle rebinding to simplify the code path.
Rather than trying to make all idle workers bound before rebinding
busy workers, we simply remove all to-be-bound idle workers from the
idle list and let them add themselves back after completing rebinding
(successful or not).
As only workers which finished rebinding can on on the idle worker
list, the idle worker list is guaranteed to have only bound workers
unless CPU went down again and local wake-ups are safe.
After the change, @worker_pool->nr_idle may deviate than the actual
number of idle workers on @worker_pool->idle_list. More specifically,
nr_idle may be non-zero while ->idle_list is empty. All users of
->nr_idle and ->idle_list are audited. The only affected one is
too_many_workers() which is updated to check %false if ->idle_list is
empty regardless of ->nr_idle.
After this patch, rebind_workers() no longer performs the nasty
idle-rebind retries which require temporary release of gcwq->lock, and
both unbinding and rebinding are atomic w.r.t. global_cwq->lock.
worker->idle_rebind and global_cwq->rebind_hold are now unnecessary
and removed along with the definition of struct idle_rebind.
Changed from V1:
1) remove unlikely from too_many_workers(), ->idle_list can be empty
anytime, even before this patch, no reason to use unlikely.
2) fix a small rebasing mistake.
(which is from rebasing the orignal fixing patch to for-next)
3) add a lot of comments.
4) clear WORKER_REBIND unconditionaly in idle_worker_rebind()
tj: Updated comments and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2012-09-18 09:59:22 -07:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-18 23:39:55 -07:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2007-10-18 23:39:55 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-08 16:45:40 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
2013-09-05 12:30:04 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-05-06 17:44:55 -04:00
|
|
|
|
2012-08-19 00:52:42 +03:00
|
|
|
|
2012-08-15 23:25:39 +09:00
|
|
|
|
2012-08-19 00:52:42 +03:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-08-19 00:52:42 +03:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2012-08-19 00:52:42 +03:00
|
|
|
|
2011-02-21 09:52:50 +01:00
|
|
|
|
2013-04-24 17:12:54 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-10-05 10:41:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-03-20 03:28:21 +08:00
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-07 09:58:33 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-07 09:58:33 +01:00
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-10 09:52:35 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-09-10 09:52:35 +08:00
|
|
|
|
|
|
|
|
|
Linux 3.9-rc5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iQEcBAABAgAGBQJRWLTrAAoJEHm+PkMAQRiGe8oH/iMy48mecVWvxVZn74Tx3Cef
xmW/PnAIj28EhSPqK49N/Ow6AfQToFKf7AP0ge20KAf5teTq95AY+tH74DAANt8F
BjKXXTZiR5xwBvRkq7CR5wDcCvEcBAAz8fgTEd6SEDB2d2VXFf5eKdKUqt1avTCh
Z6Hup5kuwX+ddtwY2DCBXtp2n6fL0Rm5yLzY1A3OOBye1E7VyLTF7M5BR603Q44P
4kRLxn8+R7jy3hTuZIhAeoS8TKUoBwVk7DmKxEzrhTHZVOmvwE9lEHybRnIyOpd/
k1JnbRbiPsLsCVFOn10SQkGDAIk00lro3tuWP2C1ljERiD/OOh5Ui9nXYAhMkbI=
=q15K
-----END PGP SIGNATURE-----
Merge tag 'v3.9-rc5' into wq/for-3.10
Writeback conversion to workqueue will be based on top of wq/for-3.10
branch to take advantage of custom attrs and NUMA support for unbound
workqueues. Mainline currently contains two commits which result in
non-trivial merge conflicts with wq/for-3.10 and because
block/for-3.10/core is based on v3.9-rc3 which contains one of the
conflicting commits, we need a pre-merge-window merge anyway. Let's
pull v3.9-rc5 into wq/for-3.10 so that the block tree doesn't suffer
from workqueue merge conflicts.
The two conflicts and their resolutions:
* e68035fb65 ("workqueue: convert to idr_alloc()") in mainline changes
worker_pool_assign_id() to use idr_alloc() instead of the old idr
interface. worker_pool_assign_id() goes through multiple locking
changes in wq/for-3.10 causing the following conflict.
static int worker_pool_assign_id(struct worker_pool *pool)
{
int ret;
<<<<<<< HEAD
lockdep_assert_held(&wq_pool_mutex);
do {
if (!idr_pre_get(&worker_pool_idr, GFP_KERNEL))
return -ENOMEM;
ret = idr_get_new(&worker_pool_idr, pool, &pool->id);
} while (ret == -EAGAIN);
=======
mutex_lock(&worker_pool_idr_mutex);
ret = idr_alloc(&worker_pool_idr, pool, 0, 0, GFP_KERNEL);
if (ret >= 0)
pool->id = ret;
mutex_unlock(&worker_pool_idr_mutex);
>>>>>>> c67bf5361e7e66a0ff1f4caf95f89347d55dfb89
return ret < 0 ? ret : 0;
}
We want locking from the former and idr_alloc() usage from the
latter, which can be combined to the following.
static int worker_pool_assign_id(struct worker_pool *pool)
{
int ret;
lockdep_assert_held(&wq_pool_mutex);
ret = idr_alloc(&worker_pool_idr, pool, 0, 0, GFP_KERNEL);
if (ret >= 0) {
pool->id = ret;
return 0;
}
return ret;
}
* eb2834285c ("workqueue: fix possible pool stall bug in
wq_unbind_fn()") updated wq_unbind_fn() such that it has single
larger for_each_std_worker_pool() loop instead of two separate loops
with a schedule() call inbetween. wq/for-3.10 renamed
pool->assoc_mutex to pool->manager_mutex causing the following
conflict (earlier function body and comments omitted for brevity).
static void wq_unbind_fn(struct work_struct *work)
{
...
spin_unlock_irq(&pool->lock);
<<<<<<< HEAD
mutex_unlock(&pool->manager_mutex);
}
=======
mutex_unlock(&pool->assoc_mutex);
>>>>>>> c67bf5361e7e66a0ff1f4caf95f89347d55dfb89
schedule();
<<<<<<< HEAD
for_each_cpu_worker_pool(pool, cpu)
=======
>>>>>>> c67bf5361e7e66a0ff1f4caf95f89347d55dfb89
atomic_set(&pool->nr_running, 0);
spin_lock_irq(&pool->lock);
wake_up_worker(pool);
spin_unlock_irq(&pool->lock);
}
}
The resolution is mostly trivial. We want the control flow of the
latter with the rename of the former.
static void wq_unbind_fn(struct work_struct *work)
{
...
spin_unlock_irq(&pool->lock);
mutex_unlock(&pool->manager_mutex);
schedule();
atomic_set(&pool->nr_running, 0);
spin_lock_irq(&pool->lock);
wake_up_worker(pool);
spin_unlock_irq(&pool->lock);
}
}
Signed-off-by: Tejun Heo <tj@kernel.org>
2013-04-01 17:08:13 -07:00
|
|
|
|
2013-03-13 14:59:38 -07:00
|
|
|
|
Linux 3.9-rc5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iQEcBAABAgAGBQJRWLTrAAoJEHm+PkMAQRiGe8oH/iMy48mecVWvxVZn74Tx3Cef
xmW/PnAIj28EhSPqK49N/Ow6AfQToFKf7AP0ge20KAf5teTq95AY+tH74DAANt8F
BjKXXTZiR5xwBvRkq7CR5wDcCvEcBAAz8fgTEd6SEDB2d2VXFf5eKdKUqt1avTCh
Z6Hup5kuwX+ddtwY2DCBXtp2n6fL0Rm5yLzY1A3OOBye1E7VyLTF7M5BR603Q44P
4kRLxn8+R7jy3hTuZIhAeoS8TKUoBwVk7DmKxEzrhTHZVOmvwE9lEHybRnIyOpd/
k1JnbRbiPsLsCVFOn10SQkGDAIk00lro3tuWP2C1ljERiD/OOh5Ui9nXYAhMkbI=
=q15K
-----END PGP SIGNATURE-----
Merge tag 'v3.9-rc5' into wq/for-3.10
Writeback conversion to workqueue will be based on top of wq/for-3.10
branch to take advantage of custom attrs and NUMA support for unbound
workqueues. Mainline currently contains two commits which result in
non-trivial merge conflicts with wq/for-3.10 and because
block/for-3.10/core is based on v3.9-rc3 which contains one of the
conflicting commits, we need a pre-merge-window merge anyway. Let's
pull v3.9-rc5 into wq/for-3.10 so that the block tree doesn't suffer
from workqueue merge conflicts.
The two conflicts and their resolutions:
* e68035fb65 ("workqueue: convert to idr_alloc()") in mainline changes
worker_pool_assign_id() to use idr_alloc() instead of the old idr
interface. worker_pool_assign_id() goes through multiple locking
changes in wq/for-3.10 causing the following conflict.
static int worker_pool_assign_id(struct worker_pool *pool)
{
int ret;
<<<<<<< HEAD
lockdep_assert_held(&wq_pool_mutex);
do {
if (!idr_pre_get(&worker_pool_idr, GFP_KERNEL))
return -ENOMEM;
ret = idr_get_new(&worker_pool_idr, pool, &pool->id);
} while (ret == -EAGAIN);
=======
mutex_lock(&worker_pool_idr_mutex);
ret = idr_alloc(&worker_pool_idr, pool, 0, 0, GFP_KERNEL);
if (ret >= 0)
pool->id = ret;
mutex_unlock(&worker_pool_idr_mutex);
>>>>>>> c67bf5361e7e66a0ff1f4caf95f89347d55dfb89
return ret < 0 ? ret : 0;
}
We want locking from the former and idr_alloc() usage from the
latter, which can be combined to the following.
static int worker_pool_assign_id(struct worker_pool *pool)
{
int ret;
lockdep_assert_held(&wq_pool_mutex);
ret = idr_alloc(&worker_pool_idr, pool, 0, 0, GFP_KERNEL);
if (ret >= 0) {
pool->id = ret;
return 0;
}
return ret;
}
* eb2834285c ("workqueue: fix possible pool stall bug in
wq_unbind_fn()") updated wq_unbind_fn() such that it has single
larger for_each_std_worker_pool() loop instead of two separate loops
with a schedule() call inbetween. wq/for-3.10 renamed
pool->assoc_mutex to pool->manager_mutex causing the following
conflict (earlier function body and comments omitted for brevity).
static void wq_unbind_fn(struct work_struct *work)
{
...
spin_unlock_irq(&pool->lock);
<<<<<<< HEAD
mutex_unlock(&pool->manager_mutex);
}
=======
mutex_unlock(&pool->assoc_mutex);
>>>>>>> c67bf5361e7e66a0ff1f4caf95f89347d55dfb89
schedule();
<<<<<<< HEAD
for_each_cpu_worker_pool(pool, cpu)
=======
>>>>>>> c67bf5361e7e66a0ff1f4caf95f89347d55dfb89
atomic_set(&pool->nr_running, 0);
spin_lock_irq(&pool->lock);
wake_up_worker(pool);
spin_unlock_irq(&pool->lock);
}
}
The resolution is mostly trivial. We want the control flow of the
latter with the rename of the former.
static void wq_unbind_fn(struct work_struct *work)
{
...
spin_unlock_irq(&pool->lock);
mutex_unlock(&pool->manager_mutex);
schedule();
atomic_set(&pool->nr_running, 0);
spin_lock_irq(&pool->lock);
wake_up_worker(pool);
spin_unlock_irq(&pool->lock);
}
}
Signed-off-by: Tejun Heo <tj@kernel.org>
2013-04-01 17:08:13 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2007-05-23 13:57:57 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-05-23 13:57:57 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2006-11-22 14:54:49 +00:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2006-11-22 14:54:49 +00:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2006-11-22 14:54:49 +00:00
|
|
|
|
|
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-08-13 17:08:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2006-01-08 01:05:12 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2007-05-09 02:34:12 -07:00
|
|
|
|
2010-07-22 14:14:25 +02:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-07-22 14:14:25 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-04-23 17:40:40 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2006-11-22 14:54:49 +00:00
|
|
|
|
2010-07-22 14:14:25 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-07 13:14:20 -08:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2013-02-07 13:14:20 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-02-07 13:14:20 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-13 22:16:45 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2007-05-09 02:34:17 -07:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2007-05-09 02:34:17 -07:00
|
|
|
|
|
|
|
|
|
[PATCH] WorkStruct: Use direct assignment rather than cmpxchg()
Use direct assignment rather than cmpxchg() as the latter is unavailable
and unimplementable on some platforms and is actually unnecessary.
The use of cmpxchg() was to guard against two possibilities, neither of
which can actually occur:
(1) The pending flag may have been unset or may be cleared. However, given
where it's called, the pending flag is _always_ set. I don't think it
can be unset whilst we're in set_wq_data().
Once the work is enqueued to be actually run, the only way off the queue
is for it to be actually run.
If it's a delayed work item, then the bit can't be cleared by the timer
because we haven't started the timer yet. Also, the pending bit can't be
cleared by cancelling the delayed work _until_ the work item has had its
timer started.
(2) The workqueue pointer might change. This can only happen in two cases:
(a) The work item has just been queued to actually run, and so we're
protected by the appropriate workqueue spinlock.
(b) A delayed work item is being queued, and so the timer hasn't been
started yet, and so no one else knows about the work item or can
access it (the pending bit protects us).
Besides, set_wq_data() _sets_ the workqueue pointer unconditionally, so
it can be assigned instead.
So, replacing the set_wq_data() with a straight assignment would be okay
in most cases.
The problem is where we end up tangling with test_and_set_bit() emulated
using spinlocks, and even then it's not a problem _provided_
test_and_set_bit() doesn't attempt to modify the word if the bit was
set.
If that's a problem, then a bitops-proofed assignment will be required -
equivalent to atomic_set() vs other atomic_xxx() ops.
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-07 11:33:26 +00:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
[PATCH] WorkStruct: Use direct assignment rather than cmpxchg()
Use direct assignment rather than cmpxchg() as the latter is unavailable
and unimplementable on some platforms and is actually unnecessary.
The use of cmpxchg() was to guard against two possibilities, neither of
which can actually occur:
(1) The pending flag may have been unset or may be cleared. However, given
where it's called, the pending flag is _always_ set. I don't think it
can be unset whilst we're in set_wq_data().
Once the work is enqueued to be actually run, the only way off the queue
is for it to be actually run.
If it's a delayed work item, then the bit can't be cleared by the timer
because we haven't started the timer yet. Also, the pending bit can't be
cleared by cancelling the delayed work _until_ the work item has had its
timer started.
(2) The workqueue pointer might change. This can only happen in two cases:
(a) The work item has just been queued to actually run, and so we're
protected by the appropriate workqueue spinlock.
(b) A delayed work item is being queued, and so the timer hasn't been
started yet, and so no one else knows about the work item or can
access it (the pending bit protects us).
Besides, set_wq_data() _sets_ the workqueue pointer unconditionally, so
it can be assigned instead.
So, replacing the set_wq_data() with a straight assignment would be okay
in most cases.
The problem is where we end up tangling with test_and_set_bit() emulated
using spinlocks, and even then it's not a problem _provided_
test_and_set_bit() doesn't attempt to modify the word if the bit was
set.
If that's a problem, then a bitops-proofed assignment will be required -
equivalent to atomic_set() vs other atomic_xxx() ops.
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-07 11:33:26 +00:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2006-11-22 14:54:49 +00:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
[PATCH] WorkStruct: Use direct assignment rather than cmpxchg()
Use direct assignment rather than cmpxchg() as the latter is unavailable
and unimplementable on some platforms and is actually unnecessary.
The use of cmpxchg() was to guard against two possibilities, neither of
which can actually occur:
(1) The pending flag may have been unset or may be cleared. However, given
where it's called, the pending flag is _always_ set. I don't think it
can be unset whilst we're in set_wq_data().
Once the work is enqueued to be actually run, the only way off the queue
is for it to be actually run.
If it's a delayed work item, then the bit can't be cleared by the timer
because we haven't started the timer yet. Also, the pending bit can't be
cleared by cancelling the delayed work _until_ the work item has had its
timer started.
(2) The workqueue pointer might change. This can only happen in two cases:
(a) The work item has just been queued to actually run, and so we're
protected by the appropriate workqueue spinlock.
(b) A delayed work item is being queued, and so the timer hasn't been
started yet, and so no one else knows about the work item or can
access it (the pending bit protects us).
Besides, set_wq_data() _sets_ the workqueue pointer unconditionally, so
it can be assigned instead.
So, replacing the set_wq_data() with a straight assignment would be okay
in most cases.
The problem is where we end up tangling with test_and_set_bit() emulated
using spinlocks, and even then it's not a problem _provided_
test_and_set_bit() doesn't attempt to modify the word if the bit was
set.
If that's a problem, then a bitops-proofed assignment will be required -
equivalent to atomic_set() vs other atomic_xxx() ops.
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-07 11:33:26 +00:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2006-11-22 14:54:49 +00:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
workqueue: reimplement idle worker rebinding
Currently rebind_workers() uses rebinds idle workers synchronously
before proceeding to requesting busy workers to rebind. This is
necessary because all workers on @worker_pool->idle_list must be bound
before concurrency management local wake-ups from the busy workers
take place.
Unfortunately, the synchronous idle rebinding is quite complicated.
This patch reimplements idle rebinding to simplify the code path.
Rather than trying to make all idle workers bound before rebinding
busy workers, we simply remove all to-be-bound idle workers from the
idle list and let them add themselves back after completing rebinding
(successful or not).
As only workers which finished rebinding can on on the idle worker
list, the idle worker list is guaranteed to have only bound workers
unless CPU went down again and local wake-ups are safe.
After the change, @worker_pool->nr_idle may deviate than the actual
number of idle workers on @worker_pool->idle_list. More specifically,
nr_idle may be non-zero while ->idle_list is empty. All users of
->nr_idle and ->idle_list are audited. The only affected one is
too_many_workers() which is updated to check %false if ->idle_list is
empty regardless of ->nr_idle.
After this patch, rebind_workers() no longer performs the nasty
idle-rebind retries which require temporary release of gcwq->lock, and
both unbinding and rebinding are atomic w.r.t. global_cwq->lock.
worker->idle_rebind and global_cwq->rebind_hold are now unnecessary
and removed along with the definition of struct idle_rebind.
Changed from V1:
1) remove unlikely from too_many_workers(), ->idle_list can be empty
anytime, even before this patch, no reason to use unlikely.
2) fix a small rebasing mistake.
(which is from rebasing the orignal fixing patch to for-next)
3) add a lot of comments.
4) clear WORKER_REBIND unconditionaly in idle_worker_rebind()
tj: Updated comments and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2012-09-18 09:59:22 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2006-11-22 14:54:49 +00:00
|
|
|
|
|
|
|
|
|
2010-04-23 17:40:40 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-26 23:03:49 +09:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2012-10-26 23:03:49 +09:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: It is likely that WORKER_NOT_RUNNING is true
Running the annotate branch profiler on three boxes, including my
main box that runs firefox, evolution, xchat, and is part of the distcc farm,
showed this with the likelys in the workqueue code:
correct incorrect % Function File Line
------- --------- - -------- ---- ----
96 996253 99 wq_worker_sleeping workqueue.c 703
96 996247 99 wq_worker_waking_up workqueue.c 677
The likely()s in this case were assuming that WORKER_NOT_RUNNING will
most likely be false. But this is not the case. The reason is
(and shown by adding trace_printks and testing it) that most of the time
WORKER_PREP is set.
In worker_thread() we have:
worker_clr_flags(worker, WORKER_PREP);
[ do work stuff ]
worker_set_flags(worker, WORKER_PREP, false);
(that 'false' means not to wake up an idle worker)
The wq_worker_sleeping() is called from schedule when a worker thread
is putting itself to sleep. Which happens most of the time outside
of that [ do work stuff ].
The wq_worker_waking_up is called by the wakeup worker code, which
is also callod outside that [ do work stuff ].
Thus, the likely and unlikely used by those two functions are actually
backwards.
Remove the annotation and let gcc figure it out.
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
2010-12-03 23:12:33 -05:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2011-01-11 15:58:49 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2012-12-18 10:35:02 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-04-23 17:40:40 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-04-23 17:40:40 +02:00
|
|
|
|
2012-12-17 10:01:23 -05:00
|
|
|
|
|
|
|
|
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 17:06:00 -08:00
|
|
|
|
2012-12-18 10:35:02 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-17 10:01:23 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-04-23 17:40:40 +02:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
workqueue: fix possible stall on try_to_grab_pending() of a delayed work item
Currently, when try_to_grab_pending() grabs a delayed work item, it
leaves its linked work items alone on the delayed_works. The linked
work items are always NO_COLOR and will cause future
cwq_activate_first_delayed() increase cwq->nr_active incorrectly, and
may cause the whole cwq to stall. For example,
state: cwq->max_active = 1, cwq->nr_active = 1
one work in cwq->pool, many in cwq->delayed_works.
step1: try_to_grab_pending() removes a work item from delayed_works
but leaves its NO_COLOR linked work items on it.
step2: Later on, cwq_activate_first_delayed() activates the linked
work item increasing ->nr_active.
step3: cwq->nr_active = 1, but all activated work items of the cwq are
NO_COLOR. When they finish, cwq->nr_active will not be
decreased due to NO_COLOR, and no further work items will be
activated from cwq->delayed_works. the cwq stalls.
Fix it by ensuring the target work item is activated before stealing
PENDING in try_to_grab_pending(). This ensures that all the linked
work items are activated without incorrectly bumping cwq->nr_active.
tj: Updated comment and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: stable@kernel.org
2012-09-18 10:40:00 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
workqueue: fix possible stall on try_to_grab_pending() of a delayed work item
Currently, when try_to_grab_pending() grabs a delayed work item, it
leaves its linked work items alone on the delayed_works. The linked
work items are always NO_COLOR and will cause future
cwq_activate_first_delayed() increase cwq->nr_active incorrectly, and
may cause the whole cwq to stall. For example,
state: cwq->max_active = 1, cwq->nr_active = 1
one work in cwq->pool, many in cwq->delayed_works.
step1: try_to_grab_pending() removes a work item from delayed_works
but leaves its NO_COLOR linked work items on it.
step2: Later on, cwq_activate_first_delayed() activates the linked
work item increasing ->nr_active.
step3: cwq->nr_active = 1, but all activated work items of the cwq are
NO_COLOR. When they finish, cwq->nr_active will not be
decreased due to NO_COLOR, and no further work items will be
activated from cwq->delayed_works. the cwq stalls.
Fix it by ensuring the target work item is activated before stealing
PENDING in try_to_grab_pending(). This ensures that all the linked
work items are activated without incorrectly bumping cwq->nr_active.
tj: Updated comment and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: stable@kernel.org
2012-09-18 10:40:00 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
workqueue: fix possible stall on try_to_grab_pending() of a delayed work item
Currently, when try_to_grab_pending() grabs a delayed work item, it
leaves its linked work items alone on the delayed_works. The linked
work items are always NO_COLOR and will cause future
cwq_activate_first_delayed() increase cwq->nr_active incorrectly, and
may cause the whole cwq to stall. For example,
state: cwq->max_active = 1, cwq->nr_active = 1
one work in cwq->pool, many in cwq->delayed_works.
step1: try_to_grab_pending() removes a work item from delayed_works
but leaves its NO_COLOR linked work items on it.
step2: Later on, cwq_activate_first_delayed() activates the linked
work item increasing ->nr_active.
step3: cwq->nr_active = 1, but all activated work items of the cwq are
NO_COLOR. When they finish, cwq->nr_active will not be
decreased due to NO_COLOR, and no further work items will be
activated from cwq->delayed_works. the cwq stalls.
Fix it by ensuring the target work item is activated before stealing
PENDING in try_to_grab_pending(). This ensures that all the linked
work items are activated without incorrectly bumping cwq->nr_active.
tj: Updated comment and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: stable@kernel.org
2012-09-18 10:40:00 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2012-09-18 10:40:00 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
workqueue: simplify is-work-item-queued-here test
Currently, determining whether a work item is queued on a locked pool
involves somewhat convoluted memory barrier dancing. It goes like the
following.
* When a work item is queued on a pool, work->data is updated before
work->entry is linked to the pending list with a wmb() inbetween.
* When trying to determine whether a work item is currently queued on
a pool pointed to by work->data, it locks the pool and looks at
work->entry. If work->entry is linked, we then do rmb() and then
check whether work->data points to the current pool.
This works because, work->data can only point to a pool if it
currently is or were on the pool and,
* If it currently is on the pool, the tests would obviously succeed.
* It it left the pool, its work->entry was cleared under pool->lock,
so if we're seeing non-empty work->entry, it has to be from the work
item being linked on another pool. Because work->data is updated
before work->entry is linked with wmb() inbetween, work->data update
from another pool is guaranteed to be visible if we do rmb() after
seeing non-empty work->entry. So, we either see empty work->entry
or we see updated work->data pointin to another pool.
While this works, it's convoluted, to put it mildly. With recent
updates, it's now guaranteed that work->data points to cwq only while
the work item is queued and that updating work->data to point to cwq
or back to pool is done under pool->lock, so we can simply test
whether work->data points to cwq which is associated with the
currently locked pool instead of the convoluted memory barrier
dancing.
This patch replaces the memory barrier based "are you still here,
really?" test with much simpler "does work->data points to me?" test -
if work->data points to a cwq which is associated with the currently
locked pool, the work item is guaranteed to be queued on the pool as
work->data can start and stop pointing to such cwq only under
pool->lock and the start and stop coincide with queue and dequeue.
tj: Rewrote the comments and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2013-02-06 18:04:53 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: simplify is-work-item-queued-here test
Currently, determining whether a work item is queued on a locked pool
involves somewhat convoluted memory barrier dancing. It goes like the
following.
* When a work item is queued on a pool, work->data is updated before
work->entry is linked to the pending list with a wmb() inbetween.
* When trying to determine whether a work item is currently queued on
a pool pointed to by work->data, it locks the pool and looks at
work->entry. If work->entry is linked, we then do rmb() and then
check whether work->data points to the current pool.
This works because, work->data can only point to a pool if it
currently is or were on the pool and,
* If it currently is on the pool, the tests would obviously succeed.
* It it left the pool, its work->entry was cleared under pool->lock,
so if we're seeing non-empty work->entry, it has to be from the work
item being linked on another pool. Because work->data is updated
before work->entry is linked with wmb() inbetween, work->data update
from another pool is guaranteed to be visible if we do rmb() after
seeing non-empty work->entry. So, we either see empty work->entry
or we see updated work->data pointin to another pool.
While this works, it's convoluted, to put it mildly. With recent
updates, it's now guaranteed that work->data points to cwq only while
the work item is queued and that updating work->data to point to cwq
or back to pool is done under pool->lock, so we can simply test
whether work->data points to cwq which is associated with the
currently locked pool instead of the convoluted memory barrier
dancing.
This patch replaces the memory barrier based "are you still here,
really?" test with much simpler "does work->data points to me?" test -
if work->data points to a cwq which is associated with the currently
locked pool, the work item is guaranteed to be queued on the pool as
work->data can start and stop pointing to such cwq only under
pool->lock and the start and stop coincide with queue and dequeue.
tj: Rewrote the comments and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2013-02-06 18:04:53 -08:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
implement flush_work()
A basic problem with flush_scheduled_work() is that it blocks behind _all_
presently-queued works, rather than just the work whcih the caller wants to
flush. If the caller holds some lock, and if one of the queued work happens
to want that lock as well then accidental deadlocks can occur.
One example of this is the phy layer: it wants to flush work while holding
rtnl_lock(). But if a linkwatch event happens to be queued, the phy code will
deadlock because the linkwatch callback function takes rtnl_lock.
So we implement a new function which will flush a *single* work - just the one
which the caller wants to free up. Thus we avoid the accidental deadlocks
which can arise from unrelated subsystems' callbacks taking shared locks.
flush_work() non-blockingly dequeues the work_struct which we want to kill,
then it waits for its handler to complete on all CPUs.
Add ->current_work to the "struct cpu_workqueue_struct", it points to
currently running "struct work_struct". When flush_work(work) detects
->current_work == work, it inserts a barrier at the _head_ of ->worklist
(and thus right _after_ that work) and waits for completition. This means
that the next work fired on that CPU will be this barrier, or another
barrier queued by concurrent flush_work(), so the caller of flush_work()
will be woken before any "regular" work has a chance to run.
When wait_on_work() unlocks workqueue_mutex (or whatever we choose to protect
against CPU hotplug), CPU may go away. But in that case take_over_work() will
move a barrier we queued to another CPU, it will be fired sometime, and
wait_on_work() will be woken.
Actually, we are doing cleanup_workqueue_thread()->kthread_stop() before
take_over_work(), so cwq->thread should complete its ->worklist (and thus
the barrier), because currently we don't check kthread_should_stop() in
run_workqueue(). But even if we did, everything should be ok.
[akpm@osdl.org: cleanup]
[akpm@osdl.org: add flush_work_keventd() wrapper]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:33:52 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2008-07-25 01:47:47 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
implement flush_work()
A basic problem with flush_scheduled_work() is that it blocks behind _all_
presently-queued works, rather than just the work whcih the caller wants to
flush. If the caller holds some lock, and if one of the queued work happens
to want that lock as well then accidental deadlocks can occur.
One example of this is the phy layer: it wants to flush work while holding
rtnl_lock(). But if a linkwatch event happens to be queued, the phy code will
deadlock because the linkwatch callback function takes rtnl_lock.
So we implement a new function which will flush a *single* work - just the one
which the caller wants to free up. Thus we avoid the accidental deadlocks
which can arise from unrelated subsystems' callbacks taking shared locks.
flush_work() non-blockingly dequeues the work_struct which we want to kill,
then it waits for its handler to complete on all CPUs.
Add ->current_work to the "struct cpu_workqueue_struct", it points to
currently running "struct work_struct". When flush_work(work) detects
->current_work == work, it inserts a barrier at the _head_ of ->worklist
(and thus right _after_ that work) and waits for completition. This means
that the next work fired on that CPU will be this barrier, or another
barrier queued by concurrent flush_work(), so the caller of flush_work()
will be woken before any "regular" work has a chance to run.
When wait_on_work() unlocks workqueue_mutex (or whatever we choose to protect
against CPU hotplug), CPU may go away. But in that case take_over_work() will
move a barrier we queued to another CPU, it will be fired sometime, and
wait_on_work() will be woken.
Actually, we are doing cleanup_workqueue_thread()->kthread_stop() before
take_over_work(), so cwq->thread should complete its ->worklist (and thus
the barrier), because currently we don't check kthread_should_stop() in
run_workqueue(). But even if we did, everything should be ok.
[akpm@osdl.org: cleanup]
[akpm@osdl.org: add flush_work_keventd() wrapper]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:33:52 -07:00
|
|
|
|
|
|
|
|
|
2010-12-20 19:32:04 +01:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:10 -08:00
|
|
|
|
2010-12-20 19:32:04 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:10 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-12-20 19:32:04 +01:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-08-25 10:33:56 +02:00
|
|
|
|
2012-08-15 23:25:37 +09:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-09-09 13:13:58 +08:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-12-20 19:32:04 +01:00
|
|
|
|
2010-08-24 14:22:47 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
workqueue: make all workqueues non-reentrant
By default, each per-cpu part of a bound workqueue operates separately
and a work item may be executing concurrently on different CPUs. The
behavior avoids some cross-cpu traffic but leads to subtle weirdities
and not-so-subtle contortions in the API.
* There's no sane usefulness in allowing a single work item to be
executed concurrently on multiple CPUs. People just get the
behavior unintentionally and get surprised after learning about it.
Most either explicitly synchronize or use non-reentrant/ordered
workqueue but this is error-prone.
* flush_work() can't wait for multiple instances of the same work item
on different CPUs. If a work item is executing on cpu0 and then
queued on cpu1, flush_work() can only wait for the one on cpu1.
Unfortunately, work items can easily cross CPU boundaries
unintentionally when the queueing thread gets migrated. This means
that if multiple queuers compete, flush_work() can't even guarantee
that the instance queued right before it is finished before
returning.
* flush_work_sync() was added to work around some of the deficiencies
of flush_work(). In addition to the usual flushing, it ensures that
all currently executing instances are finished before returning.
This operation is expensive as it has to walk all CPUs and at the
same time fails to address competing queuer case.
Incorrectly using flush_work() when flush_work_sync() is necessary
is an easy error to make and can lead to bugs which are difficult to
reproduce.
* Similar problems exist for flush_delayed_work[_sync]().
Other than the cross-cpu access concern, there's no benefit in
allowing parallel execution and it's plain silly to have this level of
contortion for workqueue which is widely used from core code to
extremely obscure drivers.
This patch makes all workqueues non-reentrant. If a work item is
executing on a different CPU when queueing is requested, it is always
queued to that CPU. This guarantees that any given work item can be
executing on one CPU at maximum and if a work item is queued and
executing, both are on the same CPU.
The only behavior change which may affect workqueue users negatively
is that non-reentrancy overrides the affinity specified by
queue_work_on(). On a reentrant workqueue, the affinity specified by
queue_work_on() is always followed. Now, if the work item is
executing on one of the CPUs, the work item will be queued there
regardless of the requested affinity. I've reviewed all workqueue
users which request explicit affinity, and, fortunately, none seems to
be crazy enough to exploit parallel execution of the same work item.
This adds an additional busy_hash lookup if the work item was
previously queued on a different CPU. This shouldn't be noticeable
under any sane workload. Work item queueing isn't a very
high-frequency operation and they don't jump across CPUs all the time.
In a micro benchmark to exaggerate this difference - measuring the
time it takes for two work items to repeatedly jump between two CPUs a
number (10M) of times with busy_hash table densely populated, the
difference was around 3%.
While the overhead is measureable, it is only visible in pathological
cases and the difference isn't huge. This change brings much needed
sanity to workqueue and makes its behavior consistent with timer. I
think this is the right tradeoff to make.
This enables significant simplification of workqueue API.
Simplification patches will follow.
Signed-off-by: Tejun Heo <tj@kernel.org>
2012-08-20 14:51:23 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-04-13 22:06:58 +03:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-04-13 22:06:58 +03:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-10-05 10:49:55 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2010-08-25 10:33:56 +02:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-08-25 10:33:56 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2006-07-30 03:03:42 -07:00
|
|
|
|
2008-07-23 21:28:39 -07:00
|
|
|
|
|
|
|
|
|
2006-07-30 03:03:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-07-23 21:28:39 -07:00
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-08-03 10:30:44 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-08-03 10:30:44 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2008-07-25 01:47:53 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2008-07-23 21:28:39 -07:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2012-08-03 10:30:44 -07:00
|
|
|
|
2008-07-23 21:28:39 -07:00
|
|
|
|
2008-07-25 01:47:53 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-05-06 17:44:55 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-11-22 14:54:01 +00:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-01-24 16:36:31 +04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-12-04 07:40:39 -08:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-12-01 16:23:42 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
2012-08-08 09:38:42 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2006-07-30 03:03:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-12-22 01:06:52 -08:00
|
|
|
|
2006-07-30 03:03:42 -07:00
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
2006-07-30 03:03:42 -07:00
|
|
|
|
2012-08-03 10:30:44 -07:00
|
|
|
|
|
|
|
|
|
2006-06-28 13:50:33 -07:00
|
|
|
|
2006-11-22 14:54:01 +00:00
|
|
|
|
2012-08-03 10:30:44 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2006-06-28 13:50:33 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
|
|
|
|
|
2006-06-28 13:50:33 -07:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2012-08-03 10:30:44 -07:00
|
|
|
|
2006-06-28 13:50:33 -07:00
|
|
|
|
2008-05-01 04:35:14 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2006-06-28 13:50:33 -07:00
|
|
|
|
|
|
|
|
|
2013-05-06 17:44:55 -04:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2012-08-03 10:30:47 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2012-08-03 10:30:47 -07:00
|
|
|
|
|
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
2012-08-03 10:30:47 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2012-08-03 10:30:47 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-05-09 02:34:16 -07:00
|
|
|
|
2012-08-03 10:30:47 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-06-28 13:50:33 -07:00
|
|
|
|
2012-08-03 10:30:47 -07:00
|
|
|
|
|
|
|
|
|
2006-06-28 13:50:33 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:47 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
2012-05-14 15:04:50 -07:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
|
|
|
|
|
2012-05-14 15:04:50 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-02-19 12:17:02 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-19 12:17:02 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-02-19 12:17:02 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2007-10-18 23:39:55 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2007-10-18 23:39:55 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-02-19 12:17:02 -08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2011-04-29 18:08:37 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2011-04-29 18:08:37 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-14 12:56:18 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2012-07-13 22:16:45 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:50 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2006-12-06 20:37:26 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2006-12-06 20:37:26 -08:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-07-14 11:31:20 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-07-14 11:31:20 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2007-05-09 02:34:20 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2013-08-21 08:50:40 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2012-09-10 10:03:44 -07:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
|
|
|
|
|
2012-09-10 10:03:44 -07:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-05-01 00:07:00 +09:00
|
|
|
|
2012-09-10 10:03:44 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
lockdep: fix oops in processing workqueue
Under memory load, on x86_64, with lockdep enabled, the workqueue's
process_one_work() has been seen to oops in __lock_acquire(), barfing
on a 0xffffffff00000000 pointer in the lockdep_map's class_cache[].
Because it's permissible to free a work_struct from its callout function,
the map used is an onstack copy of the map given in the work_struct: and
that copy is made without any locking.
Surprisingly, gcc (4.5.1 in Hugh's case) uses "rep movsl" rather than
"rep movsq" for that structure copy: which might race with a workqueue
user's wait_on_work() doing lock_map_acquire() on the source of the
copy, putting a pointer into the class_cache[], but only in time for
the top half of that pointer to be copied to the destination map.
Boom when process_one_work() subsequently does lock_map_acquire()
on its onstack copy of the lockdep_map.
Fix this, and a similar instance in call_timer_fn(), with a
lockdep_copy_map() function which additionally NULLs the class_cache[].
Note: this oops was actually seen on 3.4-next, where flush_work() newly
does the racing lock_map_acquire(); but Tejun points out that 3.4 and
earlier are already vulnerable to the same through wait_on_work().
* Patch orginally from Peter. Hugh modified it a bit and wrote the
description.
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Reported-by: Hugh Dickins <hughd@google.com>
LKML-Reference: <alpine.LSU.2.00.1205070951170.1544@eggly.anvils>
Signed-off-by: Tejun Heo <tj@kernel.org>
2012-05-15 08:06:19 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2012-07-22 10:16:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-22 10:16:34 -07:00
|
|
|
|
2012-09-18 09:59:23 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-12-18 10:35:02 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:15 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-08-13 17:08:19 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-08-21 13:07:26 -07:00
|
|
|
|
2012-12-18 10:35:02 -08:00
|
|
|
|
2010-08-21 13:07:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2012-08-19 00:52:42 +03:00
|
|
|
|
|
|
|
|
|
2012-12-18 10:35:02 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-28 17:33:37 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:15 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2012-12-17 10:01:23 -05:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-12-18 10:35:02 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-04-30 15:27:22 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-11-28 07:17:18 +01:00
|
|
|
|
|
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-11-28 07:17:18 +01:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-02-19 12:17:02 -08:00
|
|
|
|
2013-02-19 12:17:02 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-02-14 14:04:46 +01:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2011-02-14 14:04:46 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-12 14:46:37 -07:00
|
|
|
|
|
|
|
|
|
2011-02-14 14:04:46 +01:00
|
|
|
|
2013-02-19 12:17:02 -08:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2007-05-09 02:33:51 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2007-05-09 02:33:51 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-10-26 14:22:34 -07:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2007-05-09 02:33:51 -07:00
|
|
|
|
2007-05-09 02:33:54 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-11-16 01:09:48 +09:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2007-05-09 02:33:51 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-04-02 16:58:24 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2007-05-09 02:33:51 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2007-05-09 02:33:54 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2007-05-09 02:33:51 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-04-02 16:58:24 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2007-05-23 13:57:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2006-07-30 03:03:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-07-30 03:03:42 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-02-08 04:19:53 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-08-11 10:30:30 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-06-30 01:40:45 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2011-09-14 16:22:28 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2011-09-14 16:22:28 -07:00
|
|
|
|
|
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2012-08-19 00:52:42 +03:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-20 14:51:23 -07:00
|
|
|
|
workqueues: implement flush_work()
Most of users of flush_workqueue() can be changed to use cancel_work_sync(),
but sometimes we really need to wait for the completion and cancelling is not
an option. schedule_on_each_cpu() is good example.
Add the new helper, flush_work(work), which waits for the completion of the
specific work_struct. More precisely, it "flushes" the result of of the last
queue_work() which is visible to the caller.
For example, this code
queue_work(wq, work);
/* WINDOW */
queue_work(wq, work);
flush_work(work);
doesn't necessary work "as expected". What can happen in the WINDOW above is
- wq starts the execution of work->func()
- the caller migrates to another CPU
now, after the 2nd queue_work() this work is active on the previous CPU, and
at the same time it is queued on another. In this case flush_work(work) may
return before the first work->func() completes.
It is trivial to add another helper
int flush_work_sync(struct work_struct *work)
{
return flush_work(work) || wait_on_work(work);
}
which works "more correctly", but it has to iterate over all CPUs and thus
it much slower than flush_work().
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Acked-by: Max Krasnyansky <maxk@qualcomm.com>
Acked-by: Jarek Poplawski <jarkao2@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-25 01:47:49 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
workqueues: implement flush_work()
Most of users of flush_workqueue() can be changed to use cancel_work_sync(),
but sometimes we really need to wait for the completion and cancelling is not
an option. schedule_on_each_cpu() is good example.
Add the new helper, flush_work(work), which waits for the completion of the
specific work_struct. More precisely, it "flushes" the result of of the last
queue_work() which is visible to the caller.
For example, this code
queue_work(wq, work);
/* WINDOW */
queue_work(wq, work);
flush_work(work);
doesn't necessary work "as expected". What can happen in the WINDOW above is
- wq starts the execution of work->func()
- the caller migrates to another CPU
now, after the 2nd queue_work() this work is active on the previous CPU, and
at the same time it is queued on another. In this case flush_work(work) may
return before the first work->func() completes.
It is trivial to add another helper
int flush_work_sync(struct work_struct *work)
{
return flush_work(work) || wait_on_work(work);
}
which works "more correctly", but it has to iterate over all CPUs and thus
it much slower than flush_work().
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Acked-by: Max Krasnyansky <maxk@qualcomm.com>
Acked-by: Jarek Poplawski <jarkao2@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-25 01:47:49 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2010-09-16 10:42:16 +02:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
workqueues: implement flush_work()
Most of users of flush_workqueue() can be changed to use cancel_work_sync(),
but sometimes we really need to wait for the completion and cancelling is not
an option. schedule_on_each_cpu() is good example.
Add the new helper, flush_work(work), which waits for the completion of the
specific work_struct. More precisely, it "flushes" the result of of the last
queue_work() which is visible to the caller.
For example, this code
queue_work(wq, work);
/* WINDOW */
queue_work(wq, work);
flush_work(work);
doesn't necessary work "as expected". What can happen in the WINDOW above is
- wq starts the execution of work->func()
- the caller migrates to another CPU
now, after the 2nd queue_work() this work is active on the previous CPU, and
at the same time it is queued on another. In this case flush_work(work) may
return before the first work->func() completes.
It is trivial to add another helper
int flush_work_sync(struct work_struct *work)
{
return flush_work(work) || wait_on_work(work);
}
which works "more correctly", but it has to iterate over all CPUs and thus
it much slower than flush_work().
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Acked-by: Max Krasnyansky <maxk@qualcomm.com>
Acked-by: Jarek Poplawski <jarkao2@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-25 01:47:49 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
workqueue: simplify is-work-item-queued-here test
Currently, determining whether a work item is queued on a locked pool
involves somewhat convoluted memory barrier dancing. It goes like the
following.
* When a work item is queued on a pool, work->data is updated before
work->entry is linked to the pending list with a wmb() inbetween.
* When trying to determine whether a work item is currently queued on
a pool pointed to by work->data, it locks the pool and looks at
work->entry. If work->entry is linked, we then do rmb() and then
check whether work->data points to the current pool.
This works because, work->data can only point to a pool if it
currently is or were on the pool and,
* If it currently is on the pool, the tests would obviously succeed.
* It it left the pool, its work->entry was cleared under pool->lock,
so if we're seeing non-empty work->entry, it has to be from the work
item being linked on another pool. Because work->data is updated
before work->entry is linked with wmb() inbetween, work->data update
from another pool is guaranteed to be visible if we do rmb() after
seeing non-empty work->entry. So, we either see empty work->entry
or we see updated work->data pointin to another pool.
While this works, it's convoluted, to put it mildly. With recent
updates, it's now guaranteed that work->data points to cwq only while
the work item is queued and that updating work->data to point to cwq
or back to pool is done under pool->lock, so we can simply test
whether work->data points to cwq which is associated with the
currently locked pool instead of the convoluted memory barrier
dancing.
This patch replaces the memory barrier based "are you still here,
really?" test with much simpler "does work->data points to me?" test -
if work->data points to a cwq which is associated with the currently
locked pool, the work item is guaranteed to be queued on the pool as
work->data can start and stop pointing to such cwq only under
pool->lock and the start and stop coincide with queue and dequeue.
tj: Rewrote the comments and description.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2013-02-06 18:04:53 -08:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2012-08-20 14:51:23 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2012-08-20 14:51:23 -07:00
|
|
|
|
workqueues: implement flush_work()
Most of users of flush_workqueue() can be changed to use cancel_work_sync(),
but sometimes we really need to wait for the completion and cancelling is not
an option. schedule_on_each_cpu() is good example.
Add the new helper, flush_work(work), which waits for the completion of the
specific work_struct. More precisely, it "flushes" the result of of the last
queue_work() which is visible to the caller.
For example, this code
queue_work(wq, work);
/* WINDOW */
queue_work(wq, work);
flush_work(work);
doesn't necessary work "as expected". What can happen in the WINDOW above is
- wq starts the execution of work->func()
- the caller migrates to another CPU
now, after the 2nd queue_work() this work is active on the previous CPU, and
at the same time it is queued on another. In this case flush_work(work) may
return before the first work->func() completes.
It is trivial to add another helper
int flush_work_sync(struct work_struct *work)
{
return flush_work(work) || wait_on_work(work);
}
which works "more correctly", but it has to iterate over all CPUs and thus
it much slower than flush_work().
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Acked-by: Max Krasnyansky <maxk@qualcomm.com>
Acked-by: Jarek Poplawski <jarkao2@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-25 01:47:49 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2011-01-09 23:32:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2011-01-09 23:32:15 +01:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
|
|
|
|
|
2011-01-09 23:32:15 +01:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
workqueues: implement flush_work()
Most of users of flush_workqueue() can be changed to use cancel_work_sync(),
but sometimes we really need to wait for the completion and cancelling is not
an option. schedule_on_each_cpu() is good example.
Add the new helper, flush_work(work), which waits for the completion of the
specific work_struct. More precisely, it "flushes" the result of of the last
queue_work() which is visible to the caller.
For example, this code
queue_work(wq, work);
/* WINDOW */
queue_work(wq, work);
flush_work(work);
doesn't necessary work "as expected". What can happen in the WINDOW above is
- wq starts the execution of work->func()
- the caller migrates to another CPU
now, after the 2nd queue_work() this work is active on the previous CPU, and
at the same time it is queued on another. In this case flush_work(work) may
return before the first work->func() completes.
It is trivial to add another helper
int flush_work_sync(struct work_struct *work)
{
return flush_work(work) || wait_on_work(work);
}
which works "more correctly", but it has to iterate over all CPUs and thus
it much slower than flush_work().
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Acked-by: Max Krasnyansky <maxk@qualcomm.com>
Acked-by: Jarek Poplawski <jarkao2@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-25 01:47:49 -07:00
|
|
|
|
2010-09-16 10:42:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-20 14:51:23 -07:00
|
|
|
|
|
|
|
|
|
2010-09-16 10:42:16 +02:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-09-16 10:42:16 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-18 11:00:29 -07:00
|
|
|
|
|
|
|
|
|
2012-04-20 17:28:50 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-18 11:00:29 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2012-08-20 14:51:23 -07:00
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2007-07-15 23:41:44 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
2007-07-15 23:41:44 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-20 14:51:23 -07:00
|
|
|
|
2007-07-15 23:41:44 -07:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-20 14:51:23 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2007-07-15 23:41:44 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-07-15 23:41:44 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
implement flush_work()
A basic problem with flush_scheduled_work() is that it blocks behind _all_
presently-queued works, rather than just the work whcih the caller wants to
flush. If the caller holds some lock, and if one of the queued work happens
to want that lock as well then accidental deadlocks can occur.
One example of this is the phy layer: it wants to flush work while holding
rtnl_lock(). But if a linkwatch event happens to be queued, the phy code will
deadlock because the linkwatch callback function takes rtnl_lock.
So we implement a new function which will flush a *single* work - just the one
which the caller wants to free up. Thus we avoid the accidental deadlocks
which can arise from unrelated subsystems' callbacks taking shared locks.
flush_work() non-blockingly dequeues the work_struct which we want to kill,
then it waits for its handler to complete on all CPUs.
Add ->current_work to the "struct cpu_workqueue_struct", it points to
currently running "struct work_struct". When flush_work(work) detects
->current_work == work, it inserts a barrier at the _head_ of ->worklist
(and thus right _after_ that work) and waits for completition. This means
that the next work fired on that CPU will be this barrier, or another
barrier queued by concurrent flush_work(), so the caller of flush_work()
will be woken before any "regular" work has a chance to run.
When wait_on_work() unlocks workqueue_mutex (or whatever we choose to protect
against CPU hotplug), CPU may go away. But in that case take_over_work() will
move a barrier we queued to another CPU, it will be fired sometime, and
wait_on_work() will be woken.
Actually, we are doing cleanup_workqueue_thread()->kthread_stop() before
take_over_work(), so cwq->thread should complete its ->worklist (and thus
the barrier), because currently we don't check kthread_should_stop() in
run_workqueue(). But even if we did, everything should be ok.
[akpm@osdl.org: cleanup]
[akpm@osdl.org: add flush_work_keventd() wrapper]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:33:52 -07:00
|
|
|
|
2007-05-09 02:34:22 -07:00
|
|
|
|
implement flush_work()
A basic problem with flush_scheduled_work() is that it blocks behind _all_
presently-queued works, rather than just the work whcih the caller wants to
flush. If the caller holds some lock, and if one of the queued work happens
to want that lock as well then accidental deadlocks can occur.
One example of this is the phy layer: it wants to flush work while holding
rtnl_lock(). But if a linkwatch event happens to be queued, the phy code will
deadlock because the linkwatch callback function takes rtnl_lock.
So we implement a new function which will flush a *single* work - just the one
which the caller wants to free up. Thus we avoid the accidental deadlocks
which can arise from unrelated subsystems' callbacks taking shared locks.
flush_work() non-blockingly dequeues the work_struct which we want to kill,
then it waits for its handler to complete on all CPUs.
Add ->current_work to the "struct cpu_workqueue_struct", it points to
currently running "struct work_struct". When flush_work(work) detects
->current_work == work, it inserts a barrier at the _head_ of ->worklist
(and thus right _after_ that work) and waits for completition. This means
that the next work fired on that CPU will be this barrier, or another
barrier queued by concurrent flush_work(), so the caller of flush_work()
will be woken before any "regular" work has a chance to run.
When wait_on_work() unlocks workqueue_mutex (or whatever we choose to protect
against CPU hotplug), CPU may go away. But in that case take_over_work() will
move a barrier we queued to another CPU, it will be fired sometime, and
wait_on_work() will be woken.
Actually, we are doing cleanup_workqueue_thread()->kthread_stop() before
take_over_work(), so cwq->thread should complete its ->worklist (and thus
the barrier), because currently we don't check kthread_should_stop() in
run_workqueue(). But even if we did, everything should be ok.
[akpm@osdl.org: cleanup]
[akpm@osdl.org: add flush_work_keventd() wrapper]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:33:52 -07:00
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-07-15 23:41:44 -07:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
2012-08-03 10:30:45 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-09-16 10:48:29 +02:00
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
|
|
|
|
|
2010-09-16 10:48:29 +02:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-09-16 10:48:29 +02:00
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
2010-09-16 10:48:29 +02:00
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
2010-09-16 10:48:29 +02:00
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
2012-10-18 16:31:37 -07:00
|
|
|
|
2010-09-16 10:48:29 +02:00
|
|
|
|
2012-08-21 13:18:24 -07:00
|
|
|
|
2010-09-16 10:48:29 +02:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-09-16 10:36:00 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2012-08-03 10:30:46 -07:00
|
|
|
|
make cancel_rearming_delayed_work() reliable
Thanks to Jarek Poplawski for the ideas and for spotting the bug in the
initial draft patch.
cancel_rearming_delayed_work() currently has many limitations, because it
requires that dwork always re-arms itself via queue_delayed_work(). So it
hangs forever if dwork doesn't do this, or cancel_rearming_delayed_work/
cancel_delayed_work was already called. It uses flush_workqueue() in a
loop, so it can't be used if workqueue was freezed, and it is potentially
live- lockable on busy system if delay is small.
With this patch cancel_rearming_delayed_work() doesn't make any assumptions
about dwork, it can re-arm itself via queue_delayed_work(), or
queue_work(), or do nothing.
As a "side effect", cancel_work_sync() was changed to handle re-arming works
as well.
Disadvantages:
- this patch adds wmb() to insert_work().
- slowdowns the fast path (when del_timer() succeeds on entry) of
cancel_rearming_delayed_work(), because wait_on_work() is called
unconditionally. In that case, compared to the old version, we are
doing "unneeded" lock/unlock for each online CPU.
On the other hand, this means we don't need to use cancel_work_sync()
after cancel_rearming_delayed_work().
- complicates the code (.text grows by 130 bytes).
[akpm@linux-foundation.org: fix speling]
Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru>
Cc: David Chinner <dgc@sgi.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Acked-by: Jarek Poplawski <jarkao2@o2.pl>
Cc: Srivatsa Vaddagiri <vatsa@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 02:34:46 -07:00
|
|
|
|
2007-07-15 23:41:44 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-06-25 05:47:49 -07:00
|
|
|
|
2010-10-19 11:14:49 +02:00
|
|
|
|
2006-06-25 05:47:49 -07:00
|
|
|
|
|
|
|
|
|
2010-10-19 11:14:49 +02:00
|
|
|
|
|
|
|
|
|
2006-06-25 05:47:49 -07:00
|
|
|
|
2010-10-19 11:14:49 +02:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-10-19 11:14:49 +02:00
|
|
|
|
2006-06-25 05:47:49 -07:00
|
|
|
|
2006-11-22 14:55:48 +00:00
|
|
|
|
2006-01-08 01:00:43 -08:00
|
|
|
|
|
|
|
|
|
2010-08-08 14:24:09 +02:00
|
|
|
|
2006-01-08 01:00:43 -08:00
|
|
|
|
2006-06-25 05:47:49 -07:00
|
|
|
|
|
|
|
|
|
2006-01-08 01:00:43 -08:00
|
|
|
|
2006-06-25 05:47:49 -07:00
|
|
|
|
2009-11-17 14:06:20 -08:00
|
|
|
|
|
|
|
|
|
2006-01-08 01:00:43 -08:00
|
|
|
|
2006-12-18 20:05:09 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2009-10-14 06:22:47 +02:00
|
|
|
|
2009-11-17 14:06:20 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-01-25 21:08:02 +01:00
|
|
|
|
2006-06-25 05:47:49 -07:00
|
|
|
|
2006-01-08 01:00:43 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-02-12 17:39:21 +09:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-06-30 01:40:45 -04:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-02-23 12:43:43 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2006-02-23 12:43:43 -06:00
|
|
|
|
|
|
|
|
|
2006-11-22 14:55:48 +00:00
|
|
|
|
2006-02-23 12:43:43 -06:00
|
|
|
|
|
|
|
|
|
2006-11-22 14:55:48 +00:00
|
|
|
|
2006-02-23 12:43:43 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-11-22 14:55:48 +00:00
|
|
|
|
2006-02-23 12:43:43 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-23 14:24:41 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-08-01 09:56:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 03:28:01 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
2013-08-01 09:56:36 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 10:52:30 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-09 14:29:11 +08:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
2013-04-01 11:23:31 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-09-05 12:30:04 -04:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-09-05 12:30:04 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-19 00:52:42 +03:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2013-04-08 16:45:40 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
|
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2010-06-29 10:07:13 +02:00
|
|
|
|
2008-01-16 09:51:58 +01:00
|
|
|
|
2007-05-09 02:34:13 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-17 17:16:24 -08:00
|
|
|
|
|
|
|
|
|
2012-01-10 15:11:35 -08:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-19 13:45:20 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:10 +02:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2011-04-05 18:01:44 +02:00
|
|
|
|
|
|
|
|
|
2010-12-20 19:32:04 +01:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-04 10:05:38 +08:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-03-25 16:57:18 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-03-12 11:30:05 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-08-16 09:54:28 +02:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
2009-11-17 14:06:20 -08:00
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2006-01-08 01:00:43 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2006-01-08 01:00:43 -08:00
|
|
|
|
2013-03-12 17:41:37 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2013-03-12 17:41:37 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 03:28:03 +08:00
|
|
|
|
2013-03-12 17:41:37 -07:00
|
|
|
|
|
|
|
|
|
2010-02-12 17:39:21 +09:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-02-12 17:39:21 +09:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-02-12 17:39:21 +09:00
|
|
|
|
2013-05-10 11:10:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-02-12 17:39:21 +09:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-03-20 03:28:10 +08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-05-10 11:10:17 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:35 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-20 03:28:10 +08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-02-06 18:04:53 -08:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-04-30 15:27:22 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2007-05-09 02:34:15 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2012-07-17 12:39:26 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2013-03-08 15:18:28 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-05-09 02:34:15 -07:00
|
|
|
|
2013-03-08 15:18:28 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:39:44 -08:00
|
|
|
|
2013-03-08 15:18:28 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
workqueue: restore CPU affinity of unbound workers on CPU_ONLINE
With the recent addition of the custom attributes support, unbound
pools may have allowed cpumask which isn't full. As long as some of
CPUs in the cpumask are online, its workers will maintain cpus_allowed
as set on worker creation; however, once no online CPU is left in
cpus_allowed, the scheduler will reset cpus_allowed of any workers
which get scheduled so that they can execute.
To remain compliant to the user-specified configuration, CPU affinity
needs to be restored when a CPU becomes online for an unbound pool
which doesn't currently have any online CPUs before.
This patch implement restore_unbound_workers_cpumask(), which is
called from CPU_ONLINE for all unbound pools, checks whether the
coming up CPU is the first allowed online one, and, if so, invokes
set_cpus_allowed_ptr() with the configured cpumask on all workers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:28 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-06-19 14:53:51 -04:00
|
|
|
|
2012-07-17 12:39:28 -07:00
|
|
|
|
|
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
workqueue: restore CPU affinity of unbound workers on CPU_ONLINE
With the recent addition of the custom attributes support, unbound
pools may have allowed cpumask which isn't full. As long as some of
CPUs in the cpumask are online, its workers will maintain cpus_allowed
as set on worker creation; however, once no online CPU is left in
cpus_allowed, the scheduler will reset cpus_allowed of any workers
which get scheduled so that they can execute.
To remain compliant to the user-specified configuration, CPU affinity
needs to be restored when a CPU becomes online for an unbound pool
which doesn't currently have any online CPUs before.
This patch implement restore_unbound_workers_cpumask(), which is
called from CPU_ONLINE for all unbound pools, checks whether the
coming up CPU is the first allowed online one, and, if so, invokes
set_cpus_allowed_ptr() with the configured cpumask on all workers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2012-07-17 12:39:28 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2012-07-17 12:39:27 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2012-07-17 12:39:28 -07:00
|
|
|
|
2007-05-09 02:34:09 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
workqueue: restore CPU affinity of unbound workers on CPU_ONLINE
With the recent addition of the custom attributes support, unbound
pools may have allowed cpumask which isn't full. As long as some of
CPUs in the cpumask are online, its workers will maintain cpus_allowed
as set on worker creation; however, once no online CPU is left in
cpus_allowed, the scheduler will reset cpus_allowed of any workers
which get scheduled so that they can execute.
To remain compliant to the user-specified configuration, CPU affinity
needs to be restored when a CPU becomes online for an unbound pool
which doesn't currently have any online CPUs before.
This patch implement restore_unbound_workers_cpumask(), which is
called from CPU_ONLINE for all unbound pools, checks whether the
coming up CPU is the first allowed online one, and, if so, invokes
set_cpus_allowed_ptr() with the configured cpumask on all workers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
workqueue: restore CPU affinity of unbound workers on CPU_ONLINE
With the recent addition of the custom attributes support, unbound
pools may have allowed cpumask which isn't full. As long as some of
CPUs in the cpumask are online, its workers will maintain cpus_allowed
as set on worker creation; however, once no online CPU is left in
cpus_allowed, the scheduler will reset cpus_allowed of any workers
which get scheduled so that they can execute.
To remain compliant to the user-specified configuration, CPU affinity
needs to be restored when a CPU becomes online for an unbound pool
which doesn't currently have any online CPUs before.
This patch implement restore_unbound_workers_cpumask(), which is
called from CPU_ONLINE for all unbound pools, checks whether the
coming up CPU is the first allowed online one, and, if so, invokes
set_cpus_allowed_ptr() with the configured cpumask on all workers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
Rebinding workers of a per-cpu pool after a CPU comes online involves
a lot of back-and-forth mostly because only the task itself could
adjust CPU affinity if PF_THREAD_BOUND was set.
As CPU_ONLINE itself couldn't adjust affinity, it had to somehow
coerce the workers themselves to perform set_cpus_allowed_ptr(). Due
to the various states a worker can be in, this led to three different
paths a worker may be rebound. worker->rebind_work is queued to busy
workers. Idle ones are signaled by unlinking worker->entry and call
idle_worker_rebind(). The manager isn't covered by either and
implements its own mechanism.
PF_THREAD_BOUND has been relaced with PF_NO_SETAFFINITY and CPU_ONLINE
itself now can manipulate CPU affinity of workers. This patch
replaces the existing rebind mechanism with direct one where
CPU_ONLINE iterates over all workers using for_each_pool_worker(),
restores CPU affinity, and clears WORKER_UNBOUND.
There are a couple subtleties. All bound idle workers should have
their runqueues set to that of the bound CPU; however, if the target
task isn't running, set_cpus_allowed_ptr() just updates the
cpus_allowed mask deferring the actual migration to when the task
wakes up. This is worked around by waking up idle workers after
restoring CPU affinity before any workers can become bound.
Another subtlety is stems from matching @pool->nr_running with the
number of running unbound workers. While DISASSOCIATED, all workers
are unbound and nr_running is zero. As workers become bound again,
nr_running needs to be adjusted accordingly; however, there is no good
way to tell whether a given worker is running without poking into
scheduler internals. Instead of clearing UNBOUND directly,
rebind_workers() replaces UNBOUND with another new NOT_RUNNING flag -
REBOUND, which will later be cleared by the workers themselves while
preparing for the next round of work item execution. The only change
needed for the workers is clearing REBOUND along with PREP.
* This patch leaves for_each_busy_worker() without any user. Removed.
* idle_worker_rebind(), busy_worker_rebind_fn(), worker->rebind_work
and rebind logic in manager_workers() removed.
* worker_thread() now looks at WORKER_DIE instead of testing whether
@worker->entry is empty to determine whether it needs to do
something special as dying is the only special thing now.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
workqueue: restore CPU affinity of unbound workers on CPU_ONLINE
With the recent addition of the custom attributes support, unbound
pools may have allowed cpumask which isn't full. As long as some of
CPUs in the cpumask are online, its workers will maintain cpus_allowed
as set on worker creation; however, once no online CPU is left in
cpus_allowed, the scheduler will reset cpus_allowed of any workers
which get scheduled so that they can execute.
To remain compliant to the user-specified configuration, CPU affinity
needs to be restored when a CPU becomes online for an unbound pool
which doesn't currently have any online CPUs before.
This patch implement restore_unbound_workers_cpumask(), which is
called from CPU_ONLINE for all unbound pools, checks whether the
coming up CPU is the first allowed online one, and, if so, invokes
set_cpus_allowed_ptr() with the configured cpumask on all workers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
workqueue: restore CPU affinity of unbound workers on CPU_ONLINE
With the recent addition of the custom attributes support, unbound
pools may have allowed cpumask which isn't full. As long as some of
CPUs in the cpumask are online, its workers will maintain cpus_allowed
as set on worker creation; however, once no online CPU is left in
cpus_allowed, the scheduler will reset cpus_allowed of any workers
which get scheduled so that they can execute.
To remain compliant to the user-specified configuration, CPU affinity
needs to be restored when a CPU becomes online for an unbound pool
which doesn't currently have any online CPUs before.
This patch implement restore_unbound_workers_cpumask(), which is
called from CPU_ONLINE for all unbound pools, checks whether the
coming up CPU is the first allowed online one, and, if so, invokes
set_cpus_allowed_ptr() with the configured cpumask on all workers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-03-19 13:45:21 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2008-04-29 01:00:27 -07:00
|
|
|
|
2012-07-17 12:39:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-06-19 14:53:51 -04:00
|
|
|
|
2012-07-17 12:39:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2012-07-17 12:39:28 -07:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2012-07-17 12:39:28 -07:00
|
|
|
|
2012-07-17 12:39:26 -07:00
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2012-08-15 23:25:41 +09:00
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:28 -07:00
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
2009-01-16 15:31:15 -08:00
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
2012-09-18 12:48:43 -07:00
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-09-18 12:48:43 -07:00
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
2012-09-18 12:48:43 -07:00
|
|
|
|
|
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-16 15:31:15 -08:00
|
|
|
|
2009-04-09 09:50:37 -06:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
|
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
2013-03-12 11:29:59 -07:00
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
2012-09-18 12:48:43 -07:00
|
|
|
|
2009-04-09 09:50:37 -06:00
|
|
|
|
2012-09-18 12:48:43 -07:00
|
|
|
|
|
|
|
|
|
2013-11-18 11:00:29 -07:00
|
|
|
|
2008-11-05 13:39:10 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-02-16 09:25:31 +01:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-02-16 09:25:31 +01:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-07-31 14:59:24 -07:00
|
|
|
|
2011-02-16 09:25:31 +01:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 03:28:10 +08:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
2013-02-13 19:29:12 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-20 03:28:10 +08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 03:28:10 +08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2013-03-13 16:51:36 -07:00
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
2013-03-13 19:47:40 -07:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:29:58 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2013-03-13 16:51:35 -07:00
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:19 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:38 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
workqueue: implement NUMA affinity for unbound workqueues
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-05-15 14:24:24 -07:00
|
|
|
|
|
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-07-30 14:57:37 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:11 +02:00
|
|
|
|
2013-03-12 11:29:57 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-17 12:39:26 -07:00
|
|
|
|
2012-09-18 09:59:23 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-04-01 11:23:32 -07:00
|
|
|
|
|
|
|
|
|
2013-01-24 11:01:34 -08:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-04-01 11:23:34 -07:00
|
|
|
|
2013-03-12 11:30:00 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2013-01-24 11:01:33 -08:00
|
|
|
|
2013-03-25 16:57:17 -07:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
2010-06-29 10:07:12 +02:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
2013-03-13 19:47:39 -07:00
|
|
|
|
2012-07-13 22:16:44 -07:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
|
|
|
|
|
2013-09-05 12:30:04 -04:00
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-09-05 12:30:04 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-12 11:30:03 -07:00
|
|
|
|
|
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2012-08-15 23:25:39 +09:00
|
|
|
|
2010-06-29 10:07:14 +02:00
|
|
|
|
2010-07-02 10:03:51 +02:00
|
|
|
|
|
|
|
|
|
2011-02-21 09:52:50 +01:00
|
|
|
|
|
|
|
|
|
2013-04-24 17:12:54 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-08-15 23:25:39 +09:00
|
|
|
|
2013-04-24 17:12:54 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-07-30 14:57:37 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-07-30 14:57:37 -07:00
|
|
|
|
