smp: Enable preemption early in smp_call_function_many_cond()

Browse source

smp_call_function_many_cond() still has to keep the caller pinned to the
current CPU while the remote IPI request is built and dispatched. This
protects the queueing state and CPU-hotplug boundary that are required
before the synchronous wait starts:

 - It protects the current CPU's per-CPU scratch cpumask,
   cfd->cpumask_ipi. Another task running on the same CPU could otherwise
   enter smp_call_function_many_cond() and reuse that scratch cpumask
   before the current caller has finished building and sending the IPI
   request.

 - It provides the CPU-hotplug exclusion required by the CSD queueing
   side. New CSDs must not be queued after smpcfd_dying_cpu() has flushed
   the outgoing CPU's callback queue. Keeping preemption disabled until
   all required CSDs have been queued and the corresponding IPIs have
   been sent prevents CPU offline from crossing that boundary in the
   middle of the queueing operation.

The CSD acquisition side also relies on that caller-side CPU pinning.
csd_lock() waits for CSD_FLAG_LOCK to clear and then marks the CSD busy
with a regular store, so another task on the same CPU must not be
allowed to acquire and reinitialize the same per-CPU CSD concurrently.

After the callbacks have been queued and the IPIs have been sent, the
caller only performs the final csd_lock_wait() completion wait. If it is
preempted there, another task running on the original CPU may enter
smp_call_function_many_cond(), but any attempt to reuse the same per-CPU
CSD will block in csd_lock() until the previous callback clears
CSD_FLAG_LOCK. The final csd_lock_wait() does not acquire or reinitialize
the CSD, so it does not need the same caller-side preemption-disabled
protection.

The wait mask is task-local, so it cannot be overwritten by another task
on the original CPU. The per-CPU CSD storage also remains allocated
across CPU offline, so csd_lock_wait() can safely dereference it even if
the target CPU is offlined after the caller is unpinned.

With those requirements satisfied, enable preemption before the
synchronous csd_lock_wait() loop. This makes the potentially long wait
preemptible and migratable while keeping the CPU-pinned section around
the remote CPU selection and IPI dispatch.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260709122933.4021501-7-zhouchuyi@bytedance.com

...

This commit is contained in:

Chuyi Zhou 2026-07-09 20:29:25 +08:00 • committed by Thomas Gleixner

parent 9f483e5b2f

commit 8df8a60283

1 changed files with 18 additions and 10 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

28

kernel/smp.c

View file