linux/include
Yi Tao 0568f89d4f cgroup: replace global percpu_rwsem with per threadgroup resem when writing to cgroup.procs
The static usage pattern of creating a cgroup, enabling controllers,
and then seeding it with CLONE_INTO_CGROUP doesn't require write
locking cgroup_threadgroup_rwsem and thus doesn't benefit from this
patch.

To avoid affecting other users, the per threadgroup rwsem is only used
when the favordynmods is enabled.

As computer hardware advances, modern systems are typically equipped
with many CPU cores and large amounts of memory, enabling the deployment
of numerous applications. On such systems, container creation and
deletion become frequent operations, making cgroup process migration no
longer a cold path. This leads to noticeable contention with common
process operations such as fork, exec, and exit.

To alleviate the contention between cgroup process migration and
operations like process fork, this patch modifies lock to take the write
lock on signal_struct->group_rwsem when writing pid to
cgroup.procs/threads instead of holding a global write lock.

Cgroup process migration has historically relied on
signal_struct->group_rwsem to protect thread group integrity. In commit
<1ed1328792> ("sched, cgroup: replace signal_struct->group_rwsem with
a global percpu_rwsem"), this was changed to a global
cgroup_threadgroup_rwsem. The advantage of using a global lock was
simplified handling of process group migrations. This patch retains the
use of the global lock for protecting process group migration, while
reducing contention by using per thread group lock during
cgroup.procs/threads writes.

The locking behavior is as follows:

write cgroup.procs/threads  | process fork,exec,exit | process group migration
------------------------------------------------------------------------------
cgroup_lock()               | down_read(&g_rwsem)    | cgroup_lock()
down_write(&p_rwsem)        | down_read(&p_rwsem)    | down_write(&g_rwsem)
critical section            | critical section       | critical section
up_write(&p_rwsem)          | up_read(&p_rwsem)      | up_write(&g_rwsem)
cgroup_unlock()             | up_read(&g_rwsem)      | cgroup_unlock()

g_rwsem denotes cgroup_threadgroup_rwsem, p_rwsem denotes
signal_struct->group_rwsem.

This patch eliminates contention between cgroup migration and fork
operations for threads that belong to different thread groups, thereby
reducing the long-tail latency of cgroup migrations and lowering system
load.

With this patch, under heavy fork and exec interference, the long-tail
latency of cgroup migration has been reduced from milliseconds to
microseconds. Under heavy cgroup migration interference, the multi-CPU
score of the spawn test case in UnixBench increased by 9%.

tj: Update comment in cgroup_favor_dynmods() and switch WARN_ONCE() to
    pr_warn_once().

Signed-off-by: Yi Tao <escape@linux.alibaba.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
History 2025-09-10 07:44:51 -10:00
..
acpi mailbox/pcc: support mailbox management of the shared buffer 2025-08-07 23:49:56 -05:00
asm-generic Deferred unwind changes for 6.17 2025-08-01 09:46:24 -07:00
clocksource KVM/arm64 updates for 6.14 2025-01-28 09:01:36 -08:00
crypto This push fixes a regression that breaks hmac(sha3-224-s390). 2025-08-09 07:26:19 +03:00
cxl cxl/events: Trace Memory Sparing Event Record 2025-07-18 08:19:56 -07:00
drm drm for 6.17-rc1 2025-07-30 19:26:49 -07:00
dt-bindings Pin control changes for v6.17 2025-08-02 12:07:09 -07:00
hyperv hyperv-next for v6.16 2025-06-03 08:39:20 -07:00
keys rxrpc: Add YFS RxGK (GSSAPI) security class 2025-04-14 17:36:41 -07:00
kunit linux_kselftest-kunit-6.17-rc1 2025-07-29 12:43:10 -07:00
kvm KVM/arm64 changes for 6.17, round #1 2025-07-29 12:27:40 -04:00
linux cgroup: replace global percpu_rwsem with per threadgroup resem when writing to cgroup.procs 2025-09-10 07:44:51 -10:00
math-emu powerpc/math-emu: Remove -w build flag and fix warnings 2022-09-05 17:28:25 +10:00
media media: core: export v4l2_translate_cmd 2025-07-11 19:27:28 +02:00
memory memory: renesas-rpc-if: Add RZ/G3E xSPI support 2025-05-01 12:22:25 +02:00
misc cxl: Remove driver 2025-03-16 22:04:27 +11:00
net Previous releases - regressions: 2025-08-08 07:03:25 +03:00
pcmcia ARM: pxa/sa1100: move I/O space to PCI_IOBASE 2022-05-07 22:56:17 +02:00
ras tracing, AER: Hide PCIe AER event when PCIEAER is not configured 2025-07-21 16:40:56 -04:00
rdma IB: Extend UVERBS_METHOD_REG_MR to get DMAH 2025-07-23 01:42:11 -04:00
rv rv: Retry when da monitor detects race conditions 2025-07-28 16:47:34 -04:00
scsi Merge patch series "libsas cleanups" 2025-07-25 09:05:23 -04:00
soc This is the usual collection of primarily clk driver updates. The big part of 2025-07-31 13:36:27 -07:00
sound ASoC: Fixes for v6.17 2025-08-06 20:20:25 +02:00
target scsi: target: core: Constify enabled() in struct target_opcode_descriptor 2025-05-20 22:11:03 -04:00
trace tracing changes for 6.17 2025-08-01 10:29:36 -07:00
uapi TTY revert fix for 6.16-rc1 2025-08-09 18:12:23 +03:00
ufs Merge patch series "ufs: ufs-qcom: Align programming sequence as per HW spec" 2025-07-14 21:00:41 -04:00
vdso vdso/gettimeofday: Add support for auxiliary clocks 2025-07-18 14:09:39 +02:00
video video: Make global edid_info depend on CONFIG_FIRMWARE_EDID 2025-06-16 11:00:29 +02:00
xen Significant patch series in this pull request: 2025-08-03 16:23:09 -07:00
Kbuild drm: ensure drm headers are self-contained and pass kernel-doc 2025-02-12 10:44:43 +02:00