kernel/linux
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ synced 2026-10-22 19:14:51 +00:00
Code Issues Releases Packages Activity
No description
1,322,339 commits 117 branches 5,803 tags 5.7 GiB
  • C 96.8%
  • Assembly 0.9%
  • Rust 0.7%
  • Shell 0.6%
  • Python 0.5%
  • Other 0.3%
Find a file
Nam Cao b99bc5a48d eventpoll: Replace rwlock with spinlock 
commit 0c43094f8c upstream.

The ready event list of an epoll object is protected by read-write
semaphore:

  - The consumer (waiter) acquires the write lock and takes items.
  - the producer (waker) takes the read lock and adds items.

The point of this design is enabling epoll to scale well with large number
of producers, as multiple producers can hold the read lock at the same
time.

Unfortunately, this implementation may cause scheduling priority inversion
problem. Suppose the consumer has higher scheduling priority than the
producer. The consumer needs to acquire the write lock, but may be blocked
by the producer holding the read lock. Since read-write semaphore does not
support priority-boosting for the readers (even with CONFIG_PREEMPT_RT=y),
we have a case of priority inversion: a higher priority consumer is blocked
by a lower priority producer. This problem was reported in [1].

Furthermore, this could also cause stall problem, as described in [2].

Fix this problem by replacing rwlock with spinlock.

This reduces the event bandwidth, as the producers now have to contend with
each other for the spinlock. According to the benchmark from
https://github.com/rouming/test-tools/blob/master/stress-epoll.c:

    On 12 x86 CPUs:
                  Before     After        Diff
        threads  events/ms  events/ms
              8       7162       4956     -31%
             16       8733       5383     -38%
             32       7968       5572     -30%
             64      10652       5739     -46%
            128      11236       5931     -47%

    On 4 riscv CPUs:
                  Before     After        Diff
        threads  events/ms  events/ms
              8       2958       2833      -4%
             16       3323       3097      -7%
             32       3451       3240      -6%
             64       3554       3178     -11%
            128       3601       3235     -10%

Although the numbers look bad, it should be noted that this benchmark
creates multiple threads who do nothing except constantly generating new
epoll events, thus contention on the spinlock is high. For real workload,
the event rate is likely much lower, and the performance drop is not as
bad.

Using another benchmark (perf bench epoll wait) where spinlock contention
is lower, improvement is even observed on x86:

    On 12 x86 CPUs:
        Before: Averaged 110279 operations/sec (+- 1.09%), total secs = 8
        After:  Averaged 114577 operations/sec (+- 2.25%), total secs = 8

    On 4 riscv CPUs:
        Before: Averaged 175767 operations/sec (+- 0.62%), total secs = 8
        After:  Averaged 167396 operations/sec (+- 0.23%), total secs = 8

In conclusion, no one is likely to be upset over this change. After all,
spinlock was used originally for years, and the commit which converted to
rwlock didn't mention a real workload, just that the benchmark numbers are
nice.

This patch is not exactly the revert of commit a218cc4914 ("epoll: use
rwlock in order to reduce ep_poll_callback() contention"), because git
revert conflicts in some places which are not obvious on the resolution.
This patch is intended to be backported, therefore go with the obvious
approach:

  - Replace rwlock_t with spinlock_t one to one

  - Delete list_add_tail_lockless() and chain_epi_lockless(). These were
    introduced to allow producers to concurrently add items to the list.
    But now that spinlock no longer allows producers to touch the event
    list concurrently, these two functions are not necessary anymore.

Fixes: a218cc4914 ("epoll: use rwlock in order to reduce ep_poll_callback() contention")
Signed-off-by: Nam Cao <namcao@linutronix.de>
Link: https://lore.kernel.org/ec92458ea357ec503c737ead0f10b2c6e4c37d47.1752581388.git.namcao@linutronix.de
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: stable@vger.kernel.org
Reported-by: Frederic Weisbecker <frederic@kernel.org>
Closes: https://lore.kernel.org/linux-rt-users/20210825132754.GA895675@lothringen/ [1]
Reported-by: Valentin Schneider <vschneid@redhat.com>
Closes: https://lore.kernel.org/linux-rt-users/xhsmhttqvnall.mognet@vschneid.remote.csb/ [2]
Signed-off-by: Christian Brauner <brauner@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
History 2025-10-19 16:33:49 +02:00
arch ARM: OMAP2+: pm33xx-core: ix device node reference leaks in amx3_idle_init 2025-10-19 16:33:45 +02:00
block blk-crypto: fix missing blktrace bio split events 2025-10-19 16:33:48 +02:00
certs sign-file,extract-cert: use pkcs11 provider for OPENSSL MAJOR >= 3 2024-09-20 19:52:48 +03:00
crypto crypto: essiv - Check ssize for decryption and in-place encryption 2025-10-19 16:33:42 +02:00
Documentation dt-bindings: phy: rockchip-inno-csi-dphy: make power-domains non-required 2025-10-19 16:33:45 +02:00
drivers crypto: rockchip - Fix dma_unmap_sg() nents value 2025-10-19 16:33:49 +02:00
fs eventpoll: Replace rwlock with spinlock 2025-10-19 16:33:49 +02:00
include asm-generic/io.h: Skip trace helpers if rwmmio events are disabled 2025-10-19 16:33:35 +02:00
init init: INITRAMFS_PRESERVE_MTIME should depend on BLK_DEV_INITRD 2025-10-15 11:59:53 +02:00
io_uring io_uring/waitid: always prune wait queue entry in io_waitid_wait() 2025-10-15 12:00:20 +02:00
ipc ipc: fix to protect IPCS lookups using RCU 2025-06-27 11:11:22 +01:00
kernel copy_sighand: Handle architectures where sizeof(unsigned long) < sizeof(u64) 2025-10-19 16:33:48 +02:00
lib lib/sbitmap: convert shallow_depth from one word to the whole sbitmap 2025-08-20 18:30:49 +02:00
LICENSES LICENSES: add 0BSD license text 2024-09-01 20:43:24 -07:00
mm mm: hugetlb: avoid soft lockup when mprotect to large memory area 2025-10-15 12:00:23 +02:00
net bridge: br_vlan_fill_forward_path_pvid: use br_vlan_group_rcu() 2025-10-19 16:33:42 +02:00
rust rust: block: fix srctree/ links 2025-10-12 12:57:16 +02:00
samples ftrace/samples: Fix function size computation 2025-09-19 16:35:44 +02:00
scripts gcc-plugins: Remove TODO_verify_il for GCC >= 16 2025-10-06 11:17:52 +02:00
security lsm: CONFIG_LSM can depend on CONFIG_SECURITY 2025-10-15 11:59:55 +02:00
sound ASoC: SOF: Intel: Read the LLP via the associated Link DMA channel 2025-10-19 16:33:39 +02:00
tools selftests: netfilter: query conntrack state to check for port clash resolution 2025-10-19 16:33:42 +02:00
usr kbuild: hdrcheck: fix cross build with clang 2025-03-13 13:02:18 +01:00
virt KVM: Allow CPU to reschedule while setting per-page memory attributes 2025-07-17 18:37:08 +02:00
.clang-format clang-format: Update with v6.11-rc1's for_each macro list 2024-08-02 13:20:31 +02:00
.clippy.toml rust: give Clippy the minimum supported Rust version 2025-08-01 09:48:44 +01:00
.cocciconfig scripts: add Linux .cocciconfig for coccinelle 2016-07-22 12:13:39 +02:00
.editorconfig .editorconfig: remove trim_trailing_whitespace option 2024-06-13 16:47:52 +02:00
.get_maintainer.ignore Add Jeff Kirsher to .get_maintainer.ignore 2024-03-08 11:36:54 +00:00
.gitattributes .gitattributes: set diff driver for Rust source code files 2023-05-31 17:48:25 +02:00
.gitignore rust: introduce .clippy.toml 2025-03-13 13:01:42 +01:00
.mailmap mailmap: add entry for Thorsten Blum 2024-11-07 14:14:59 -08:00
.rustfmt.toml rust: add .rustfmt.toml 2022-09-28 09:02:20 +02:00
COPYING COPYING: state that all contributions really are covered by this file 2020-02-10 13:32:20 -08:00
CREDITS MAINTAINERS: Remove self from DSA entry 2024-11-03 12:52:38 -08:00
Kbuild Kbuild updates for v6.1 2022-10-10 12:00:45 -07:00
Kconfig kbuild: ensure full rebuild when the compiler is updated 2020-05-12 13:28:33 +09:00
MAINTAINERS MAINTAINERS: Update Alexey Makhalov's email address 2025-05-22 14:29:46 +02:00
Makefile Linux 6.12.53 2025-10-15 12:00:25 +02:00
README README: Fix spelling 2024-03-18 03:36:32 -06:00

README 

Linux kernel
============

There are several guides for kernel developers and users. These guides can
be rendered in a number of formats, like HTML and PDF. Please read
Documentation/admin-guide/README.rst first.

In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.  The formatted documentation can also be read online at:

    https://www.kernel.org/doc/html/latest/

There are various text files in the Documentation/ subdirectory,
several of them using the reStructuredText markup notation.

Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.