2019-05-20 19:08:00 +02:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-26 10:39:56 +01:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-10 10:16:21 +02:00
|
|
|
|
2016-04-01 15:29:48 +03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
2011-12-27 15:10:04 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
2011-12-27 15:10:04 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
2011-12-27 15:10:04 +05:30
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-30 22:57:33 -03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
squashfs: avoid thundering-herd cache wakeups
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e99 ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-08-07 10:24:21 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-06 21:28:55 +08:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-04-01 15:29:48 +03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-12-29 23:37:49 +00:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
2024-12-29 23:37:49 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-21 16:37:42 -08:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
2024-12-29 23:37:49 +00:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
2026-02-21 16:37:42 -08:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-27 15:10:04 +05:30
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 15:29:47 +03:00
|
|
|
|
2009-05-13 02:56:39 +01:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 15:29:47 +03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-12-29 23:37:49 +00:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-03-30 22:57:33 -03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-07 17:29:51 +01:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 15:29:47 +03:00
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 15:29:47 +03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-29 03:50:20 +00:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-07-29 12:44:46 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-17 05:09:55 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
2011-12-29 03:50:20 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-29 03:50:20 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-20 02:26:43 +01:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 15:29:47 +03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
2011-05-20 02:26:43 +01:00
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
2011-05-20 02:26:43 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 15:29:47 +03:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
2011-05-20 02:26:43 +01:00
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
|
|
|
|
|
2011-05-20 02:26:43 +01:00
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
2011-05-20 02:26:43 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-11-18 02:59:12 +00:00
|
|
|
|
|
|
|
|
|
2011-05-20 02:26:43 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-05 08:46:26 +00:00
|
|
|
|
