block: skip blkcg walk in blk_cgroup_congested() when nothing throttled

Browse source

blk_cgroup_congested() walks the current task's blkcg ancestor chain on
every readahead decision and, once swap is in use, on every anonymous and
shmem folio allocation.  The answer is almost always "no", but finding that
out costs two loads per level on two cold cache lines, plus an out-of-line
kthread_blkcg() and an RCU read-side pair.  On a fleet profile of hosts
running containers with 5-10 level hierarchies it costs about as much as
all of mutex_lock(), 99.4% of it under __folio_throttle_swaprate().

Gate the walk on a global count of blkcgs with a non-zero
congestion_count.  The counter only moves on the 0 <-> 1 transitions of
each blkcg's congestion_count, so the extra atomic stays in the throttle
arm/disarm paths and never appears in steady state. When something is
throttled the counter is non-zero and the walk runs as before.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260814165712.510132-4-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>

...

This commit is contained in:

Usama Arif 2026-08-14 09:56:39 -07:00 • committed by Jens Axboe

parent 97cb95d214

commit 4febfe7d98

3 changed files with 57 additions and 6 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

15

block/blk-cgroup.c

View file

25

block/blk-cgroup.h

View file

23

include/linux/blk-cgroup.h

View file