sched_ext: Build the cid tables privately and publish them with RCU

Browse source

The cid tables are visible to the cid kfuncs while being modified: the
first enable publishes the global pointers before filling them,
ops.init_cids() overrides rewrite them in place, and re-enables rebuild
them in place. A racing TRACING or SYSCALL program can read unfilled
entries, including uninitialized memory in the kmalloc'd tables, or torn
topo updates.

Tie the tables' lifetimes to the root sched instead: each root enable
builds a fresh set privately and publishes the per-table __rcu globals once
the layout is final, and root disable unpublishes and RCU-frees the set. A
non-NULL global is now always a fully built table which stays valid for the
reader's RCU read section, and lookups stay two loads. Kfuncs treat NULL as
no-mapping, also after the scheduler exits instead of reporting the stale
last mapping.

The cid kfuncs are available whether the root scheduler is cid-form or
cpu-form, the latter to allow gradual migration to cids. Every root
therefore builds and publishes a default mapping.

Every reader must either be gated on scheduler liveness or NULL-check
inside an RCU read section. Fix the two kfuncs that were neither:
scx_bpf_this_cid() read the table with no RCU or preemption protection and
scx_bpf_task_cid() relied on KF_RCU, which doesn't put a sleepable program
in an RCU read section. The hotplug callbacks are instead serialized by
retiring the tables inside the cpus_read_lock() section that clears
scx_root.

v2: Document why every root builds the tables (desc + cid.c comment).

Reported-by: Andrea Righi <arighi@nvidia.com>
Closes: https://lore.kernel.org/r/al3tLtPZZkFjMveK@gpd4
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

...

This commit is contained in:

Tejun Heo 2026-07-21 22:22:18 -10:00

parent 8cc4909134

commit 3a773220d3

5 changed files with 276 additions and 142 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

254

kernel/sched/ext/cid.c

View file

57

kernel/sched/ext/cid.h

View file

54

kernel/sched/ext/ext.c

View file

5

kernel/sched/ext/internal.h

View file

42

kernel/sched/ext/sub.c

View file