When a sub-scheduler enters bypass mode, its tasks must be scheduled by an
ancestor to guarantee forward progress. Tasks from bypassing descendants are
queued in the bypass DSQs of the nearest non-bypassing ancestor, or the root
scheduler if all ancestors are bypassing. This requires coordination between
bypassing schedulers and their hosts.
Add bypass_enq_target_dsq() to find the correct bypass DSQ by walking up the
hierarchy until reaching a non-bypassing ancestor. When a sub-scheduler starts
bypassing, all its runnable tasks are re-enqueued after scx_bypassing() is set,
ensuring proper migration to ancestor bypass DSQs.
Update scx_dispatch_sched() to handle hosting bypassed descendants. When a
scheduler is not bypassing but has bypassing descendants, it must schedule both
its own tasks and bypassed descendant tasks. A simple policy is implemented
where every Nth dispatch attempt (SCX_BYPASS_HOST_NTH=2) consumes from the
bypass DSQ. A fallback consumption is also added at the end of dispatch to
ensure bypassed tasks make progress even when normal scheduling is idle.
Update enable_bypass_dsp() and disable_bypass_dsp() to increment
bypass_dsp_enable_depth on both the bypassing scheduler and its parent host,
ensuring both can detect that bypass dispatch is active through
bypass_dsp_enabled().
Add SCX_EV_SUB_BYPASS_DISPATCH event counter to track scheduling of bypassed
descendant tasks.
v2: Fix comment typos (Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The bypass_depth field tracks nesting of bypass operations but is also used
to determine whether the bypass dispatch path should be active. With
hierarchical scheduling, child schedulers may need to activate their parent's
bypass dispatch path without affecting the parent's bypass_depth, requiring
separation of these concerns.
Add bypass_dsp_enable_depth and bypass_dsp_claim to independently control
bypass dispatch path activation. The new enable_bypass_dsp() and
disable_bypass_dsp() functions manage this state with proper claim semantics
to prevent races. The bypass dispatch path now only activates when
bypass_dsp_enabled() returns true, which checks the new enable_depth counter.
The disable operation is carefully ordered after all tasks are moved out of
bypass DSQs to ensure they are drained before the dispatch path is disabled.
During scheduler teardown, disable_bypass_dsp() is called explicitly to ensure
cleanup even if bypass mode was never entered normally.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The @prev parameter passed into ops.dispatch() is expected to be on the
same sched. Passing in @prev which isn't on the sched can spuriously
trigger failures that can kill the scheduler. Pass in @prev iff it's on
the same sched.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
In preparation of multiple scheduler support, factor out
scx_dispatch_sched() from balance_one(). The function boundary makes
remembering $prev_on_scx and $prev_on_rq less useful. Open code $prev_on_scx
in balance_one() and $prev_on_rq in both balance_one() and
scx_dispatch_sched().
No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Bypass mode is used to simplify enable and disable paths and guarantee
forward progress when something goes wrong. When enabled, all tasks skip BPF
scheduling and fall back to simple in-kernel FIFO scheduling. While this
global behavior can be used as-is when dealing with sub-scheds, that would
allow any sub-sched instance to affect the whole system in a significantly
disruptive manner.
Make bypass state hierarchical by propagating it to descendants and updating
per-cpu flags accordingly. This allows an scx_sched to bypass if itself or
any of its ancestors are in bypass mode. However, this doesn't make the
actual bypass enqueue and dispatch paths hierarchical yet. That will be done
in later patches.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
In preparation of multiple scheduler support, make bypass state
per-scx_sched. Move scx_bypass_depth, bypass_timestamp and bypass_lb_timer
from globals into scx_sched. Move SCX_RQ_BYPASSING from rq to scx_sched_pcpu
as SCX_SCHED_PCPU_BYPASSING.
scx_bypass() now takes @sch and scx_rq_bypassing(rq) is replaced with
scx_bypassing(sch, cpu). All callers updated.
scx_bypassed_for_enable existed to balance the global scx_bypass_depth when
enable failed. Now that bypass_depth is per-scheduler, the counter is
destroyed along with the scheduler on enable failure. Remove
scx_bypassed_for_enable.
As all tasks currently use the root scheduler, there's no observable behavior
change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
To support bypass mode for sub-schedulers, move bypass_dsq from struct scx_rq
to struct scx_sched_pcpu. Add bypass_dsq() helper. Move bypass_dsq
initialization from init_sched_ext_class() to scx_alloc_and_attach_sched().
bypass_lb_cpu() now takes a CPU number instead of rq pointer. All callers
updated. No behavior change as all tasks use the root scheduler.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The abort state was tracked in the global scx_aborting flag which was used to
break out of potential live-lock scenarios when an error occurs. With
hierarchical scheduling, each scheduler instance must track its own abort
state independently so that an aborting scheduler doesn't interfere with
others.
Move the aborting flag into struct scx_sched and update all access sites. The
early initialization check in scx_root_enable() that warned about residual
aborting state is no longer needed as each scheduler instance now starts with
a clean state.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The default time slice was stored in the global scx_slice_dfl variable which
was dynamically modified when entering and exiting bypass mode. With
hierarchical scheduling, each scheduler instance needs its own default slice
configuration so that bypass operations on one scheduler don't affect others.
Move slice_dfl into struct scx_sched and update all access sites. The bypass
logic now modifies the root scheduler's slice_dfl. At task initialization in
init_scx_entity(), use the SCX_SLICE_DFL constant directly since the task may
not yet be associated with a specific scheduler.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Call ops.core_sched_before() iff both tasks belong to the same scx_sched.
Otherwise, use timestamp based ordering.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
- Add the @sch parameter to scx_init_task() and drop @tg as it can be
obtained from @p. Separate out __scx_init_task() which does everything
except for the task state transition.
- Add the @sch parameter to scx_enable_task(). Separate out
__scx_enable_task() which does everything except for the task state
transition.
- Add the @sch parameter to scx_disable_task().
- Rename scx_exit_task() to scx_disable_and_exit_task() and separate out
__scx_disable_and_exit_task() which does everything except for the task
state transition.
While some task state transitions are relocated, no meaningful behavior
changes are expected.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_bpf_dsq_move[_vtime]() calls scx_dsq_move() to move task from a DSQ to
another. However, @p doesn't necessarily have to come form the containing
iteration and can thus be a task which belongs to another scx_sched. Verify
that @p is on the same scx_sched as the DSQ being iterated.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
scx_bpf_task_set_slice() and scx_bpf_task_set_dsq_vtime() now verify that
the calling scheduler has authority over the task before allowing updates.
This prevents schedulers from modifying tasks that don't belong to them in
hierarchical scheduling configurations.
Direct writes to p->scx.slice and p->scx.dsq_vtime are deprecated and now
trigger warnings. They will be disallowed in a future release.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Add checks to enforce scheduling authority boundaries when multiple
schedulers are present:
1. In scx_dsq_insert_preamble() and the dispatch retry path, ignore attempts
to insert tasks that the scheduler doesn't own, counting them via
SCX_EV_INSERT_NOT_OWNED. As BPF schedulers are allowed to ignore
dequeues, such attempts can occur legitimately during sub-scheduler
enabling when tasks move between schedulers. The counter helps distinguish
normal cases from scheduler bugs.
2. For scx_bpf_dsq_insert_vtime() and scx_bpf_select_cpu_and(), error out
when sub-schedulers are attached. These functions lack the aux__prog
parameter needed to identify the calling scheduler, so they cannot be used
safely with multiple schedulers. BPF programs should use the arg-wrapped
versions (__scx_bpf_dsq_insert_vtime() and __scx_bpf_select_cpu_and())
instead.
These checks ensure that with multiple concurrent schedulers, scheduler
identity can be properly determined and unauthorized task operations are
prevented or tracked.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
In preparation for multiple scheduler support, introduce scx_prog_sched()
accessor which returns the scx_sched instance associated with a BPF program.
The association is determined via the special KF_IMPLICIT_ARGS kfunc
parameter, which provides access to bpf_prog_aux. This aux can be used to
retrieve the struct_ops (sched_ext_ops) that the program is associated with,
and from there, the corresponding scx_sched instance.
For compatibility, when ops.sub_attach is not implemented (older schedulers
without sub-scheduler support), unassociated programs fall back to scx_root.
A warning is logged once per scheduler for such programs.
As scx_root is still the only scheduler, this shouldn't introduce
user-visible behavior changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
In preparation of multiple scheduler support, add p->scx.sched which points
to the scx_sched instance that the task is scheduled by, which is currently
always scx_root. Add scx_task_sched[_rcu]() accessors which return the
associated scx_sched of the specified task and replace the raw scx_root
dereferences with it where applicable. scx_task_on_sched() is also added to
test whether a given task is on the specified sched.
As scx_root is still the only scheduler, this shouldn't introduce
user-visible behavior changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
A system often runs multiple workloads especially in multi-tenant server
environments where a system is split into partitions servicing separate
more-or-less independent workloads each requiring an application-specific
scheduler. To support such and other use cases, sched_ext is in the process
of growing multiple scheduler support.
When partitioning a system in terms of CPUs for such use cases, an
oft-taken approach is hard partitioning the system using cpuset. While it
would be possible to tie sched_ext multiple scheduler support to cpuset
partitions, such an approach would have fundamental limitations stemming
from the lack of dynamism and flexibility.
Users often don't care which specific CPUs are assigned to which workload
and want to take advantage of optimizations which are enabled by running
workloads on a larger machine - e.g. opportunistic over-commit, improving
latency critical workload characteristics while maintaining bandwidth
fairness, employing control mechanisms based on different criteria than
on-CPU time for e.g. flexible memory bandwidth isolation, packing similar
parts from different workloads on same L3s to improve cache efficiency,
and so on.
As this sort of dynamic behaviors are impossible or difficult to implement
with hard partitioning, sched_ext is implementing cgroup sub-sched support
where schedulers can be attached to the cgroup hierarchy and a parent
scheduler is responsible for controlling the CPUs that each child can use
at any given moment. This makes CPU distribution dynamically controlled by
BPF allowing high flexibility.
This patch adds the skeletal sched_ext cgroup sub-sched support:
- sched_ext_ops.sub_cgroup_id and .sub_attach/detach() are added. Non-zero
sub_cgroup_id indicates that the scheduler is to be attached to the
identified cgroup. A sub-sched is attached to the cgroup iff the nearest
ancestor scheduler implements .sub_attach() and grants the attachment. Max
nesting depth is limited by SCX_SUB_MAX_DEPTH.
- When a scheduler exits, all its descendant schedulers are exited
together. Also, cgroup.scx_sched added which points to the effective
scheduler instance for the cgroup. This is updated on scheduler
init/exit and inherited on cgroup online. When a cgroup is offlined, the
attached scheduler is automatically exited.
- Sub-sched support is gated on CONFIG_EXT_SUB_SCHED which is
automatically enabled if both SCX and cgroups are enabled. Sub-sched
support is not tied to the CPU controller but rather the cgroup
hierarchy itself. This is intentional as the support for cpu.weight and
cpu.max based resource control is orthogonal to sub-sched support. Note
that CONFIG_CGROUPS around cgroup subtree iteration support for
scx_task_iter is replaced with CONFIG_EXT_SUB_SCHED for consistency.
- This allows loading sub-scheds and most framework operations such as
propagating disable down the hierarchy work. However, sub-scheds are not
operational yet and all tasks stay with the root sched. This will serve
as the basis for building up full sub-sched support.
- DSQs point to the scx_sched they belong to.
- scx_qmap is updated to allow attachment of sub-scheds and also serving
as sub-scheds.
- scx_is_descendant() is added but not yet used in this patch. It is used by
later changes in the series and placed here as this is where the function
belongs.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
In preparation for multiple scheduler support, reorganize the enable and
disable paths to make scheduler instances explicit. Extract
scx_root_disable() from scx_disable_workfn(). Rename scx_enable_workfn()
to scx_root_enable_workfn(). Change scx_disable() to take @sch parameter
and only queue disable_work if scx_claim_exit() succeeds for consistency.
Move exit_kind validation into scx_claim_exit(). The sysrq handler now
prints a message when no scheduler is loaded.
These changes don't materially affect user-visible behavior.
v2: Keep scx_enable() name as-is and only rename the workfn to
scx_root_enable_workfn(). Change scx_enable() return type to s32.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
- Always trigger the warning if p->scx.disallow is set for fork inits. There
is no reason to set it during forks.
- Flip the positions of if/else arms to ease adding error conditions.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
The planned sched_ext cgroup sub-scheduler support needs the newly forked
task to be associated with its cgroup in its post_fork() hook. There is no
existing ordering requirement between the two now. Swap them and note the
new ordering requirement.
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Cc: Ingo Molnar <mingo@redhat.com>
Make sched_cgroup_fork() pass @kargs to scx_fork(). This will be used to
determine @p's cgroup for cgroup sub-sched support.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Cc: Peter Zijlstra <peterz@infradead.org>
For the planned cgroup sub-scheduler support, enable/disable operations are
going to be subtree specific and iterating all tasks in the system for those
operations can be unnecessarily expensive and disruptive.
cgroup already has mechanisms to perform subtree task iterations. Implement
cgroup subtree iteration for scx_task_iter:
- Add optional @cgrp to scx_task_iter_start() which enables cgroup subtree
iteration.
- Make scx_task_iter use css_next_descendant_pre() and css_task_iter to
iterate all tasks in the cgroup subtree.
- Update all existing callers to pass NULL to maintain current behavior.
The two iteration mechanisms are independent and duplicate. It's likely that
scx_tasks can be removed in favor of always using cgroup iteration if
CONFIG_SCHED_CLASS_EXT depends on CONFIG_CGROUPS.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Fixes and New HW Support
- alienware-wmi-wmax: Add G-Mode support to m18 laptops
- asus-armoury: Add support for FA401UM, G733QS, GX650RX
- dell-wmi-sysman: Don't hex dump plaintext password data
- hp-bioscfg: Support large number of enumeration attributes
- hp-wmi: Add support for Omen 14-fb1xxx, 16-xd0xxx and 16-wf0xxx,
Victus-d0xxx
- int3472: Handle GPIO type 0x10 (DOVDD)
- intel-hid:
- Add Dell 14 & 16 Plus 2-in-1 to dmi_vgbs_allow_list
- Enable 5-button array on ThinkPad X1 Fold 16 Gen 1
- mellanox: mlxreg: Fix kernel-doc warnings
- oxpec: Add support for OneXPlayer X1 Air, X1z, APEX, and Aokzoe A2 Pro
- redmi-wmi: Add more Fn hotkey mappings
- thinkpad_acpi: Fix errors reading battery thresholds
- touchscreen_dmi: Add quirk for y-inverted Goodix touchscreen on SUPI S10
- uniwill-laptop:
- FN lock/super key lock attributes rename
- Fix crash on unexpected battery event
- A special key combination can alter FN lock status so mark it volatile
- Handle FN lock event
The following is an automated shortlog grouped by driver:
alienware-wmi-wmax:
- Add G-Mode support to m18 laptops
asus-armoury:
- add support for FA401UM
- add support for G733QS
- add support for GX650RX
dell-wmi:
- Add audio/mic mute key codes
dell-wmi-sysman:
- Don't hex dump plaintext password data
hp-bioscfg:
- Support allocations of larger data
hp-wmi:
- add Omen 14-fb1xxx (board 8E41) support
- Add Omen 16-wf0xxx fan and thermal support
- Add Omen 16-xd0xxx fan and thermal support
- Add Victus 16-d0xxx support
int3472:
- Handle GPIO type 0x10 (DOVDD)
intel-hid:
- Add Dell 14 Plus 2-in-1 to dmi_vgbs_allow_list
- Add Dell 16 Plus 2-in-1 to dmi_vgbs_allow_list
- Enable 5-button array on ThinkPad X1 Fold 16 Gen 1
oxpec:
- Add support for Aokzoe A2 Pro
- Add support for OneXPlayer APEX
- Add support for OneXPlayer X1 Air
- Add support for OneXPlayer X1z
platform_data/mlxreg: mlxreg.h:
- fix all kernel-doc warnings
redmi-wmi:
- Add more hotkey mappings
thinkpad_acpi:
- Fix errors reading battery thresholds
touchscreen_dmi:
- Add quirk for y-inverted Goodix touchscreen on SUPI S10
uniwill-laptop:
- Fix crash on unexpected battery event
- Handle FN lock event
- Mark FN lock status as being volatile
- Rename FN lock and super key lock attrs
-----BEGIN PGP SIGNATURE-----
iHUEABYIAB0WIQSCSUwRdwTNL2MhaBlZrE9hU+XOMQUCaarJlgAKCRBZrE9hU+XO
MY05AQCR1izfLKgDn88+dlYyLvEaZ7CPTH+nhUddRxb/W5eF4wEAxgkiQaSMk7IV
7GBLuoazK+XDgCK3J1Coq1BiU+JpWgM=
=oUiT
-----END PGP SIGNATURE-----
Merge tag 'platform-drivers-x86-v7.0-2' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86
Pull x86 platform driver fixes from Ilpo Järvinen:
- alienware-wmi-wmax: Add G-Mode support to m18 laptops
- asus-armoury: Add support for FA401UM, G733QS, GX650RX
- dell-wmi-sysman: Don't hex dump plaintext password data
- hp-bioscfg: Support large number of enumeration attributes
- hp-wmi: Add support for Omen 14-fb1xxx, 16-xd0xxx, 16-wf0xxx, and
Victus-d0xxx
- int3472: Handle GPIO type 0x10 (DOVDD)
- intel-hid:
- Add Dell 14 & 16 Plus 2-in-1 to dmi_vgbs_allow_list
- Enable 5-button array on ThinkPad X1 Fold 16 Gen 1
- mellanox: mlxreg: Fix kernel-doc warnings
- oxpec: Add support for OneXPlayer X1 Air, X1z, APEX, and Aokzoe A2
Pro
- redmi-wmi: Add more Fn hotkey mappings
- thinkpad_acpi: Fix errors reading battery thresholds
- touchscreen_dmi: Add quirk for y-inverted Goodix touchscreen on SUPI
S10
- uniwill-laptop:
- FN lock/super key lock attributes rename
- Fix crash on unexpected battery event
- A special key combination can alter FN lock status so mark it
volatile
- Handle FN lock event
* tag 'platform-drivers-x86-v7.0-2' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86: (27 commits)
platform/x86: dell-wmi-sysman: Don't hex dump plaintext password data
platform_data/mlxreg: mlxreg.h: fix all kernel-doc warnings
platform/x86: asus-armoury: add support for FA401UM
platform/x86: asus-armoury: add support for GX650RX
platform/x86: hp-bioscfg: Support allocations of larger data
platform/x86: oxpec: Add support for Aokzoe A2 Pro
platform/x86: oxpec: Add support for OneXPlayer X1 Air
platform/x86: oxpec: Add support for OneXPlayer X1z
platform/x86: oxpec: Add support for OneXPlayer APEX
platform/x86: uniwill-laptop: Handle FN lock event
platform/x86: uniwill-laptop: Mark FN lock status as being volatile
platform/x86: uniwill-laptop: Fix crash on unexpected battery event
platform/x86: uniwill-laptop: Rename FN lock and super key lock attrs
platform/x86: redmi-wmi: Add more hotkey mappings
platform/x86: alienware-wmi-wmax: Add G-Mode support to m18 laptops
platform/x86: hp-wmi: add Omen 14-fb1xxx (board 8E41) support
platform/x86: dell-wmi: Add audio/mic mute key codes
platform/x86: hp-wmi: Add Victus 16-d0xxx support
platform/x86: intel-hid: Enable 5-button array on ThinkPad X1 Fold 16 Gen 1
platform/x86: int3472: Handle GPIO type 0x10 (DOVDD)
...
The .remove() callback only prints out a debug message, remove
this otherwise unneeded callback.
Signed-off-by: Andrew Davis <afd@ti.com>
Link: https://lore.kernel.org/r/20260302202728.322073-2-afd@ti.com
Signed-off-by: Mathieu Poirier <mathieu.poirier@linaro.org>
-----BEGIN PGP SIGNATURE-----
iQFPBAABCAA5FiEEe7vIQRWZI0iWSE3xu+CwddJFiJoFAmmqujcbFIAAAAAABAAO
bWFudTIsMi41KzEuMTIsMiwyAAoJELvgsHXSRYiakO0H/RtqrDjh7evZtXlOu5l2
7cg2HKYeytPwlKbyerIZb7bt0rgBOVIugZWNswluvCXFWf8ypioBPUEKnkxuXtXd
9+8pdt8GOdW6XwobvmnupEWeN7xXMygPtMABh9E5GX1flxha5DVlspL6m4RUadJ9
Or6uo33NB9s5CQGFadb3CjTUnj5tlcKt48hvDisaxzjr1UaYrE9pauwL+UUB58Zi
Xd38HET0a6mpBOsdPuzgrvGmtXL8dhwnNVY6aSMvpqeHY4w03iUXM08XdHbaqOW9
X1RYzmZNOR1poxFTsnhVGPvRNuVTpktm7hqh0JnbdEHM8zF72ZmXS8OxciCGZE4H
Zr4=
=tIX2
-----END PGP SIGNATURE-----
Merge tag 'slab-for-7.0-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab
Pull slab fixes from Vlastimil Babka:
- Fix for slab->stride truncation on 64k page systems due to short
type. It was not due to races and lack of barriers in the end. (Harry
Yoo)
- Fix for severe performance regression due to unnecessary sheaf refill
restrictions exposed by mempool allocation strategy. (Vlastimil
Babka)
- Stable fix for potential silent percpu sheaf flushing failures on
PREEMPT_RT. (Vlastimil Babka)
* tag 'slab-for-7.0-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab:
mm/slab: change stride type from unsigned short to unsigned int
mm/slab: allow sheaf refill if blocking is not allowed
slab: distinguish lock and trylock for sheaf_flush_main()
This came up as a result of the tracing fix pull request, and commit
e39bb9e02b ("tracing: Fix WARN_ON in tracing_buffers_mmap_close") in
particular.
The use of MADV_DOFORK confused the ring buffer mapping reference
counting just because it was unexpected, since the mapping was
originally done with VM_DONTCOPY.
The tracing code may well be the only case of this (and fixed it all by
just using the mmap open callback to unconfuse itself), but it's just
strange that we allow MADV_DOFORK on special mappings where the kernel
has set the "don't copy this" bit.
The code already disallowed it for VM_IO mappings (going back to the
original commit f822566165: "madvise MADV_DONTFORK/MADV_DOFORK"), so
just extend it to any of the VM_SPECIAL cases (which includes
VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP in addition to VM_IO).
We could also allow MADV_DOFORK only on mappings that had been marked
DONTFORK by the user. But that would require us to track that
(presumably with another VM_xyz bit), so let's just do this trivial and
straightforward modifications.
If anybody notices, Lorenzo will be boarding Flying Pig Airlines.
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Link: https://lore.kernel.org/all/a8907468-d7e9-4727-af28-66d905093230@kernel.org/
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Besides deferring the call to housekeeping_update(), commit 6df415aa46
("cgroup/cpuset: Defer housekeeping_update() calls from CPU hotplug
to workqueue") also defers the rebuild_sched_domains() call to
the workqueue. So a new offline CPU may still be in a sched domain
or new online CPU not showing up in the sched domains for a short
transition period. That could be a problem in some corner cases and
can be the cause of a reported test failure[1]. Fix it by calling
rebuild_sched_domains_cpuslocked() directly in hotplug as before. If
isolated partition invalidation or recreation is being done, the
housekeeping_update() call to update the housekeeping cpumasks will
still be deferred to a workqueue.
In commit 3bfe479671 ("cgroup/cpuset: Move
housekeeping_update()/rebuild_sched_domains() together"),
housekeeping_update() is called before rebuild_sched_domains() because
it needs to access the HK_TYPE_DOMAIN housekeeping cpumask. That is now
changed to use the static HK_TYPE_DOMAIN_BOOT cpumask as HK_TYPE_DOMAIN
cpumask is now changeable at run time. As a result, we can move the
rebuild_sched_domains() call before housekeeping_update() with
the slight advantage that it will be done in the same cpus_read_lock
critical section without the possibility of interference by a concurrent
cpu hot add/remove operation.
As it doesn't make sense to acquire cpuset_mutex/cpuset_top_mutex after
calling housekeeping_update() and immediately release them again, move
the cpuset_full_unlock() operation inside update_hk_sched_domains()
and rename it to cpuset_update_sd_hk_unlock() to signify that it will
release the full set of locks.
[1] https://lore.kernel.org/lkml/1a89aceb-48db-4edd-a730-b445e41221fe@nvidia.com
Fixes: 6df415aa46 ("cgroup/cpuset: Defer housekeeping_update() calls from CPU hotplug to workqueue")
Tested-by: Jon Hunter <jonathanh@nvidia.com>
Reviewed-by: Chen Ridong <chenridong@huaweicloud.com>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Replace unbounded sprintf() with the safer snprintf(). While the
current code works correctly, snprintf() is safer and follows secure
coding best practices.
No functional changes.
Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Reviewed-by: Lukasz Luba <lukasz.luba@arm.com>
[ rjw: Subject tweaks ]
Link: https://patch.msgid.link/20260223073245.321298-2-thorsten.blum@linux.dev
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Commit 0927780c90 ("sched_ext: Use READ_ONCE() for lock-free reads
of module param variables") annotated the plain reads of
scx_slice_bypass_us and scx_bypass_lb_intv_us in bypass_lb_cpu(), but
missed a third site in scx_bypass():
WRITE_ONCE(scx_slice_dfl, scx_slice_bypass_us * NSEC_PER_USEC);
scx_slice_bypass_us is a module parameter writable via sysfs in
process context through set_slice_us() -> param_set_uint_minmax(),
which performs a plain store without holding bypass_lock. scx_bypass()
reads the variable under bypass_lock, but since the writer does not
take that lock, the two accesses are concurrent.
WRITE_ONCE() only applies volatile semantics to the store of
scx_slice_dfl -- the val expression containing scx_slice_bypass_us is
evaluated as a plain read, providing no protection against concurrent
writes.
Wrap the read with READ_ONCE() to complete the annotation started by
commit 0927780c90 and make the access KCSAN-clean, consistent with
the existing READ_ONCE(scx_slice_bypass_us) in bypass_lb_cpu().
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
- Fix use-after-free in ccp.
- Fix bug when SEV is disabled in ccp.
- Fix tfm_count leak in atmel-sha204a.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEn51F/lCuNhUwmDeSxycdCkmxi6cFAmmqQ0cACgkQxycdCkmx
i6eUzg/+PHIyLgwiTcFljyxlJo8g0jDkbaaA+U3E+QImkgCgh+LdhvWp9g1nU2m/
dTcSAue2Vkgvnht7QOm2rQ/CWX+yjs9QOlztpyl7imajFIoTxQ/wRbJ+YO1T07Vr
ZQmEO6B/N/GqIlSGvUXVVy9TZN5DhgXol7jWZtmQU+EYjFrLsWUMZd2Q3fDT6eoF
3QdbJBymblv78WxJ0PtMkEpqCF2ikcDCrX4mokrrXZXyakp263pzH7pbWl1xLXR5
fUZoTsGcyh2Me2Fcsanip8SOqj8DK80UguDZZki8tcHjH4tbVPyjJ0TqQwq8cMMb
rCkGCfbV1g1+YNRuZb4YbMUeSXDqwf7G2FaI/DaxWKPYPik9yUnRt0WXAUXAX9wK
NQ4V+PTlihkT+r4WqjVH1E8EpWSoS0oCWOE+1NwsW1idpo7UHNZPoR5BVrzp5jJ/
iW/hO28djHFebzQ1DZmiDCWvMKAHdUVN9gAFB0uyOoYgdWn1TMbuk83glGhYqHHV
mCP62GuFJrZov4MO4EpiwlJi5pN1uRS69ZOFkHrM9db/3xnBO9zE48d9ePPLu+VM
ng5N1Nwh1VXeR5mwk/OBSvHHN3vI76NU2fcfgLOQy5Z0YD9kk6jyydcKhmZP+BBN
zXiJrsOhxUOpe2/TLgYCL+RNqzGMpuAtftcdi817TwopEXkheE4=
=P2SQ
-----END PGP SIGNATURE-----
Merge tag 'v7.0-p2' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
Pull crypto fixes from Herbert Xu:
- Fix use-after-free in ccp
- Fix bug when SEV is disabled in ccp
- Fix tfm_count leak in atmel-sha204a
* tag 'v7.0-p2' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6:
crypto: atmel-sha204a - Fix OOM ->tfm_count leak
crypto: ccp - Fix use-after-free on error path
crypto: ccp - allow callers to use HV-Fixed page API when SEV is disabled
- Fix a problem where the deferred non-NCQ command would incorrectly get
completed as a failed command, if there was another command that timed
out. Found by Gemini. (Guenter)
- The deferred non-NCQ command work is only supposed to run after the
last NCQ command finishes. However, because the work was never canceled
on error (e.g. a timeout), the work could incorrectly run when commands
were still in flight. Found by syzbot. (me)
- Add a quirk to make sure that QEMU harddrives can potentially use up to
32 MiB I/Os. (Pedro)
- Add a quirk to disable LPM on Seagate ST1000DM010-2EP102. (Maximilian)
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQRN+ES/c4tHlMch3DzJZDGjmcZNcgUCaarKRAAKCRDJZDGjmcZN
cvqoAQCU9tcNQZcVRtR+QpWmk9ZDqPFpI0qde9TWxVxByx1lUQEAubgAkf0F0Dp3
4eBP43Tniv/7SlPCKjbhEWEq6rvX7gA=
=YBsd
-----END PGP SIGNATURE-----
Merge tag 'ata-7.0-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/libata/linux
Pull ata fixes from Niklas Cassel:
- Fix a problem where the deferred non-NCQ command would incorrectly
get completed as a failed command, if there was another command that
timed out. Found by Gemini (Guenter)
- The deferred non-NCQ command work is only supposed to run after the
last NCQ command finishes. However, because the work was never
canceled on error (e.g. a timeout), the work could incorrectly run
when commands were still in flight. Found by syzbot (me)
- Add a quirk to make sure that QEMU harddrives can potentially use up
to 32 MiB I/Os (Pedro)
- Add a quirk to disable LPM on Seagate ST1000DM010-2EP102 (Maximilian)
* tag 'ata-7.0-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/libata/linux:
ata: libata-eh: Fix detection of deferred qc timeouts
ata: libata-core: Add BRIDGE_OK quirk for QEMU drives
ata: libata: cancel pending work after clearing deferred_qc
ata: libata-core: Disable LPM on ST1000DM010-2EP102
show_cpu_pool_hog() and show_cpu_pools_hogs() no longer only dump CPU
hogs — since commit 8823eaef45 ("workqueue: Show all busy workers in
stall diagnostics"), they dump every in-flight worker in the pool's
busy_hash.
Rename them to show_cpu_pool_busy_workers() and
show_cpu_pools_busy_workers() to accurately describe what they do.
Also fix the pr_info() message to say "stalled worker pools" instead of
"stalled CPU-bound worker pools", since sleeping/blocked workers are now
included.
No functional change.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmmqPRMQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgplf5D/9uOsBr+OGXtkLUJtD6MiwoJUsYgYF2dMIx
epcp+8RdMaOGtigtx69QXzTP5aPjA+AvBLAMYM+QDQDAPMWbRPsD7LaCYHy7ekwA
OL68R3QRTMYPPgpuf7pKyhif7olozAvoWAnRaoWlo67rbK+mTzZsTIsgTwF4zUu6
T0dL9thbWqtJMxKSuUk+DywggvGyNZWICJ3rAZ6os2htruH0fPhsJNGVFgNXMnpe
Cy2OvWxBWRQkZnpDEocZUdYyCRVhHr7hu311j6nSLNXufqpgFmWLGO4C3vetOlgx
ulEHfGNINcSLcw9R8pNWRxU14V6iw8Oy4nU9RtZhUpF32Iasvxb4H0w76Dp9Ukq1
/DuoSkWg/Ahn24xSYxJwwZpOEE8L92pn0M2ukCfC6h7ytmDjjEL1AQ2kyFHV4mR3
nc/3FkQ0abe3HHk8Rit6+txe3sSQo5no1z8kFlb9yp2MwAmonxCCQ9N1s7pxeeP+
iLaPbGMaZ7Ra1GswD/vzxFQtkglsxLuM5D0JkjHe99a54ZnF0vF3y9jeDVOQbV1C
H6/bU/2DI3SQ8xqv6tIXQ22reyRen3ao5VKLSrmrT/tDQVoEBV5SMnJFO1J8jBP4
QST03wiu8ShHSyZ98KefwlsndrTX02V9UVD4FVj+TZXwCWltulnIR4dVYFdySWwW
d613iUsWJw==
=NNcQ
-----END PGP SIGNATURE-----
Merge tag 'block-7.0-20260305' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block fixes from Jens Axboe:
- NVMe pull request via Keith:
- Improve quirk visibility and configurability (Maurizio)
- Fix runtime user modification to queue setup (Keith)
- Fix multipath leak on try_module_get failure (Keith)
- Ignore ambiguous spec definitions for better atomics support
(John)
- Fix admin queue leak on controller reset (Ming)
- Fix large allocation in persistent reservation read keys
(Sungwoo Kim)
- Fix fcloop callback handling (Justin)
- Securely free DHCHAP secrets (Daniel)
- Various cleanups and typo fixes (John, Wilfred)
- Avoid a circular lock dependency issue in the sysfs nr_requests or
scheduler store handling
- Fix a circular lock dependency with the pcpu mutex and the queue
freeze lock
- Cleanup for bio_copy_kern(), using __bio_add_page() rather than the
bio_add_page(), as adding a page here cannot fail. The exiting code
had broken cleanup for the error condition, so make it clear that the
error condition cannot happen
- Fix for a __this_cpu_read() in preemptible context splat
* tag 'block-7.0-20260305' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
block: use trylock to avoid lockdep circular dependency in sysfs
nvme: fix memory allocation in nvme_pr_read_keys()
block: use __bio_add_page in bio_copy_kern
block: break pcpu_alloc_mutex dependency on freeze_lock
blktrace: fix __this_cpu_read/write in preemptible context
nvme-multipath: fix leak on try_module_get failure
nvmet-fcloop: Check remoteport port_state before calling done callback
nvme-pci: do not try to add queue maps at runtime
nvme-pci: cap queue creation to used queues
nvme-pci: ensure we're polling a polled queue
nvme: fix memory leak in quirks_param_set()
nvme: correct comment about nvme_ns_remove()
nvme: stop setting namespace gendisk device driver data
nvme: add support for dynamic quirk configuration via module parameter
nvme: fix admin queue leak on controller reset
nvme-fabrics: use kfree_sensitive() for DHCHAP secrets
nvme: stop using AWUPF
nvme: expose active quirks in sysfs
nvme/host: fixup some typos
Mention the scheduling class precedence of fair and sched_ext to
clear up how sched_ext partial mode works.
Signed-off-by: Christian Loehle <christian.loehle@arm.com>
Acked-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmmqPTAQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpueIEACHF0uws+uZSEsy9LUyC9ha8+5YN9szIJ3K
QUGxa9pmCnQG5K50KpxyEYP6buaJDy1smJGgD2obkeGncC4w6xKK2kQTQV1U+C1C
YA+7B/3HLhz5AWS6GIbRy6VZ599I4evlF8W79dX8BTnF8Y1ddkSuUnKx//q0AoQZ
hr3foglcFlchy8JuQ2/MpxzfOouvNMdMmeUN4O+t8iXDrmePFYIOgrLcT+ObgC5D
SXWx2cc3hMJ35hcSzedMWEBFcXnkX9nfh8Hd/+uPRcKsIwS8kCo6z01GoT/BCPRA
jdrxAfoYSL16HPfq6GU52n6iCaRd+5NK+tt/ECCzTxGL32Hadrr+nxw4O7g3Q96u
07zeaqHSoTGUchtlqrGjALQLP2yxdACEjxMh3rfdStRv3x3bbbVVDdioVEzPukCr
EBA+AbqaaG3LIYXwcY+15zx5NrAfeBAP1RjLgoV0s2ch4ghEqvnZGY4NLBDkcQ2R
97tM9+OdecBrsnlQr5GBoDbwpqc2pDEqSjkYDwoXqvqXs0DrMRq2MQw1Hjjh7Z7G
FZx1KNTiLB/YQ0sSyMcUKnH+qBA0FxwN/C6dDnRjj4dH5YsoeG/GhsS3B00a+0yE
S3MKrsf+uN21OYLVPSTEN6qS+02ZvK6E/Aw7/fk2IV60JMeM5KvCccmxa53dKls8
iyEJ7nVLOg==
=xyKA
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.0-20260305' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a typo in the mock_file help text
- Fix a comment regarding IORING_SETUP_TASKRUN_FLAG in the
io_uring.h UAPI header
- Use READ_ONCE() for reading refill queue entries
- Reject SEND_VECTORIZED for fixed buffer sends, as it isn't
implemented. Currently this flag is silently ignored
This is in preparation for making these work, but first we
need a fixup so that older kernels will correctly reject them
- Ensure "0" means default for the rx page size
* tag 'io_uring-7.0-20260305' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/zcrx: use READ_ONCE with user shared RQEs
io_uring/mock: Fix typo in help text
io_uring/net: reject SEND_VECTORIZED when unsupported
io_uring: correct comment for IORING_SETUP_TASKRUN_FLAG
io_uring/zcrx: don't set rx_page_size when not requested
According to the FF-A specification (DEN0077, v1.1, §13.7), when
FFA_RXTX_UNMAP is invoked from any instance other than non-secure
physical, the w1 register must be zero (MBZ). If a non-zero value is
supplied in this context, the SPMC must return FFA_INVALID_PARAMETER.
The Arm FF-A driver operates exclusively as a guest or non-secure
physical instance where the partition ID is always zero and is not
invoked from a hypervisor context where w1 carries a VM ID. In this
execution model, the partition ID observed by the driver is always zero,
and passing a VM ID is unnecessary and potentially invalid.
Remove the vm_id parameter from ffa_rxtx_unmap() and ensure that the
SMC call is issued with w1 implicitly zeroed, as required by the
specification. This prevents invalid parameter errors and aligns the
implementation with the defined FF-A ABI behavior.
Fixes: 3bbfe98710 ("firmware: arm_ffa: Add initial Arm FFA driver support")
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
Message-Id: <20260304120953.847671-1-yeoreum.yun@arm.com>
Signed-off-by: Sudeep Holla <sudeep.holla@kernel.org>
kthread_exit became a macro to do_exit in commit 28aaa9c399
("kthread: consolidate kthread exit paths to prevent use-after-free"),
so there is no kthread_exit function BTF ID to resolve. Remove it from
noreturn_deny to avoid resolve_btfids unresolved symbol warnings.
Signed-off-by: Christian Loehle <christian.loehle@arm.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Allow correlating original locals to patched globals, and vice versa.
This is needed when:
1. User adds/removes "static" for a function.
2. CONFIG_LTO_CLANG_THIN promotes local functions and objects to global
and add .llvm.<hash> suffix.
Signed-off-by: Song Liu <song@kernel.org>
Link: https://patch.msgid.link/20260305231531.3847295-8-song@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
correlate_symbols() will always try to match full name first. If there is
no match, try match only demangled_name.
In very rare cases, it is possible to have multiple foo.llvm.<hash> in
the same kernel. Whenever there is ambiguity like this, fail the klp diff.
Signed-off-by: Song Liu <song@kernel.org>
Link: https://patch.msgid.link/20260305231531.3847295-7-song@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
With CONFIG_LTO_CLANG_THIN, it is possible to have global __UNIQUE_ID,
such as:
FUNC GLOBAL HIDDEN 19745 __UNIQUE_ID_quirk_amd_nb_node_458
Also demangle global objects.
Signed-off-by: Song Liu <song@kernel.org>
Link: https://patch.msgid.link/20260305231531.3847295-5-song@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
For klp-build with LTO, it is necessary to correlate demangled symbols,
e.g., correlate foo.llvm.<num 1> and foo.llvm.<num 2>. However, these two
symbols do not have the same str_hash(name). To be able to correlate the
two symbols, calculate hash based on demanged_name, so that these two
symbols have the same hash.
No functional changes intended.
Signed-off-by: Song Liu <song@kernel.org>
Link: https://patch.msgid.link/20260305231531.3847295-4-song@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
With CONFIG_LTO_CLANG_THIN, it is possible to have nested __UNIQUE_ID_,
such as:
__UNIQUE_ID_addressable___UNIQUE_ID_pci_invalid_bar_694_695
To remove both trailing numbers, also remove trailing '_'.
Also add comments to demangle_name().
Signed-off-by: Song Liu <song@kernel.org>
Link: https://patch.msgid.link/20260305231531.3847295-3-song@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
find_global_symbol_by_name() already compares names of the two symbols,
so there is no need to compare them again.
Signed-off-by: Song Liu <song@kernel.org>
Link: https://patch.msgid.link/20260305231531.3847295-2-song@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Commit 25eac74b6b ("objtool: Add section/symbol type helpers")
introduced several helper macros to improve code readability.
Update the remaining open-coded checks in check.c, disas.c, elf.c,
and klp-diff.c to use these new helpers.
Signed-off-by: Wentong Tian <tianwentong2000@gmail.com>
Link: https://patch.msgid.link/20260122144404.40602-1-tianwentong2000@gmail.com
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Currently several resource are fetched before we allocate our instance
data struct. The is unlike most other drivers that fill in the instance
data with resources as they are fetched. Move these down which is more
natural and also removes the need for several temporarily locals.
Signed-off-by: Andrew Davis <afd@ti.com>
Link: https://lore.kernel.org/r/20260302195616.312378-3-afd@ti.com
Signed-off-by: Mathieu Poirier <mathieu.poirier@linaro.org>
