On Sapphire Rapids (SPR) guests where pebs_baseline is not advertised,
running command
$ ./perf record -e cpu/event=0x00,umask=0x01,i\
name=INST_RETIRED.PREC_DIST/p -c 10000 sleep 1
can trigger:
unchecked MSR access error: WRMSR to 0x3f1 ... in\
intel_pmu_pebs_enable_all()
This occurs because SPR-specific PEBS constraints allow fixed-counter
scheduling (for example, INST_RETIRED.PREC_DIST on fixed counter 0).
In guests lacking pebs_baseline, KVM does not support PEBS sampling on
fixed counters, so enabling such events reaches an invalid MSR
programming path.
Starting with Icelake, regardless of whether Extended PEBS or
Architectural PEBS is in use, all counters (including fixed counters)
support PEBS sampling, and the PMU_FL_PEBS_ALL flag is set by default.
As long as PMU_FL_PEBS_ALL is set, constraint lookup automatically falls
back to the corresponding non-PEBS constraints if no matching entry is
found in the PEBS constraints table. Since non-PEBS event constraints
already contain the same fixed-counter constraints, it is safe to remove
these fixed-counter entries from the PEBS constraints table. The fallback
mechanism will ensure that fixed PEBS events are scheduled onto the
correct fixed counters.
So, directly drop the fixed-counter entries from the PEBS constraint
table. Without pebs_baseline, these fixed-counter PEBS events will now
resolve to empty constraints and will not be scheduled or enabled,
thereby avoiding the warning and bypassing the broken guest PEBS path.
Reported-by: Yi Lai <yi1.lai@intel.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260616044654.3468742-7-dapeng1.mi@linux.intel.com
The intel_pmu_init_hybrid() function allocates memory for the
x86_pmu.hybrid_pmu[] array. If this allocation fails under memory
pressure, hybrid PMU initialization will fail.
Currently, the caller does not check the return value of
intel_pmu_init_hybrid(). This can lead to a null-pointer dereference or
invalid memory access when attempting to use the uninitialized array,
potentially triggering a system panic.
Fix this by validating the return value of intel_pmu_init_hybrid().
Additionally, reset x86_pmu.num_hybrid_pmus to 0 on failure, and defer
intel_pmu_arch_lbr_init() until after hybrid PMU initialization
succeeds. This reordering avoids the need to explicitly destroy the
kmem cache if the memory allocation fails.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260616044654.3468742-6-dapeng1.mi@linux.intel.com
Before Arch LBR gained CPL filtering support, a user-only branch stack
could still contain kernel addresses. As a result, kernel branch records
may be exposed to user space even when PERF_SAMPLE_BRANCH_USER is
requested.
For example, on Intel Tiger Lake, the following command can still report
SYSRET/ERET entries with kernel-space from addresses:
$ ./perf record -e cycles:p -o - --branch-filter any,save_type,u -- \
./perf bench syscall basic --loop 1000 | \
./perf script -i - --fields brstack|tr ' ' '\n'| \
grep -E '0x[89a-f][0-9a-f]{15}'
Total time: 0.000 [sec]
0.219000 usecs/op
4,566,210 ops/sec
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.551 MB - ]
0xffffffff93c001c8/0x7f12a2b1d647/P/-/-/16959/SYSRET/-
0xffffffff93c001c8/0x7f12a2b1d5c2/P/-/-/17535/SYSRET/-
0xffffffff93c01928/0x7f12a2861000/P/-/-/6719/ERET/-
0xffffffff93c01928/0x7f12a297a000/P/-/-/8575/ERET/-
The problem is that intel_pmu_lbr_filter() does not fully validate the
privilege level of sampled entries. It filters some mismatches based on
the branch type and the to address, but it does not reject entries whose
from address violates the requested branch privilege filter.
Fix this by extending software filtering to validate both from and to
addresses against br_sel. Any LBR entry contains kernel address does not
match the requested user filter is dropped. This prevents kernel
addresses from appearing in user-only branch stacks.
Fixes: 47125db27e ("perf/x86/intel/lbr: Support Architectural LBR")
Reported-by: Ian Rogers <irogers@google.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260616044654.3468742-5-dapeng1.mi@linux.intel.com
intel_pmu_lbr_filter() currently assumes arch-LBR provides hardware
branch-type decoding and skips software decoding on that path.
However, arch-LBR may not always expose branch-type information. In
that case, treating entries as hardware-decoded can misclassify sampled
branches (for example, defaulting to JCC), which breaks branch-type
filtering results.
Fix this by using software branch-type decoding when hardware
branch-type decoding is unavailable (that is, when x86_lbr_type is not
enabled). This keeps branch classification and filtering behavior
correct across arch-LBR configurations.
Since x86_lbr_type is only set to true when arch-LBR is supported,
it's unnecessary to check if the CPU has the arch-LBR feature.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260616044654.3468742-4-dapeng1.mi@linux.intel.com
After introducing the RDPMC user disable feature, user-space RDPMC may
return 0 instead of the actual event count. This creates an inconsistency
with cap_user_rdpmc, where cap_user_rdpmc is set, but user-space RDPMC
only returns 0.
To accurately represent the user-space RDPMC capability, update
cap_user_rdpmc (depending on PERF_EVENT_FLAG_USER_READ_CNT) according to
the RDPMC user disable state. If RDPMC user disable is enabled,
cap_user_rdpmc is updated to false eventually, allowing user-space
programs to fall back to the read() syscall to obtain the real event
count.
Because PERF_EVENT_FLAG_USER_READ_CNT is evaluated in
x86_pmu_event_init(), move intel_pmu_update_rdpmc_user_disable()
earlier into intel_pmu_hw_config(). This ensures that the user-disable
state is updated before updating PERF_EVENT_FLAG_USER_READ_CNT. Note that
since event->ctx is not yet assigned at this stage, use the
PERF_ATTACH_TASK flag to detect whether the event is task-attached.
While at it, fix the indentation of x86_pmu_has_rdpmc_user_disable()
to adhere to the kernel coding style.
Fixes: 59af95e028 ("perf/x86/intel: Add support for rdpmc user disable feature")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260616044654.3468742-3-dapeng1.mi@linux.intel.com
AnyThread mode deprecation is enumerated by CPUID.0AH:EDX[15] instead of
PERF_CAPABILITIES MSR. It's not a good practice to define a bit to
represent "anythread deprecation" in perf_capabilities. It leads to the
anythread_deprecated bit could be overwritten by the real value of
PERF_CAPABILITIES MSR, just like the below code in update_pmu_cap() does.
if (!intel_pmu_broken_perf_cap()) {
/* Perf Metric (Bit 15) and PEBS via PT (Bit 16) are hybrid enumeration */
rdmsrq(MSR_IA32_PERF_CAPABILITIES, hybrid(pmu, intel_cap).capabilities);
}
It leads to the anythread_deprecated bit is cleared to 0 and the "any"
attribute is incorrectly shown in the /sys/devices/cpu/format/ folder on
these support Perfmon v6 platforms, like Clearwater Forest.
$ grep . /sys/devices/cpu/format/*
/sys/devices/cpu/format/acr_mask:config2:0-63
/sys/devices/cpu/format/any:config:21
/sys/devices/cpu/format/cmask:config:24-31
So remove the anythread_deprecated bit from perf_capabilities structure
and directly depends on CPUID.0AH:EDX[15] to judge if anythread is
deprecated.
Fixes: cadbaa039b ("perf/x86/intel: Make anythread filter support conditional")
Reported-by: Namhyung Kim <namhyung@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Zide Chen <zide.chen@intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260616044654.3468742-2-dapeng1.mi@linux.intel.com
In wx_set_vmdq_queues(), the VMDQ mask was not set for the devices not
supporting WX_FLAG_MULTI_64_FUNC, i.e., NGBE devices. A mask of 0 causes
__ALIGN_MASK(1, ~vmdq->mask) to return 0, which incorrectly sets
q_per_pool to 0 in wx_write_qde().
Fix the VMDQ 1-queue mask to 0x7F then ensures that __ALIGN_MASK(1,
~0x7F) correctly evaluates to 1.
Fixes: c52d4b8989 ("net: libwx: Redesign flow when sriov is enabled")
Signed-off-by: Jiawen Wu <jiawenwu@trustnetic.com>
Reviewed-by: Larysa Zaremba <larysa.zaremba@intel.com>
Link: https://patch.msgid.link/161F704D2C983E2C+20260626092530.551028-1-jiawenwu@trustnetic.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
ffa_partition_info_get() passes uuid_str directly to uuid_parse()
without a NULL check. When a caller passes NULL, uuid_parse() ->
__uuid_parse() -> uuid_is_valid() dereferences the pointer, causing
a kernel panic:
| Unable to handle kernel NULL pointer dereference at virtual address
| 0000000000000040
| pc : uuid_parse+0x40/0xac
| lr : ffa_partition_info_get+0x1c/0x94 [arm_ffa]
Add a NULL guard before uuid_parse() so a NULL argument returns
-ENODEV instead of crashing. Callers are expected to always supply
a valid partition UUID, so NULL is not a supported input.
Fixes: d0c0bce831 ("firmware: arm_ffa: Setup in-kernel users of FFA partitions")
Signed-off-by: Unnathi Chalicheemala <unnathi.chalicheemala@oss.qualcomm.com>
Link: https://patch.msgid.link/20260617-ffa_partition_nullptr_fix-v2-1-bc801b4ce34c@oss.qualcomm.com
Signed-off-by: Sudeep Holla <sudeep.holla@kernel.org>
The size of the mmap'ed framebuffer could be smaller than the minimum
required framebuffer size. Validate the resource size against the
framebuffer size.
Buggy firmware that triggers this check should be fixed up with a quirk
on a case-by-case base.
v3:
- fix size checks (Sashiko)
- improve error message for alignment check (Thierry)
- do not store aligned size value
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Suggested-by: Sashiko <sashiko-bot@kernel.org>
Reviewed-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Javier Martinez Canillas <javierm@redhat.com>
Link: https://patch.msgid.link/20260625094509.157581-6-tzimmermann@suse.de
Validate the computed stride against the maximum value INT_MAX.
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Javier Martinez Canillas <javierm@redhat.com>
Fixes: 7bfa5c7b28 ("drm/simpledrm: Compute linestride with drm_format_info_min_pitch()")
Cc: <stable@vger.kernel.org> # v6.1+
Link: https://patch.msgid.link/20260625094509.157581-5-tzimmermann@suse.de
The helper simplefb_get_validated_int() is only a wrapper around
drm_sysfb_get_validated_int(). Inline the function into its callers.
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Javier Martinez Canillas <javierm@redhat.com>
Link: https://patch.msgid.link/20260625094509.157581-4-tzimmermann@suse.de
Validate the panel size from the device-tree node against the
limitations of struct drm_display_mode. The type only stores sizes
in 16-bit fields. Fail transparently on errors; do not warn.
v3:
- move comments to a more prominent place (Thierry)
v2:
- only use initialized values in debugging output (Sashiko)
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Javier Martinez Canillas <javierm@redhat.com>
Fixes: 2a6d731a8f ("drm/simpledrm: Allow physical width and height configuration via panel node")
Cc: Rayyan Ansari <rayyan@ansari.sh>
Cc: <stable@vger.kernel.org> # v6.4+
Link: https://patch.msgid.link/20260625094509.157581-3-tzimmermann@suse.de
Validate the framebuffer size from the firmware against the
limitations of struct drm_display_mode. The type only stores sizes
in 16-bit fields. Fail probing on errors.
v2:
- remove unused function simplefb_get_validated_int0() (Sashiko)
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Maxime Ripard <mripard@kernel.org>
Reviewed-by: Javier Martinez Canillas <javierm@redhat.com>
Fixes: 11e8f5fd22 ("drm: Add simpledrm driver")
Cc: <stable@vger.kernel.org> # v5.14+
Fixes: 11e8f5fd22 ("drm: Add simpledrm driver")
Link: https://patch.msgid.link/20260625094509.157581-2-tzimmermann@suse.de
Mark CONFIG_SYSFB_SIMPLEFB as deprecated. Enabling it allows to
run simpledrm and simplefb on EFI/VESA framebuffers. Doing this
is discouraged in favor of using efidrm and vesadrm.
v2:
- resolve conflicting help texts (Sashiko)
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Javier Martinez Canillas <javierm@redhat.com>
Link: https://patch.msgid.link/20260625065724.15794-1-tzimmermann@suse.de
Set the GDM maximum receive size to AIROHA_MAX_RX_SIZE unconditionally
during hardware initialization instead of updating it according to the
configured MTU. This avoids dropping incoming frames that exceed the
current MTU but could still be processed by the networking stack, which
is able to fragment the reply on the TX side (e.g. ICMP echo requests).
Move the per-port MTU configuration to the PPE egress path where it
belongs, and set the tx frame size running airoha_ppe_set_xmit_frame_size()
to dynamically track the maximum MTU across running interfaces sharing
the same PPE instance.
Fix the PPE MTU register addressing to pack two port entries per
register word and add WAN_MTU0 configuration for non-LAN GDM devices.
Fixes: 54d989d58d ("net: airoha: Move min/max packet len configuration in airoha_dev_open()")
Tested-by: Madhur Agrawal <madhur.agrawal@airoha.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Link: https://patch.msgid.link/20260625-airoha-fix-rx-max-len-v1-1-45b9b827358d@kernel.org
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
While the panel_type from LFP Data Block is range checked, panel_type2
is not. Add a few helpers for range checking, and use them to not only
check panel_type2, but also improve clarity and correctness in the panel
type selection.
Discovered using AI-assisted static analysis confirmed by Intel Product
Security.
v2:
- Fix commit message typo (Michał)
- Add is_panel_type_pnp() (Ville)
Reported-by: Martin Hodo <martin.hodo@intel.com>
Fixes: 6434cf6300 ("drm/i915/bios: calculate panel type as per child device index in VBT")
Cc: stable@vger.kernel.org # v6.0+
Cc: Animesh Manna <animesh.manna@intel.com>
Cc: Ville Syrjälä <ville.syrjala@intel.com>
Reviewed-by: Michał Grzelak <michal.grzelak@intel.com> # v1
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patch.msgid.link/20260626140155.1389655-1-jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Linus Walleij expresed the need for a dedicated maintainer for Qualcomm
pin control drivers which receive a lot more changes lately. Qualcomm
volunteered me for this role so add an entry to MAINTAINERS.
Signed-off-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
Signed-off-by: Linus Walleij <linusw@kernel.org>
for_each_child_of_node() holds a reference on the iterator node that
must be released on early return. imx1_pinctrl_dt_is_flat_functions()
has two early return paths inside the loop that skip this cleanup.
Replace both loops with the scoped variant so that the reference is
automatically dropped when the iterator goes out of scope.
Fixes: 63d2059cd6 ("pinctrl: imx1: Allow parsing DT without function nodes")
Signed-off-by: Felix Gu <ustc.gu@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Linus Walleij <linusw@kernel.org>
MSR and MMIO uncore PMUs are currently registered at module init time
and appear in sysfs even when no PMU boxes are functional.
Apply the same lazy registration model used by PCI uncore PMUs: the
PMU is registered when the first box is successfully initialized, and
unregistered when the last box exits. If a box fails to initialize on
a subsequent die, the PMU is marked broken but remains registered to
avoid disrupting any in-flight perf events.
Box allocation and freeing remain at module init/exit time to avoid
repeated kfree/alloc cycles across CPU offline/online events.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260611160033.66760-9-zide.chen@intel.com
In uncore_event_cpu_online(), uncore_box_ref() was called before
uncore_change_context(). uncore_box_ref() gates on box->cpu >= 0,
but box->cpu is still -1 at that point because uncore_change_context()
has not run yet. As a result, the box is never initialized on the
first CPU to come online in a die, leaving it permanently
uninitialized in the single-CPU-per-die case.
Thus, box->refcnt is one count below the true value, and in the CPU
offline path, the box will be torn down on the second-to-last CPU.
In uncore_event_cpu_offline(), uncore_box_unref() was called after
uncore_change_context(), so box->cpu is already -1 when the collector
CPU goes offline, which prevents it from tearing down the box.
Fix by swapping the call order in both paths so that
uncore_box_{ref,unref}() runs at the point where box->cpu reflects
the correct context.
Move allocate_boxes() out of uncore_box_ref() to enable this
reordering.
Fixes: c74443d92f ("perf/x86/uncore: Support per PMU cpumask")
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Ian Rogers <irogers@google.com>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260611160033.66760-8-zide.chen@intel.com
Replace the boolean 'registered' field in intel_uncore_pmu with an
unsigned long 'flags' field, and add a PMU_BROKEN flag to track box
setup failures. The broken flag is sticky, meaning it is cleared only
by a module reload or system reboot.
Broken PMUs are skipped in the CPU hotplug and box allocation paths.
When any box fails to initialize, the PMU is marked broken. Broken
PMUs reject new event assignments and skip future box setup attempts.
If the PMU was already registered, it remains so to avoid disrupting
in-flight events on other boxes.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260611160033.66760-7-zide.chen@intel.com
The PCI uncore PMU path already implements a lazy registration model:
the PMU is registered when the first active box appears and
unregistered when the last active box is removed.
Factor this registration management into a shared helper, so the same
code can be reused by the MSR and MMIO paths in later changes.
No functional change intended.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Ian Rogers <irogers@google.com>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260611160033.66760-6-zide.chen@intel.com
uncore_event_cpu_online() returns -ENOMEM early when both the MSR and
MMIO box allocations fail. This also aborts PCI uncore setup, even
though PCI PMUs are independent of the MSR/MMIO paths.
Remove the early return so PCI uncore setup always runs regardless
of whether MSR or MMIO box allocation succeeds.
Fixes: 3da04b8a00 ("perf/x86/intel/uncore: Support MMIO type uncore blocks")
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Ian Rogers <irogers@google.com>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260611160033.66760-5-zide.chen@intel.com
The init_box() callback currently returns void, so initialization
failures are silently ignored and the box is still marked initialized.
Change the callback to return int so platform code can report errors
back to the common uncore layer.
Update uncore_box_init() to set the initialized flag only when
init_box() succeeds. Because box->refcnt guarantees that at most
one CPU calls uncore_box_init() for a given box at a time, plain
__set_bit() is safe for the initialized flag without atomic overhead.
Convert all init_box() implementations to return 0 on success or a
negative error code on failure. This is a prerequisite for propagating
initialization errors to the caller so they can be handled properly.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Ian Rogers <irogers@google.com>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260611160033.66760-4-zide.chen@intel.com
Fix typo UNCORE_BOX_FLAG_INITIATED to UNCORE_BOX_FLAG_INITIALIZED.
Rename the 'id' parameter in uncore_box_{ref,unref}() to 'die' to
reflect its actual meaning and be consistent with other functions.
box->refcnt is incremented in the PCI PMU register path but has never
been checked or decremented. Although for PCI PMUs box->refcnt
effectively tracks only a single user, add atomic_dec_return() in the
PCI PMU unregister path to make the reference counting complete and
consistent.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260611160033.66760-3-zide.chen@intel.com
When uncore_pci_pmu_register() fails, pmu->boxes[die] is set to NULL
before returning. In the uncore_pci_remove() path, this causes
uncore_pci_pmu_unregister() to be skipped entirely, leaking
pmu->activeboxes. In the uncore_bus_notify() path,
uncore_pci_pmu_unregister() may still be called and must exit early
when pmu->boxes[die] is NULL to avoid a NULL pointer dereference, and
to ensure activeboxes is only decremented for a previously active box.
Additionally, since pci_get_drvdata() returns NULL on registration
failure, uncore_pci_remove() can no longer treat NULL drvdata as an
indicator of an auxiliary PCI device. Remove the associated
WARN_ON_ONCE().
Signed-off-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://sashiko.dev/#/patchset/20260512233048.9577-1-zide.chen@intel.com?part=1
Link: https://patch.msgid.link/20260611160033.66760-2-zide.chen@intel.com
nr_migrations_cold, nr_wakeups_passive and nr_wakeups_idle are not
being updated anywhere. So remove them.
These are per process stats. So updating sched stats version isn't
necessary.
Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260625124648.802832-2-sshegde@linux.ibm.com
psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative. A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.
Measured on a 176-thread AMD EPYC 9D64 server running a compute
intensive production workload, instrumented with bpftrace over a 30s
window (irq_time_read() read directly from the per-CPU cpu_irqtime so
that delta == 0 and delta < 0 could be separated):
@total 17,229,311 (100.0%)
@ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0
@ret_samegrp 323,299 ( 1.9%) same cgroup as prev
@reached_delta 9,041,817 ( 52.5%)
@delta_positive 6,358,192 ( 36.9%) real work
@delta_zero 2,683,625 ( 15.6%) work wasted (this patch)
@delta_negative (0) ( 0.0%) monotonic clock
So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
calls that get past the early returns - hit the delta == 0 case;
delta < 0 did not occur once in the 30 s window. Under the current
code each of those ~89 k calls per second performs the full seqcount
write + cpu_clock() read + cgroup-chain walk just to add 0 to every
group's PSI_IRQ_FULL counter.
Extend the early-return to also cover delta == 0. rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change. The existing behaviour for
delta > 0 is untouched.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Link: https://patch.msgid.link/20260617175219.2494857-2-usama.arif@linux.dev
Because active_balance is initialized to 0, and need_active_balance()
is a pre-condition for setting it to 1, the condition '!active_balance
|| need_active_balance()' is a truism and can be removed.
Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260617072151.1173416-3-jackzxcui1989@163.com
Active load balancing relies on migration threads, which temporarily preempt
tasks on the source runqueue (src_rq). This preemption can negatively impact
overall system performance. The active balancing logic includes a check to
verify whether the current task (curr) on src_rq can actually run on the
destination runqueue (dst_rq). We have observed that when curr is a CFS task
and its on_rq flag is 0, the active balancing failure rate is exceptionally
high. The following table summarizes test data collected over 300 seconds on an
18-CPU platform under a specific fillback task scenario:
fair: busiest->curr->sched_class == &fair_sched_class
on_rq: busiest->curr->on_rq
total: active balance count triggered of correspondent type
fail: fail to migrate one task in active_load_balance_cpu_stop()
fair && !on_rq !fair && !on_rq
domain total fail total fail
cpu0 0x00003 0 0 0 0
cpu0 0x3ffff 33 33 1 1
cpu1 0x00003 0 0 0 0
cpu1 0x3ffff 42 42 0 0
cpu2 0x0003c 4 4 0 0
cpu2 0x3ffff 12 12 0 0
cpu3 0x0003c 3 3 0 0
cpu3 0x3ffff 8 7 0 0
cpu4 0x0003c 2 2 0 0
cpu4 0x3ffff 5 4 0 0
cpu5 0x0003c 4 4 0 0
cpu5 0x3ffff 8 8 0 0
cpu6 0x003c0 60 60 0 0
cpu6 0x3ffff 28 27 0 0
cpu7 0x003c0 194 184 0 0
cpu7 0x3ffff 35 35 1 1
cpu8 0x003c0 240 228 0 0
cpu8 0x3ffff 28 28 0 0
cpu9 0x003c0 0 0 0 0
cpu9 0x3ffff 10 10 0 0
cpu10 0x03c00 52 50 0 0
cpu10 0x3ffff 0 0 0 0
cpu11 0x03c00 70 68 0 0
cpu11 0x3ffff 1 1 0 0
cpu12 0x03c00 73 72 0 0
cpu12 0x3ffff 0 0 0 0
cpu13 0x03c00 79 76 0 0
cpu13 0x3ffff 0 0 0 0
cpu14 0x3c000 0 0 0 0
cpu14 0x3ffff 57 55 1 0
cpu15 0x3c000 53 52 1 0
cpu15 0x3ffff 30 29 0 0
cpu16 0x3c000 344 341 10 6
cpu16 0x3ffff 103 100 2 1
cpu17 0x3c000 183 179 2 2
cpu17 0x3ffff 78 77 0 0
sum 1839 1791 18 11
In __schedule(), before curr is updated to next, pick_next_task() invokes
sched_balance_rq(). This function temporarily unlocks and relocks the runqueue,
creating a window where other CPUs may observe rq->curr->on_rq as 0.
We can safely skip active balancing when src_rq->curr->on_rq == 0, as other
eligible tasks have likely already been evaluated.
We retain the affinity check on dst_rq to trigger active balancing, since such
tasks are often woken by (or wake up) tasks on src_rq that share similar
affinity constraints. Furthermore, detach_tasks() releases the runqueue lock;
any tasks awakened during this window may preempt the previous CFS task. My
testing (data not shown) indicates that active balancing succeeds in 98.4% of
cases where !fair && on_rq.
This scenario does not require a stop-work callback, but would necessitate an
additional detach/attach path. As Valentin and Vincent have already discussed,
this addition does not appear justified at this time (see [1]).
Since can_migrate_task() already checks on_cpu during the cfs_tasks traversal,
adding an on_rq check will have negligible performance overhead due to cache
locality.
There are two reasons for not combining the on_rq check with the
cpumask_test_cpu() check:
- Avoiding new scenarios that would skip the logic for resetting
balance_interval to min_interval.
- The existing check for whether the busiest CPU recently triggered active
load balancing already filters more cases than the on_rq check.
[1]: https://lore.kernel.org/lkml/20190815145107.5318-5-valentin.schneider@arm.com/
Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Valentin Schneider <vschneid@redhat.com>
Link: https://patch.msgid.link/20260617072151.1173416-2-jackzxcui1989@163.com
When a task with a shorter slice is enqueued, we protect the running
task which has a longer slice until it becomes ineligible instead of a
full slice in order to speedup the switch to other tasks until the task
with the shortest slice is scheduled. This helps to the task to not wait
too many full slices before running.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-7-vincent.guittot@linaro.org
Even if p will not preempt current, it modifies the avg_vruntime and
possibly the min slice. Make sure to update the slice protection with the
updated figures. As an example, Batch and Sched Idle tasks can otherwise
get a larger lag than their slice and finaly delay the scheduling of a
normal task, which deadline will be a later.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-6-vincent.guittot@linaro.org
If a short slice task will not be the next to be picked but is eligible,
we cancel the slice protection to speedup the time when the short slice
task will be the next to run.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-5-vincent.guittot@linaro.org
Even if resched is already set, we might want to update or even cancel
the slice protection and ensure that the newly waking task will be the
next one to run.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-4-vincent.guittot@linaro.org
Take into account the lag of current task when updating the slice
protection in order to ensure that the absolute value of lags will remain
in the range [0 : slice+tick]
A task that already has a negative lag will see its protection reduced
whereas a task with positive lag will keep a full slice protection.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-3-vincent.guittot@linaro.org
If a shorter slice task can preempt current at wakeup, we make sure that
the decision will not be overwritten in between by setting the task as the
next buddy. This still implies that the waking task remains eligible when
the scheduler will actually pick the next task to run.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-2-vincent.guittot@linaro.org
On a tangential note, I just noticed set_preempt_buddy() has two unused
parameters. Seems to have been like that since it was introduced in
commit e837456fdc ("sched/fair: Reimplement NEXT_BUDDY to align with
EEVDF goals").
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically
the complaint is latencies and/or overhead. The latency issue is due
to the intermediate entries that represent a combination of tasks and
thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including
the intermediate enqueue/dequeue but move the actual EEVDF runqueue
outside. This means things like the shares_weight approximation are
fully preserved.
That is, given a hierarchy like:
R
|
se--G1
/ \
G2--se se--G3
/ \ |
T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of
cfs_rq/se go unused for the intermediates and are instead connected
like:
_R_
/ | \
T1 T2 T3
Since the effective weight of the entities is determined by the
hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (se->h_load) is computed from the
hierarchical fraction: se->load / cfs_rq->load.
Since EEVDF is now exclusively operating on rq->cfs, it needs to
consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
only tasks can get delayed, simplifying some of the cgroup cleanup.
One place where additional information was required was
set_next_task() / put_prev_task(), where we need to track 'current'
both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
(cfs_rq->curr).
As a result of only having a single level to pick from, much of the
complications in pick_next_task() and preemption go away.
Since many of the hierarchical operations are still there, this won't
immediately fix the performance issues, but hopefully it will fix some
of the latency issues.
TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124052.227463677%40infradead.org
For all the reasons described in the preceding patches, the way cgroup
weight is computed is problematic. However, changing it is bound to
also lead to trouble. Esp. since people might have taken to inflating
the weight value where they can.
Since things are configurable, change the default and hope this serves
more people than it hurts, esp. in the longer run.
Specifically, this prepares for a flattened runqueue, where the hierarchical
weight becomes far more important (F_g^d terms), so getting rid of small F_g is
imperative.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124052.080482755%40infradead.org
Since we are exploring this space; include a scheme that scales by total number
of runnable tasks. This results in:
F_g_n' = M * F_g_n
This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact).
And while that sounds odd, it actually has a fairly straight foward meaning for
"cpu.weight": average weight per member task.
This is an entirely valid and workable option, it is however wildly different
from the traditional meaning.
Included for completeness (and curiosity).
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.921991975%40infradead.org
Improve upon the previous scheme ("max") by no longer assuming maximal
concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles
the low concurrency cases more gracefully:
F_g_n' = min(M, N) * F_g_n
Notably this is the first mode where:
avg(F_g_n) = 1
In the single task case it reduces to ("smp") and then it nicely scales up
until it hits N, where it behaves like ("max").
This is no longer clipped at nice -20. Strictly speaking it isn't different
from the normal SMP scenario where all tasks are extremely unbalanced. There
are no unnatural inflations in this scheme.
The meaning of "cpu.weight" would be: weight per active CPU.
NOTE: Compute the group wide number of tasks by extending the tg->load_avg
computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on
cfs_rq->h_nr_running.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.740585993%40infradead.org
In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N',
assume each cgroup is maximally concurrent and distrubute 'N*weight', such
that:
F_g_n' = N * F_g_n
Giving:
avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
And while this sounds like it solves things, remember what that ~ meant. There
is the corner case when a cgroup is minimally loaded, eg a single runnable
task, therefore limit the CPU fraction to that of a nice -20 task to avoid
getting too much load.
This last bit is what makes it different from a previous proposal to allow
raising cpu.weight to '100 * N', that would not limit the mininal concurrency
case and results in a very large F_g_n. And just like F_g_n << 1 is
problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the
kthreads, but it also risks overflowing the load values).
So while this might appear to be a better scheme than the current default
scheme, it doesn't really handle less than maximal concurrency nicely -- it
clips and introduces artificially large weights. So where the traditional SMP
mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime
and vice-versa.
The meaning of "cpu.weight" would be: weight per allowed CPU.
Included for completeness (and infrastructure).
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.589618504%40infradead.org
Instead of calculating the proportional fraction of the group weight for each
CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single
task in a cgroup competes on equal footing to a task in a level above.
However, as already explored, this is not a very good policy because it gets
the SMP weight distribution wrong. Included for completeness.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.450303977%40infradead.org
The effective task weight (W_t') for a task in cgroup g on CPU n is given by:
W_t
W_t' = W_g * F_g_n * ----------
\Sum W_t_n
Where W_g is the group's weight (cpu.weight), F_g_n is the fraction of the
group weight for CPU n and W_t/W is the relative weight of this task against
all other tasks in the same group on the same CPU.
Furthermore, this makes:
\Sum W_t_n
F_g_n = ----------
\Sum W_t
The fraction of weight inside the group of CPU n against the whole group.
The problem is with F_g_n, the primary goal of this fraction is to make sure
that the relative weight of tasks, when distributed over CPUs is maintained.
For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3
distribution, then if F_g_n would simply be 1 (no weight re-distribution) the
effective relative weights (W_t') of the tasks in our group would be:
CPU0 CPU1
W_g W_g/3
W_g/3
W_g/3
IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 tasks
on CPU1 combined. However, with the weight redistribution, this becomes:
CPU0 CPU1
W_g/4 W_g/4
W_g/4
W_g/4
All tasks are equal weight (as intended). However, as is already evident from
this example, the more CPUs you add, the smaller F_g_n becomes, which creates a
disparity against tasks not in our group.
Specifically:
avg(F_g_n) ~ 1/N
This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete
fairly across hierarchy levels.
*Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that when
there are at least N runnable tasks, the average of this fraction tends to 1/N.
For a hierarchy of depth d, this gets even worse, since that gets terms on the
order of:
avg(F_g_n)^d ~ 1/(N^d)
Given fixed point arithmetic, this also leads to numerical trouble.
However, the meaning of "cpu.weight" is simple and intiutive: the total weight
of the cgroup. But as explored above, there is deception in this simplicity.
Prepare to add a few alternative methods for distributing weight.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260605124051.338602724%40infradead.org
A divide-by-zero crash is observed when running hackbench:
[14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted 7.1.0-rc2+
[14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0
[14697.506799] <TASK>
[14697.507411] __dequeue_task+0x2b4/0xc70
[14697.508677] dequeue_task_fair+0x36/0x370
[14697.509047] dequeue_task+0x101/0x2f0
[14697.509426] __schedule+0x1b1/0x1a00
[14697.510868] anon_pipe_read+0x3da/0x450
[14697.511400] vfs_read+0x361/0x390
[14697.512053] __x64_sys_read+0x19/0x30
The divide-by-zero happens here:
if (scale_load_down(gcfs_rq->load.weight)) {
load_sum = div_u64(gcfs_rq->avg.load_sum,
scale_load_down(gcfs_rq->load.weight));
}
gcfs_rq->load.weight is an insane large value and is truncated
to the lower 32 bits by div_u64, which happen to be 0.
Using AI for investigation, the cause is a u32 overflow in
update_tg_cfs_runnable(), and flat pickup became a victim when using
tg_tasks():
u32 new_sum, divider;
...
new_sum = se->avg.runnable_avg * divider; <-- boom
The following sequence shows how this triggers the crash:
propagate_entity_load_avg()
update_tg_cfs_runnable() # u32 overflow corrupts runnable_sum
__update_load_avg_cfs_rq()
___update_load_avg() # computes insane runnable_avg
update_tg_load_avg() # propagates to tg->runnable_avg
update_cfs_group()
calc_concur_shares()
tg_tasks() # long-to-int truncation, negative nr
reweight_entity() # corrupted se->load.weight
update_load_add() # corrupted cfs_rq->load.weight
propagate_entity_load_avg()
update_tg_cfs_load()
div_u64() # divide-by-zero
Fix by widening new_sum from u32 to u64 (no need to force tg_tasks()
to return unsigned long after this fix)
Fixes: 95246d1ec8 ("sched/pelt: Relax the sync of runnable_sum with runnable_avg")
Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/a22eea2b-4c4a-4623-9a44-d7b18c0c91c8@intel.com
The way wakeup_preempt() works since commit 704069649b ("sched/core: Rework
sched_class::wakeup_preempt() and rq_modified_*()") is that it will call
rq->next_class->wakeup_preempt(rq, p) when p is of an equal or higher class,
and raise ->next_class when higher.
This means that:
running idle task
wakeup fair-A
(next_class == idle)
if (sched_class_above(fair, idle)) {
wakeup_preempt_idle(fair-A);
resched_curr(rq);
next_class = fair;
}
wakeup fair-B
(next_class == fair)
if (fair == fair)
wakeup_preempt_fair(fair-B);
(but current is idle)
All wakeup_preempt_$class() methods, except for wakeup_preempt_scx() (for whoem
this was build) ignore cross-class wakeups by testing if @p is of the right
class, but per the above case, it also should check current.
This is mostly harmless in the current form, but will lead to trouble with
later patches.
Fixes: 704069649b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260626074605.GB2568396%40noisy.programming.kicks-ass.net
