Commit graph

1,461,726 commits

Author SHA1 Message Date
Thomas Zimmermann
59fedf46f7 drm/ast: Split ast_detect_tx_chip() per chip generation
Gen4 and later models detect the TX chip from VGACRD1, while earlier
models detect from VGACRA3. Split up the detection helper into
two separate helpers. Use SZ_ constants instead of plain numbers.

Then inline the call into its only caller ast_device_create(). When
ast_device_create() gets split up per Gen, either call will remain.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Jocelyn Falempe <jfalempe@redhat.com>
Link: https://lore.kernel.org/r/20250922083708.45564-4-tzimmermann@suse.de
2025-09-29 13:28:09 +02:00
Thomas Zimmermann
bcb0119134 drm/ast: Move mode-detection helpers to Gen2 source files
Wide-screen modes are only available on Gen2 and later. Move the
detection helpers to the appropriate source file.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Jocelyn Falempe <jfalempe@redhat.com>
Link: https://lore.kernel.org/r/20250922083708.45564-3-tzimmermann@suse.de
2025-09-29 13:28:08 +02:00
Thomas Zimmermann
d9a9ea0fba drm/ast: Move display-clock tables to per-Gen source files
Move display-clock tables to the appropriate per-Gen source files.
The tables are almost identical, except for mode entries 0x17 and
0x1a. Rename to tables to match common style.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Jocelyn Falempe <jfalempe@redhat.com>
Link: https://lore.kernel.org/r/20250922083708.45564-2-tzimmermann@suse.de
2025-09-29 13:28:07 +02:00
Rafael J. Wysocki
40d2cf9c3c Merge branch 'pm-tools'
Merge power management utilities updates for 6.18-rc1:

 - Fix and clean up the x86_energy_perf_policy utility and update its
   documentation (Len Brown, Kaushlendra Kumar)

 - Fix incorrect sorting of PMT telemetry in turbostat (Kaushlendra
   Kumar)

 - Fix incorrect size in cpuidle_state_disable() and the error return
   value of cpupower_write_sysfs() in cpupower (Kaushlendra Kumar)

* pm-tools:
  tools/power x86_energy_perf_policy.8: Emphasize preference for SW interfaces
  tools/power x86_energy_perf_policy: Add make snapshot target
  tools/power x86_energy_perf_policy: Prefer driver HWP limits
  tools/power x86_energy_perf_policy: EPB access is only via sysfs
  tools/power x86_energy_perf_policy: Prepare for MSR/sysfs refactoring
  tools/power x86_energy_perf_policy: Enhance HWP enable
  tools/power x86_energy_perf_policy: Enhance HWP enabled check
  tools/power x86_energy_perf_policy: Fix incorrect fopen mode usage
  tools/power turbostat: Fix incorrect sorting of PMT telemetry
  tools/cpupower: Fix incorrect size in cpuidle_state_disable()
  tools/cpupower: fix error return value in cpupower_write_sysfs()
2025-09-29 13:10:10 +02:00
Rafael J. Wysocki
f58f86df6a Merge branches 'pm-core', 'pm-runtime' and 'pm-sleep'
Merge changes related to system sleep and runtime PM framework for
6.18-rc1:

 - Annotate loops walking device links in the power management core
   code as _srcu and add macros for walking device links to reduce the
   likelihood of coding mistakes related to them (Rafael Wysocki)

 - Document time units for *_time functions in the runtime PM API (Brian
   Norris)

 - Clear power.must_resume in noirq suspend error path to avoid resuming
   a dependant device under a suspended parent or supplier (Rafael
   Wysocki)

 - Fix GFP mask handling during hybrid suspend and make the amdgpu
   driver handle hybrid suspend correctly (Mario Limonciello, Rafael
   Wysocki)

 - Fix GFP mask handling after aborted hibernation in platform mode and
   combine exit paths in power_down() to avoid code duplication (Rafael
   Wysocki)

 - Use vmalloc_array() and vcalloc() in the hibernation core to avoid
   open-coded size computations (Qianfeng Rong)

 - Fix typo in hibernation core code comment (Li Jun)

 - Call pm_wakeup_clear() in the same place where other functions that do
   bookkeeping prior to suspend_prepare() are called (Samuel Wu)

* pm-core:
  PM: core: Add two macros for walking device links
  PM: core: Annotate loops walking device links as _srcu

* pm-runtime:
  PM: runtime: Documentation: ABI: Document time units for *_time

* pm-sleep:
  PM: hibernate: Combine return paths in power_down()
  PM: hibernate: Restrict GFP mask in power_down()
  PM: hibernate: Fix pm_hibernation_mode_is_suspend() build breakage
  drm/amd: Fix hybrid sleep
  PM: hibernate: Add pm_hibernation_mode_is_suspend()
  PM: hibernate: Fix hybrid-sleep
  PM: sleep: core: Clear power.must_resume in noirq suspend error path
  PM: sleep: Make pm_wakeup_clear() call more clear
  PM: hibernate: Fix typo in memory bitmaps description comment
  PM: hibernate: Use vmalloc_array() and vcalloc() to improve code
2025-09-29 12:54:01 +02:00
Rafael J. Wysocki
101642e43d Merge branches 'pm-cpuidle' and 'pm-powercap'
Merge cpuidle and power capping changes for 6.18-rc1:

 - Fail cpuidle device registration if there is one already to avoid
   sysfs-related issues (Rafael Wysocki)

 - Use sysfs_emit()/sysfs_emit_at() instead of sprintf()/scnprintf() in
   cpuidle (Vivek Yadav)

 - Fix device and OF node leaks at probe in the qcom-spm cpuidle driver
   and drop unnecessary initialisations from it (Johan Hovold)

 - Remove unnecessary address-of operators from the intel_idle cpuidle
   driver (Kaushlendra Kumar)

 - Rearrange main loop in menu_select() to make the code in that funtion
   easier to follow (Rafael Wysocki)

 - Convert values in microseconds to ktime using us_to_ktime() where
   applicable in the intel_idle power capping driver (Xichao Zhao)

* pm-cpuidle:
  cpuidle: Fail cpuidle device registration if there is one already
  cpuidle: sysfs: Use sysfs_emit()/sysfs_emit_at() instead of sprintf()/scnprintf()
  cpuidle: qcom-spm: drop unnecessary initialisations
  cpuidle: qcom-spm: fix device and OF node leaks at probe
  intel_idle: Remove unnecessary address-of operators
  cpuidle: governors: menu: Rearrange main loop in menu_select()

* pm-powercap:
  powercap: idle_inject: use us_to_ktime() where appropriate
2025-09-29 12:47:06 +02:00
Rafael J. Wysocki
d6fd599cd4 Merge branches 'pm-em', 'pm-opp' and 'pm-devfreq'
Merge energy model management, OPP (operating performance points) and
devfreq updates for 6.18-rc1:

 - Prevent CPU capacity updates after registering a perf domain from
   failing on a first CPU that is not present (Christian Loehle)

 - Add support for the cases in which frequency alone is not sufficient
   to uniquely identify an OPP (Krishna Chaitanya Chundru)

 - Use to_result() for OPP error handling in Rust (Onur Özkan)

 - Add support for LPDDR5 on Rockhip RK3588 SoC to rockchip-dfi devfreq
   driver (Nicolas Frattaroli)

 - Fix an issue where DDR cycle counts on RK3588/RK3528 with LPDDR4(X)
   are reported as half by adding a cycle multiplier to the DFI driver
   in rockchip-dfi devfreq-event driver (Nicolas Frattaroli)

 - Fix missing error pointer dereference check of regulator instance in
   the mtk-cci devfreq driver probe and remove a redundant condition from
   an if () statement in that driver (Dan Carpenter, Liao Yuanhong)

* pm-em:
  PM: EM: Fix late boot with holes in CPU topology

* pm-opp:
  OPP: Add support to find OPP for a set of keys
  rust: opp: use to_result for error handling

* pm-devfreq:
  PM / devfreq: rockchip-dfi: add support for LPDDR5
  PM / devfreq: rockchip-dfi: double count on RK3588
  PM / devfreq: mtk-cci: avoid redundant conditions
  PM / devfreq: mtk-cci: Fix potential error pointer dereference in probe()
2025-09-29 12:30:44 +02:00
Jani Nikula
97825e1c6d drm/{i915,xe}: driver agnostic drm to display pointer chase
The display driver needs to get from the struct drm_device pointer to
the struct intel_display pointer. Currently, this depends on knowledge
of the struct drm_i915_private and struct xe_device definitions, but
we'd like to hide those definitions from display.

Require the struct drm_device and struct intel_display * members within
struct drm_i915_private and struct xe_device to be placed next to each
other, to be able to figure out the display pointer without knowledge of
the structures.

Use a generic dummy device structure to define the relative offsets of
the drm and display members, and add static assertions to ensure this
holds for both i915 and xe. Use the dummy structure to do the pointer
chase from struct drm_device * to struct intel_display *.

This requires moving the display member in struct xe_device after the
drm member.

Cc: Lucas De Marchi <lucas.demarchi@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Ville Syrjala <ville.syrjala@linux.intel.com>
Suggested-by: Simona Vetter <simona.vetter@ffwll.ch>
Reviewed-by: Lucas De Marchi <lucas.demarchi@intel.com>
Link: https://lore.kernel.org/r/20250926111032.1188876-1-jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:55:50 +03:00
Jani Nikula
d630a1bdd6 drm/xe/stolen: use the same types as i915 interface
Unify the i915 and xe interfaces by switching to the same types as i915.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/f15d41bc232dfa957841f16d9a069c777af40194.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:31 +03:00
Jani Nikula
d3741f2c86 drm/{i915, xe}/stolen: convert stolen interface to struct drm_device
Make the stolen interface agnostic to i915/xe, and pass struct
drm_device instead.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/bbfc2aeaeee3156e92d49c73983be05b6feeede2.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:31 +03:00
Jani Nikula
994c74ea5e drm/{i915, xe}/stolen: use the stored i915/xe device pointer
Now that we store the i915/xe device pointer in struct
intel_stolen_node, we can reduce parameter passing in a number of
functions.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/6f31114c8113ce2254d422ca53992088b673fb2f.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
e8848d3d37 drm/{i915, xe}/stolen: add device pointer to struct intel_stolen_node
Add backpointers to i915/xe to allow simplifying some interfaces in
follow-up.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/321354d47f9e530159caefef510d5394f4177470.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
f74bab2d90 drm/{i915, xe}/stolen: make struct intel_stolen_node opaque
Add i915_gem_stolen_node_alloc() and i915_gem_stolen_node_free(),
returning struct intel_stolen_node pointer. Make struct
intel_stolen_node an opaque pointer, with different implementations in
i915 and xe.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/3fe71bbb4c75ee86b4d129fafa3d4cd6526363f4.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
33c8d948bc drm/xe/stolen: convert compat static inlines to proper functions
Add display/xe_stolen.c as the implementation for the stolen interface
exposed to display. This allows hiding the implementation details that
shouldn't be exposed to display.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/8e807c6aafc6151b18df08dda20053516813e001.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
59998644b7 drm/i915/stolen: convert intel_stolen_node into a real struct of its own
i915_gem_stolen.h simply defines intel_stolen_node as drm_mm_node. Make
struct intel_stolen_node an actual struct of its own right, and embed
struct drm_mm_node inside. This allow better unification between i915
and xe.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/36762f611566d81427e702369f4e8207ead5f26c.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
9f697958bb drm/xe/stolen: switch from BUG_ON() to WARN_ON() in compat
We're pretty much never supposed to be using BUG_ON(). Switch to
WARN_ON().

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/d14c693a3387a5d89bb88e81349639b5ec5663fb.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
b6500640ac drm/xe/stolen: convert compat stolen macros to inline functions
Improve type safety. Allows getting rid of a __maybe_unused annotation
too.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/1ec1fa59e0e54da49a1ec4fd1d535288066db502.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
511d2d70df drm/xe/stolen: rename fb to node in stolen compat header
It's more about node than fb, and this makes more sense now that the
struct is also named intel_stolen_node.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/71a7872e47da5f3fbe61cc21723bfcf8ff6518b8.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Jani Nikula
3198609ecc drm/{i915, xe}/stolen: rename i915_stolen_fb to intel_stolen_node
Use a more generic name than one that refers to "i915" and "fb".

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://lore.kernel.org/r/925fd07d3f2a6115c71984f5a40a06c9eb46a539.1758732183.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:41:30 +03:00
Aaron Kling
ba60189291 cpufreq: tegra186: Initialize all cores to max frequencies
During initialization, the EDVD_COREx_VOLT_FREQ registers for some cores
are still at reset values and not reflecting the actual frequency. This
causes get calls to fail. Set all cores to their respective max
frequency during probe to initialize the registers to working values.

Suggested-by: Mikko Perttunen <mperttunen@nvidia.com>
Signed-off-by: Aaron Kling <webgeek1234@gmail.com>
Reviewed-by: Mikko Perttunen <mperttunen@nvidia.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
2025-09-29 14:55:50 +05:30
Aaron Kling
0b1bb980fd cpufreq: tegra186: Set target frequency for all cpus in policy
The original commit set all cores in a cluster to a shared policy, but
did not update set_target to apply a frequency change to all cores for
the policy. This caused most cores to remain stuck at their boot
frequency.

Fixes: be4ae8c194 ("cpufreq: tegra186: Share policy per cluster")
Signed-off-by: Aaron Kling <webgeek1234@gmail.com>
Reviewed-by: Mikko Perttunen <mperttunen@nvidia.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
2025-09-29 14:55:35 +05:30
Jani Nikula
996ad9cd3e Merge drm/drm-next into drm-intel-next
Backmerge to sync with drm/xe changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-09-29 12:13:00 +03:00
Thorsten Blum
24287f9020 rust: cpufreq: streamline find_supply_names
Remove local variables from find_supply_names() and use .and_then() with
the more concise kernel::kvec![] macro, instead of KVec::with_capacity()
followed by .push() and Some().

No functional changes intended.

Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
2025-09-29 14:40:46 +05:30
Johan Hovold
ebc4ed14a4 cpufreq: mediatek: fix device leak on probe failure
Make sure to drop the reference to the cci device taken by
of_find_device_by_node() on probe failure (e.g. probe deferral).

Fixes: 0daa47325b ("cpufreq: mediatek: Link CCI device to CPU")
Cc: Jia-Wei Chang <jia-wei.chang@mediatek.com>
Cc: Rex-BC Chen <rex-bc.chen@mediatek.com>
Signed-off-by: Johan Hovold <johan@kernel.org>
Reviewed-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
Reviewed-by: Chen-Yu Tsai <wenst@chromium.org>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
2025-09-29 14:40:46 +05:30
Rafael J. Wysocki
17eb881291 linux-cpupower-6.18-rc1
Fixes incorrect return vale in cpupower_write_sysfs() error path
 and passing incorrect size to cpuidle_state_write_file() while
 writing status to disable file in cpuidle_state_disable().
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEPZKym/RZuOCGeA/kCwJExA0NQxwFAmjZm4IACgkQCwJExA0N
 QxwfRhAAy2ZRWq6U9zhGEeEhlm8/gAg0mtXe9SJaQFNHII9GMBfetJkBGGIszXMc
 1CTk6fbIdnzleLOriZy6K26g4gNY269zDvei+1HnLRJAdsu57VOCqVUm5Fv6J+s6
 QyJnpe1UadqIPnQu3kWvc/JtLYVGmfn5ezvW912bFpf7Qq1mat6aP0t5G5PrnK+m
 Zv5r8g0ia2h4j17+FfG198H6QHAqw/jxrbtB4rTObc/BlUor5VxlBUs0NgSi3CCD
 5S/hk8q1kIvpAv8UQ6z6Mcxj43Pnp4zMu2uz4oQNT6SDhpjI5kja1Y9vh4ZwYRMT
 CHqWmILU5d1wgULIMQBHisJFUPuW+/LrOA7vMFRaVUOKJr6aIlHWVo2IWzRvvw1q
 T/TqqRm5b+ifbM6MK2ktsYyn6jIb8PXpZ6gLkf1QHBz99ijpWuIkuVhNNM5FmHor
 m7LlLp5EwA08UXz4l6OM5XovX1O3nwSt/TQek4XokAIvddKw6G2tTt0qy6Gp/cwl
 NFrh+TsybAKfwLOFaRRSQ21AzYkuF2TYyjjJtGgITvcf/H7k+E4lf3oxbgj6K0hf
 D4NSN37UukooCdzms4/0AEwBXyfuqhajqcJW2VgaWM27TAqyY+NSgnSlALNcb9ev
 SNFLRIDZjEc8SGrd8yNftVtpsrX0k7pHzrquLD+E1+vy1XzUGLA=
 =1VZt
 -----END PGP SIGNATURE-----

Merge tag 'linux-cpupower-6.18-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux

Merge cpupower utility updates for 6.18-rc1 from Shuah Khan:

"Fixes incorrect return vale in cpupower_write_sysfs() error path
 and passing incorrect size to cpuidle_state_write_file() while
 writing status to disable file in cpuidle_state_disable()."

* tag 'linux-cpupower-6.18-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux: (1125 commits)
  tools/cpupower: Fix incorrect size in cpuidle_state_disable()
  tools/cpupower: fix error return value in cpupower_write_sysfs()
  Linux 6.17-rc6
  MAINTAINERS: Input: Drop melfas-mip4 section
  USB: core: remove the move buf action
  MAINTAINERS: Update the DMA Rust entry
  erofs: fix long xattr name prefix placement
  Revert "net: usb: asix: ax88772: drop phylink use in PM to avoid MDIO runtime PM wakeups"
  hsr: hold rcu and dev lock for hsr_get_port_ndev
  hsr: use hsr_for_each_port_rtnl in hsr_port_get_hsr
  hsr: use rtnl lock when iterating over ports
  wifi: nl80211: completely disable per-link stats for now
  net: usb: asix: ax88772: drop phylink use in PM to avoid MDIO runtime PM wakeups
  net: ethtool: fix wrong type used in struct kernel_ethtool_ts_info
  selftests/bpf: Skip timer cases when bpf_timer is not supported
  bpf: Reject bpf_timer for PREEMPT_RT
  libceph: fix invalid accesses to ceph_connection_v1_info
  PM: hibernate: Restrict GFP mask in hibernation_snapshot()
  MAINTAINERS: add Phil as netfilter reviewer
  netfilter: nf_tables: restart set lookup on base_seq change
  ...
2025-09-29 10:10:05 +02:00
Vlastimil Babka
ca74b8cada Merge series "slab: Re-entrant kmalloc_nolock()"
From the cover letter [1]:

This patch set introduces kmalloc_nolock() which is the next logical
step towards any context allocation necessary to remove bpf_mem_alloc
and get rid of preallocation requirement in BPF infrastructure.

In production BPF maps grew to gigabytes in size. Preallocation wastes
memory. Alloc from any context addresses this issue for BPF and other
subsystems that are forced to preallocate too.

This long task started with introduction of alloc_pages_nolock(), then
memcg and objcg were converted to operate from any context including
NMI, this set completes the task with kmalloc_nolock() that builds on
top of alloc_pages_nolock() and memcg changes.

After that BPF subsystem will gradually adopt it everywhere.

Link: https://lore.kernel.org/all/20250909010007.1660-1-alexei.starovoitov@gmail.com/ [1]
2025-09-29 09:46:17 +02:00
Vlastimil Babka
b912061924 Merge series "SLUB percpu sheaves"
This series adds an opt-in percpu array-based caching layer to SLUB.
It has evolved to a state where kmem caches with sheaves are compatible
with all SLUB features (slub_debug, SLUB_TINY, NUMA locality
considerations). The plan is therefore that it will be later enabled for
all kmem caches and replace the complicated cpu (partial) slabs code.

Note the name "sheaf" was invented by Matthew Wilcox so we don't call
the arrays magazines like the original Bonwick paper. The per-NUMA-node
cache of sheaves is thus called "barn".

This caching may seem similar to the arrays we had in SLAB, but there
are some important differences:

- deals differently with NUMA locality of freed objects, thus there are
  no per-node "shared" arrays (with possible lock contention) and no
  "alien" arrays that would need periodical flushing
  - instead, freeing remote objects (which is rare) bypasses the sheaves
  - percpu sheaves thus contain only local objects (modulo rare races
    and local node exhaustion)
  - NUMA restricted allocations and strict_numa mode is still honoured
- improves kfree_rcu() handling by reusing whole sheaves
- there is an API for obtaining a preallocated sheaf that can be used
  for guaranteed and efficient allocations in a restricted context, when
  the upper bound for needed objects is known but rarely reached
- opt-in, not used for every cache (for now)

The motivation comes mainly from the ongoing work related to VMA locking
scalability and the related maple tree operations. This is why VMA and
maple nodes caches are sheaf-enabled in the patchset.

A sheaf-enabled cache has the following expected advantages:

- Cheaper fast paths. For allocations, instead of local double cmpxchg,
  thanks to local_trylock() it becomes a preempt_disable() and no atomic
  operations. Same for freeing, which is otherwise a local double cmpxchg
  only for short term allocations (so the same slab is still active on the
  same cpu when freeing the object) and a more costly locked double
  cmpxchg otherwise.

- kfree_rcu() batching and recycling. kfree_rcu() will put objects to a
  separate percpu sheaf and only submit the whole sheaf to call_rcu()
  when full. After the grace period, the sheaf can be used for
  allocations, which is more efficient than freeing and reallocating
  individual slab objects (even with the batching done by kfree_rcu()
  implementation itself). In case only some cpus are allowed to handle rcu
  callbacks, the sheaf can still be made available to other cpus on the
  same node via the shared barn. The maple_node cache uses kfree_rcu() and
  thus can benefit from this.
  Note: this path is currently limited to !PREEMPT_RT

- Preallocation support. A prefilled sheaf can be privately borrowed to
  perform a short term operation that is not allowed to block in the
  middle and may need to allocate some objects. If an upper bound (worst
  case) for the number of allocations is known, but only much fewer
  allocations actually needed on average, borrowing and returning a sheaf
  is much more efficient then a bulk allocation for the worst case
  followed by a bulk free of the many unused objects. Maple tree write
  operations should benefit from this.

- Compatibility with slub_debug. When slub_debug is enabled for a cache,
  we simply don't create the percpu sheaves so that the debugging hooks
  (at the node partial list slowpaths) are reached as before. The same
  thing is done for CONFIG_SLUB_TINY. Sheaf preallocation still works by
  reusing the (ineffective) paths for requests exceeding the cache's
  sheaf_capacity. This is in line with the existing approach where
  debugging bypasses the fast paths and SLUB_TINY preferes memory
  savings over performance.

The above is adapted from the cover letter [1], which contains also
in-kernel microbenchmark results showing the lower overhead of sheaves.

Results from Suren Baghdasaryan [2] using a mmap/munmap microbenchmark
also show improvements.

Results from Sudarsan Mahendran [3] using will-it-scale show both
benefits and regressions, probably due to overall noisiness of those
tests.

Link: https://lore.kernel.org/all/20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz/ [1]
Link: https://lore.kernel.org/all/CAJuCfpEQ%3DRUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt%3DQ@mail.gmail.com/ [2]
Link: https://lore.kernel.org/all/20250913000935.1021068-1-sudarsanm@google.com/ [3]
2025-09-29 09:46:17 +02:00
Alexei Starovoitov
af92793e52 slab: Introduce kmalloc_nolock() and kfree_nolock().
kmalloc_nolock() relies on ability of local_trylock_t to detect
the situation when per-cpu kmem_cache is locked.

In !PREEMPT_RT local_(try)lock_irqsave(&s->cpu_slab->lock, flags)
disables IRQs and marks s->cpu_slab->lock as acquired.
local_lock_is_locked(&s->cpu_slab->lock) returns true when
slab is in the middle of manipulating per-cpu cache
of that specific kmem_cache.

kmalloc_nolock() can be called from any context and can re-enter
into ___slab_alloc():
  kmalloc() -> ___slab_alloc(cache_A) -> irqsave -> NMI -> bpf ->
    kmalloc_nolock() -> ___slab_alloc(cache_B)
or
  kmalloc() -> ___slab_alloc(cache_A) -> irqsave -> tracepoint/kprobe -> bpf ->
    kmalloc_nolock() -> ___slab_alloc(cache_B)

Hence the caller of ___slab_alloc() checks if &s->cpu_slab->lock
can be acquired without a deadlock before invoking the function.
If that specific per-cpu kmem_cache is busy the kmalloc_nolock()
retries in a different kmalloc bucket. The second attempt will
likely succeed, since this cpu locked different kmem_cache.

Similarly, in PREEMPT_RT local_lock_is_locked() returns true when
per-cpu rt_spin_lock is locked by current _task_. In this case
re-entrance into the same kmalloc bucket is unsafe, and
kmalloc_nolock() tries a different bucket that is most likely is
not locked by the current task. Though it may be locked by a
different task it's safe to rt_spin_lock() and sleep on it.

Similar to alloc_pages_nolock() the kmalloc_nolock() returns NULL
immediately if called from hard irq or NMI in PREEMPT_RT.

kfree_nolock() defers freeing to irq_work when local_lock_is_locked()
and (in_nmi() or in PREEMPT_RT).

SLUB_TINY config doesn't use local_lock_is_locked() and relies on
spin_trylock_irqsave(&n->list_lock) to allocate,
while kfree_nolock() always defers to irq_work.

Note, kfree_nolock() must be called _only_ for objects allocated
with kmalloc_nolock(). Debug checks (like kmemleak and kfence)
were skipped on allocation, hence obj = kmalloc(); kfree_nolock(obj);
will miss kmemleak/kfence book keeping and will cause false positives.
large_kmalloc is not supported by either kmalloc_nolock()
or kfree_nolock().

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:42:36 +02:00
Alexei Starovoitov
7612833192 slab: Reuse first bit for OBJEXTS_ALLOC_FAIL
Since the combination of valid upper bits in slab->obj_exts with
OBJEXTS_ALLOC_FAIL bit can never happen,
use OBJEXTS_ALLOC_FAIL == (1ull << 0) as a magic sentinel
instead of (1ull << 2) to free up bit 2.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:42:36 +02:00
Alexei Starovoitov
83382af9dd slab: Make slub local_(try)lock more precise for LOCKDEP
kmalloc_nolock() can be called from any context
the ___slab_alloc() can acquire local_trylock_t (which is rt_spin_lock
in PREEMPT_RT) and attempt to acquire a different local_trylock_t
while in the same task context.

The calling sequence might look like:
kmalloc() -> tracepoint -> bpf -> kmalloc_nolock()

or more precisely:
__lock_acquire+0x12ad/0x2590
lock_acquire+0x133/0x2d0
rt_spin_lock+0x6f/0x250
___slab_alloc+0xb7/0xec0
kmalloc_nolock_noprof+0x15a/0x430
my_debug_callback+0x20e/0x390 [testmod]
___slab_alloc+0x256/0xec0
__kmalloc_cache_noprof+0xd6/0x3b0

Make LOCKDEP understand that local_trylock_t-s protect
different kmem_caches. In order to do that add lock_class_key
for each kmem_cache and use that key in local_trylock_t.

This stack trace is possible on both PREEMPT_RT and !PREEMPT_RT,
but teach lockdep about it only for PREEMPT_RT, since
in !PREEMPT_RT the ___slab_alloc() code is using
local_trylock_irqsave() when lockdep is on.

Note, this patch applies this logic to local_lock_t
while the next one converts it to local_trylock_t.
Both are mapped to rt_spin_lock in PREEMPT_RT.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:42:36 +02:00
Alexei Starovoitov
d7242af864 mm: Introduce alloc_frozen_pages_nolock()
Split alloc_pages_nolock() and introduce alloc_frozen_pages_nolock()
to be used by alloc_slab_page().

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:42:35 +02:00
Alexei Starovoitov
99253de51f mm: Allow GFP_ACCOUNT to be used in alloc_pages_nolock().
Change alloc_pages_nolock() to default to __GFP_COMP when allocating
pages, since upcoming reentrant alloc_slab_page() needs __GFP_COMP.
Also allow __GFP_ACCOUNT flag to be specified,
since most of BPF infra needs __GFP_ACCOUNT except BPF streams.

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:42:35 +02:00
Alexei Starovoitov
4957089a23 locking/local_lock: Introduce local_lock_is_locked().
Introduce local_lock_is_locked() that returns true when
given local_lock is locked by current cpu (in !PREEMPT_RT) or
by current task (in PREEMPT_RT).
The goal is to detect a deadlock by the caller.

Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:42:35 +02:00
Liam R. Howlett
719a42e563 maple_tree: Convert forking to use the sheaf interface
Use the generic interface which should result in less bulk allocations
during a forking.

A part of this is to abstract the freeing of the sheaf or maple state
allocations into its own function so mas_destroy() and the tree
duplication code can use the same functionality to return any unused
resources.

[andriy.shevchenko@linux.intel.com: remove unused mt_alloc_bulk()]
Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:40:49 +02:00
Liam R. Howlett
6bf377b06c maple_tree: Add single node allocation support to maple state
The fast path through a write will require replacing a single node in
the tree.  Using a sheaf (32 nodes) is too heavy for the fast path, so
special case the node store operation by just allocating one node in the
maple state.

Signed-off-by: Liam R. Howlett <Liam.Howlett@Oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:40:46 +02:00
Liam R. Howlett
9b05890a25 maple_tree: Prefilled sheaf conversion and testing
Use prefilled sheaves instead of bulk allocations. This should speed up
the allocations and the return path of unused allocations.

Remove the push and pop of nodes from the maple state as this is now
handled by the slab layer with sheaves.

Testing has been removed as necessary since the features of the tree
have been reduced.

Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:31:41 +02:00
Liam R. Howlett
fdbebab19f tools/testing: Add support for prefilled slab sheafs
Add the prefilled sheaf structs to the slab header and the associated
functions to the testing/shared/linux.c file.

Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:25:00 +02:00
Pedro Falcato
025f93101b maple_tree: Replace mt_free_one() with kfree()
kfree() is a little shorter and works with kmem_cache_alloc'd pointers
too. Also lets us remove one more helper.

Signed-off-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:24:44 +02:00
Pedro Falcato
9b60811cb3 maple_tree: Use kfree_rcu in ma_free_rcu
kfree_rcu is an optimized version of call_rcu + kfree. It used to not be
possible to call it on non-kmalloc objects, but this restriction was
lifted ever since SLOB was dropped from the kernel, and since commit
6c6c47b063 ("mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy()").

Thus, replace call_rcu + mt_free_rcu with kfree_rcu.

Signed-off-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:24:03 +02:00
Liam R. Howlett
551a6e757a testing/radix-tree/maple: Hack around kfree_rcu not existing
liburcu doesn't have kfree_rcu (or anything similar). Despite that, we
can hack around it in a trivial fashion, by adding a wrapper.

The wrapper only works for maple_nodes because we cannot get the
kmem_cache pointer any other way in the test code.

Link: https://lore.kernel.org/all/20250812162124.59417-1-pfalcato@suse.de/
Suggested-by: Pedro Falcato <pfalcato@suse.de>
Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:23:52 +02:00
Vlastimil Babka
9f910f7d3d tools/testing: include maple-shim.c in maple.c
There's some duplicated code and we are about to add more functionality
in maple-shared.h that we will need in the userspace maple test to be
available, so include it via maple-shim.c

Co-developed-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:23:08 +02:00
Vlastimil Babka
59faa4da7c maple_tree: use percpu sheaves for maple_node_cache
Setup the maple_node_cache with percpu sheaves of size 32 to hopefully
improve its performance. Note this will not immediately take advantage
of sheaf batching of kfree_rcu() operations due to the maple tree using
call_rcu with custom callbacks. The followup changes to maple tree will
change that and also make use of the prefilled sheaves functionality.

Reviewed-by: Sidhartha Kumar <sidhartha.kumar@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:22:57 +02:00
Vlastimil Babka
3accabda4d mm, vma: use percpu sheaves for vm_area_struct cache
Create the vm_area_struct cache with percpu sheaves of size 32 to
improve its performance.

Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:22:53 +02:00
Liam R. Howlett
c4fb7f0a79 tools/testing: Add support for changes to slab for sheaves
The slab changes for sheaves requires more effort in the testing code.
Unite all the kmem_cache work into the tools/include slab header for
both the vma and maple tree testing.

The vma test code also requires importing more #defines to allow for
seamless use of the shared kmem_cache code.

This adds the pthread header to the slab header in the tools directory
to allow for the pthread_mutex in linux.c.

Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:22:44 +02:00
Vlastimil Babka
4ec1a08d20 slab: allow NUMA restricted allocations to use percpu sheaves
Currently allocations asking for a specific node explicitly or via
mempolicy in strict_numa node bypass percpu sheaves. Since sheaves
contain mostly local objects, we can try allocating from them if the
local node happens to be the requested node or allowed by the mempolicy.
If we find the object from percpu sheaves is not from the expected node,
we skip the sheaves - this should be rare.

Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:21:16 +02:00
Liam R. Howlett
d09a61a3aa tools/testing/vma: Implement vm_refcnt reset
Add the reset of the ref count in vma_lock_init().  This is needed if
the vma memory is not zeroed on allocation.

Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:21:16 +02:00
Vlastimil Babka
989b09b739 slab: skip percpu sheaves for remote object freeing
Since we don't control the NUMA locality of objects in percpu sheaves,
allocations with node restrictions bypass them. Allocations without
restrictions may however still expect to get local objects with high
probability, and the introduction of sheaves can decrease it due to
freed object from a remote node ending up in percpu sheaves.

The fraction of such remote frees seems low (5% on an 8-node machine)
but it can be expected that some cache or workload specific corner cases
exist. We can either conclude that this is not a problem due to the low
fraction, or we can make remote frees bypass percpu sheaves and go
directly to their slabs. This will make the remote frees more expensive,
but if it's only a small fraction, most frees will still benefit from
the lower overhead of percpu sheaves.

This patch thus makes remote object freeing bypass percpu sheaves,
including bulk freeing, and kfree_rcu() via the rcu_free sheaf. However
it's not intended to be 100% guarantee that percpu sheaves will only
contain local objects. The refill from slabs does not provide that
guarantee in the first place, and there might be cpu migrations
happening when we need to unlock the local_lock. Avoiding all that could
be possible but complicated so we can leave it for later investigation
whether it would be worth it. It can be expected that the more selective
freeing will itself prevent accumulation of remote objects in percpu
sheaves so any such violations would have only short-term effects.

Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:21:16 +02:00
Liam R. Howlett
e3852a1213 maple_tree: Drop bulk insert support
Bulk insert mode was added to facilitate forking faster, but forking now
uses __mt_dup() to duplicate the tree.

The addition of sheaves has made the bulk allocations difficult to
maintain - since the expected entries would preallocate into the maple
state.  A big part of the maple state node allocation was the ability to
push nodes back onto the state for later use, which was essential to the
bulk insert algorithm.

Remove mas_expected_entries() and mas_destroy_rebalance() functions as
well as the MA_STATE_BULK and MA_STATE_REBALANCE maple state flags since
there are no users anymore.  Drop the associated testing as well.

Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:21:16 +02:00
Vlastimil Babka
0829422921 slab: determine barn status racily outside of lock
The possibility of many barn operations is determined by the current
number of full or empty sheaves. Taking the barn->lock just to find out
that e.g. there are no empty sheaves results in unnecessary overhead and
lock contention. Thus perform these checks outside of the lock with a
data_race() annotated variable read and fail quickly without taking the
lock.

Checks for sheaf availability that racily succeed have to be obviously
repeated under the lock for correctness, but we can skip repeating
checks if there are too many sheaves on the given list as the limits
don't need to be strict.

Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:21:16 +02:00
Lorenzo Stoakes
da577f1fcb tools/testing/vma: clean up stubs in vma_internal.h
We do not need to references arguments just to avoid compiler warnings,
the warning in question does not arise here, so remove all of the
instances of '(void)xxx' introduced purely to avoid this warning.

As reported by WagYuli in the referenced mail, GCC 8.3 and before will
have issues compiling this file if parameter names are not provided, so
ensure these are always provided.

Finally, perform a trivial fix up of kmem_cache_alloc() which technically
has parameters in the incorrect order (as reported by Vlastimil Babka
off-list).

Link: https://lkml.kernel.org/r/20250826102824.22730-1-lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reported-by: WangYuli <wangyuli@uniontech.com>
Closes: https://lore.kernel.org/linux-mm/EFCEBE7E301589DE+20250729084700.208767-1-wangyuli@uniontech.com/
Reported-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Cc: Jann Horn <jannh@google.com>
Cc: WangYuli <wangyuli@uniontech.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
2025-09-29 09:21:16 +02:00