mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
synced 2026-11-10 20:48:12 +00:00
1,482,976 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
SJ Park
|
06befa61c4 |
mm/damon/lru_sort: error out for >10000 active_mem_bp
damos_quota_score() can trigger division by zero if the target value is
zero. DAMON_LRU_SORT lets users set the target value for the hot memory
scheme via active_mem_bp parameter. It avoids setting it as the target
value if the parameter value is zero. However, it also sets the cold
memory scheme with a target value that is calculated as '10000 -
active_mem_bp + 2'. Hence, if a user sets active_mem_bp 10002, the cold
memory scheme's quota goal target value can be zero. As a result,
division by zero can be triggered. Fix by returning an error when the
user tries to start DAMON with >10000 active_mem_bp parameter value.
It makes no sense to set active_mem_bp with 10002. It also requires
module parameters write permission to reproduce the issue. That said, the
consequence is quite bad.
One reliable way to reproduce the issue is like below:
# cd /sys/module/damon_lru_sort/parameters
# echo 1000 > wmarks_high
# echo 995 > wmarks_mid
# echo 0 > wmarks_low
# echo 10002 > active_mem_bp
# echo Y > enabled
# dmesg -w
[...]
[ 597.421247] Oops: divide error: 0000 [#1] SMP NOPTI
[ 597.428848] RIP: 0010:damos_quota_score+0x6f/0x480
This issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260803134034.15217-3-sj@kernel.org
Link: https://lore.kernel.org/20260801213028.5127-1-sj@kernel.org [1]
Fixes:
|
||
|
SJ Park
|
a16fd3ad9d |
samples/damon/mtier: error out for zero quota goal target values
Patch series "mm/damon: avoid division by zero from damos_quota_score()".
DAMON_SAMPLE_MTIER and DAMON_LRU_SORT allow the user to trigger division
by zero in damos_quota_score(). Avoid it by adding parameters validation
checks.
This patch (of 2):
damos_quota_score() can trigger division by zero if the target_value is
zero. DAMON_SAMPLE_MTIER lets users set the target_value via
node0_mem_{used,free}_bp parameters. It doesn't guard zero value case,
though. As a result, users can trigger division by zero. Fix the issue
by returning an error when the user tries to start DAMON with zero
node0_mem_{used,free}_bp parameter values.
DAMON_SAMPLE_MTIER is just a sample module, but the consequence is quite
bad. Also the zero node0_mem_free_bp parameter might look like a
reasonable setup to some users. Hence, the issue might really happen in
the real world.
One reliable way to reproduce the issue is like below:
# cd /sys/module/damon_sample_mtier/parameters
# echo 4096 > node0_start_addr
# echo 8192 > node0_end_addr
# echo 8192 > node1_start_addr
# echo 81920 > node1_end_addr
# echo 0 > node0_mem_free_bp
# echo Y > enabled
# dmesg -w
[...]
[18792.235916] Oops: divide error: 0000 [#1] SMP NOPTI
[...]
[18792.242787] RIP: 0010:damos_quota_score+0x6f/0x480
[...]
This issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260803134034.15217-1-sj@kernel.org
Link: https://lore.kernel.org/20260803134034.15217-2-sj@kernel.org
Link: https://lore.kernel.org/20260801202657.117135-1-sj@kernel.org [1]
Fixes:
|
||
Danila Tikhonov
|
4194140a51 |
mailmap: map old addresses to Danila Tikhonov
Map my old jiaxyga.com and ProtonMail addresses to the current mainlining.org address. Link: https://lore.kernel.org/20260801153333.296218-1-danila@mainlining.org Signed-off-by: Danila Tikhonov <danila@mainlining.org> Cc: Jakub Kacinski <kuba@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
Lorenzo Stoakes (ARM)
|
98bd3af0bb |
mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic
Rather than mixing the refcounted and non-refcounted
CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic, separate the two out cleanly so
it is clear what happens when this configuration option is set and what
happens when it is not.
Introduce HUGE_ZERO_UNSET_PFN to abstract the ~0UL assignment, only
introduce the refcount, lock and shrinker if
!CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, abstract initialisation and teardown,
abstract the huge zero folio allocation from refcounting.
Also change a BUG_ON() to WARN_ON_ONCE() while we're at it.
No functional change intended.
Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-2-c5d8a41b317f@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
33192a26cd |
mm/huge_memory: fix huge_zero_pfn race
Patch series "mm/huge_memory: fix huge_zero_pfn race", v2.
There is a subtle race in the reference-counted huge_zero_folio
implementation.
The fast path atomic logic fails to account for the fact that the shrinker
(which drops the final huge_zero_refcount pin) can overwrite huge_zero_pfn
with the ~0UL sentinel value in shrink_huge_zero_folio_scan() after a
racing get_huge_zero_folio() installed a valid value there.
This results in huge_zero_folio being correctly set but huge_zero_pfn
being set incorrectly and thus is_huge_zero_pfn() and consequently
is_huge_zero_pmd() will misidentify the huge zero folio as being an
ordinary THP folio.
This can result in the huge zero folio being split and otherwise treated
incorrectly.
The solution to this is very subtle as there is an atomic fast path, and
thus ordering in weakly ordered architectures has to be treated very
carefully.
The first commit fixes the issue by introducing a spinlock around
huge_zero_[pfn, folio, refcount] write, with careful consideration paid to
load/store ordering in the fast path. It is placed first and kept as
small as possible so that it can be backported on its own.
The second commit is a pure cleanup which reworks the
CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic to better separate the persistent
logic from the dynamically allocated one.
This patch (of 2):
If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted
by huge_zero_refcount and returned by mm_get_huge_zero_folio().
When the caller is done with the huge zero page, its reference count is
decremented. Only a shrinker can set the reference count to zero.
A race can unfortunately occur between a shrinker decrementing the
reference count to zero and a concurrent page fault.
This is because shrink_huge_zero_folio_scan() might, if very unlucky, be
preempted between setting huge_zero_refcount to zero and writing an
invalid value.
During this time get_huge_zero_folio() could write to huge_zero_pfn before
shrink_huge_zero_folio_scan() resumes.
In this event the huge zero folio will be persistently misidentified
causing the THP code path to be entered inappropriately for the huge zero
folio:
CPU 0 CPU 1
=======================================|=================================
shrink_huge_zero_folio_scan() |
atomic_cmpxchg() sets refcount to 0 |
xchg() sets huge_zero_folio to NULL | get_huge_zero_folio()
| | atomic_inc_not_zero() -> zero
preempted for a long time | Allocate new huge zero folio
| | Write valid huge_zero_folio
v | Write valid huge_zero_pfn
Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite!
This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly
returning false for a huge zero page which could result in issues like the
huge zero folio being incorrectly split.
Note that the issue is with huge_zero_pfn not huge_zero_folio, as
get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL
with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set
huge_zero_folio.
Fix the issue by introducing a spinlock, huge_zero_lock, to prevent
concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount.
There needs to be significant care taken here to ensure correctness:
The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which
is outside of the critical section, and means huge zero allocation is
gated on zero huge_zero_refcount.
The fast path doesn't use huge_zero_lock, so the critical section is
irrelevant to it.
So invariants are required - huge_zero_refcount MUST:
* Only be set in the huge_zero_lock critical section to ensure
serialisation of huge_zero_pfn, huge_zero_folio and huge_zero_refcount
writes.
* Be set non-zero only AFTER huge_zero_[pfn, folio] are set to valid values
so installation of the huge zero folio on read page fault ensures
concurrent is_huge_zero_*() calls correctly identify the huge zero folio.
* Be set zero only BEFORE huge_zero_[pfn, folio] are set to NULL and ~0UL
respectively, and atomically.
Establish these by:
* Only setting huge_zero_refcount to zero or an absolute value in the
huge_zero_lock critical section in get_huge_zero_folio() and
shrink_huge_zero_folio_scan(), and always updating atomically there
and elsewhere.
* Using atomic_set_release(&huge_zero_refcount) in get_huge_zero_folio()
after huge_zero_[pfn, folio] are set. This is paired with
atomic_inc_not_zero() to ensure atomic_inc_not_zero() only observes a
non-zero value if huge_zero_[pfn, folio] are set.
* Using atomic_cmpxchg() in shrink_huge_zero_folio_scan() (as before) to
ensure that it is set zero only when equal to 1 and set atomically.
* atomic_cmpxchg() being fully ordered ensures this is done prior to
huge_zero_[folio, pfn] being set to NULL and ~0UL respectively.
Eliminate the retry loop in get_huge_zero_folio() as the atomic_cmpxchg()
in shrink_huge_zero_folio_scan() is now performed under the lock, and
replace with an equally locked atomic_inc() to set the reference count
should the caller be raced on huge zero folio installation.
folio_put() naturally implies a full memory barrier so its ordering is
maintained correctly.
The huge zero folio also cannot be released except when the shrinker does
so as it is non-LRU and non-rmappable.
Note that only the huge zero shrinker (via shrink_huge_zero_folio_scan())
can actually set huge_zero_refcount to zero, which is the count of mm's
which have at least one huge zero folio installed plus one shrinker pin.
Additionally convert a BUG_ON() to a VM_WARN_ON_ONCE().
Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-0-c5d8a41b317f@kernel.org
Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-1-c5d8a41b317f@kernel.org
Fixes:
|
||
Brendan Jackman
|
50124648db |
MAINTAINERS: update address for Brendan Jackman
Switch my entry in MAINTAINERS and .mailmap to my @linux.dev email address Link: https://lore.kernel.org/20260729-email-change-v1-1-666ae7c2b7fc@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Signed-off-by: Brendan Jackman <brendan.jackman@linux.dev> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Cc: David Hildenbrand <david@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
Hugh Dickins
|
86da3f7e1e |
mm/filemap: __filemap_add_folio() restore index before retrying
In __filemap_add_folio()'s split-a-conflict loop, xas_set_order() is
applied repeatedly: each application modifies xas.xa_index, rounding it
down according to the split_order attempted at that stage: and if all goes
as intended, it eventually (or immediately) converges on an
xas_try_split() to the required folio_order, with xas.xa_index now the
same as index: then xas_store() puts the new folio into the xarray there.
But if a new node was needed, and GFP_NOWAIT allocation did not get one,
the lock is dropped, xas_nomem() used to allocate, and sequence retried.
If (that part of) the xarray is unchanged when the lock is reacquired, no
problem. But what if the conflict was meanwhile resolved by another
thread (perhaps even doing the same thing, inserting a folio at that same
index)? Isn't there a danger of now putting our folio into the xarray at
an intermediate rounded-down index? With !folio_contains() bug to follow,
when CONFIG_DEBUG_VM=y is checking for that.
Fix this with an xas_set_order() to restore the original xas.xa_index at
the bottom of the loop, so the retry does a full re-evaluation after
reacquiring the lock, and cannot reach xas_store() with the wrong index.
Production was suffering from rare SIGILLs and SIGSEGVs, executable text
found a page away from where it belonged, !folio_contains() bug hit when
debug enabled: symptoms not seen since this patch went in.
Link: https://lore.kernel.org/562fbfa6-dd6d-0b6a-2461-ed2ff1173bc8@google.com
Fixes:
|
||
Ramin Moussavi
|
c2689266e5 |
microblaze: restore the page alignment of swapper_pg_dir
microblaze handles TLB misses in software, and the handler builds the address of the L1 entry by ORing the index into the page directory base instead of adding it (hw_exception_handler.S): bsrli r5, r3, PGDIR_SHIFT - 2 andi r5, r5, PAGE_SIZE - 4 /* Assume pgdir aligned on 4K boundary, no need for "andi r4,r4,0xfffff003" */ or r4, r4, r5 lwi r4, r4, 0 /* Get L1 entry */ The index is masked to the low 12 bits, so the OR only works if those bits of the base are zero -- which is exactly the assumption the comment states and the reason the masking of the base can be skipped. swapper_pg_dir had no alignment directive of its own. It was aligned because it followed empty_zero_page in head.S, and that one carried the .align 12: .section .data .global empty_zero_page .align 12 empty_zero_page: .space PAGE_SIZE .global swapper_pg_dir swapper_pg_dir: .space PAGE_SIZE Commit |
||
|
Lorenzo Stoakes (ARM)
|
9d3277b2c0 |
arm64: remove redundant concurrent ptdump UAF mitigation
This partially reverts commit |
||
|
Lorenzo Stoakes (ARM)
|
27c32e5538 |
mm/ptdump: always stabilise against page table freeing using init_mm
Previous commits have established the invariant that kernel page table freeing is performed while an mmap read lock on init_mm is held, which fixes races between ptdump and kernel page table freeing over init_mm. However, x86 and arm64 can perform a ptdump over an mm other than init_mm via ptdump_walk_pgd() and since kernel memory ranges are shared across non-kernel mm's, this means that the race still exists for these cases. Fix this by acquiring a nested mmap write lock for init_mm in ptdump_walk_pgd(). This is safe as we take this after mmap write locking the mm, and nothing acquires the init_mm lock first before locking an arbitrary mm, so no deadlock is possible. Also update walk_page_range_debug() to assert that init_mm is write locked, add a comment explaining why and remove some redundant code, and eliminate the unnecessary and confusing invocation of walk_kernel_page_table_range(). We can safely remove the non-NULL check for walk.mm, as the mmap lock asserts would NULL pointer deref if it was (and of course no callers do this). The first point at which ptdump can race kernel page table freeing is commit |
||
|
Lorenzo Stoakes (ARM)
|
26444eb714 |
mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF
Patch series "mm: fix UAF caused by race between ptdump and vmap pgtable freeing", v6. Kernel page table walkers fall into two broad categories - those ranges where no exclusion is required via walk_kernel_page_table_range_lockless() and those where exclusion is required via walk_kernel_page_table_range() or walk_page_range_debug(). The former category is used only by arm64 arch code operating on ranges it both wholly owns and does not concurrently write. The latter category consists of kernel page table walkers operating on ranges that are wholly owned (but which need exclusion against concurrent writers). The lock used for exclusion is the mmap lock, and for kernel ranges this is the mmap lock on init_mm. ptdump is a special case being both the only user of walk_page_range_debug(), and the only case in which it walks ranges it does not own. This presents a problem, as page tables may be freed under ptdump. And indeed there is a use-after-free bug in the kernel as a result, which this series addresses. vmap promotes page tables to huge leaf entries where possible, freeing the lower page table when it does. It does this with no meaningful locks held against concurrent ptdump walks. As a result, use-after-free can currently occur. This series addresses the issue by having the vmap huge promotion logic acquire the mmap read lock while both setting the huge page table entry and freeing the prior leaf page table. The ptdump code already acquires the mmap write lock, so by doing so we ensure that the ptdump walker only ever observes either the huge page table entry or the existing page table entry, and nothing is freed underneath it. A mitigation for this issue was already applied for arm64 in commit |
||
Jakub Kicinski
|
a23b36233d |
Merge branch 'bnge-add-more-functionality'
Vikas Gupta says:
====================
bnge: add more functionality
This patch series adds a few functionality for bnge driver:
Patch 1-2: Implements ndo_set_rx_mode_async().
Patch 3: Adds a dedicated HWRM (Hardware Resource Management) command
sequence to handle explicit interface down and up transitions cleanly.
====================
Link: https://patch.msgid.link/20260731163712.3463362-1-vikas.gupta@broadcom.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
Vikas Gupta
|
1b1e855e43 |
bnge: send hwrm for interface down/up transitions
Firmware expects HWRM_FUNC_DRV_IF_CHANGE on interface down/up transitions to coordinate resource management. Add bnge_hwrm_if_change() to send this notification. Signed-off-by: Vikas Gupta <vikas.gupta@broadcom.com> Reviewed-by: Dharmender Garg <dharmender.garg@broadcom.com> Reviewed-by: Rahul Gupta <rahul-rg.gupta@broadcom.com> Link: https://patch.msgid.link/20260731163712.3463362-4-vikas.gupta@broadcom.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
|
Vikas Gupta
|
fbe3647fd4 |
bnge: add ndo_set_rx_mode_async support
Register bnge_set_rx_mode() as ndo_set_rx_mode_async to handle unicast, multicast, broadcast, and promiscuous filter updates via CFA_L2_SET_RX_MASK. The async variant receives pre-snapshotted address lists from the kernel, allowing the driver to issue sleepable HWRM firmware commands without holding the addr lock. Move uc_update detection to the caller so the async path can compute it directly from the snapshotted UC list before calling bnge_cfg_rx_mode(). Handle -EAGAIN from bnge_hwrm_set_vnic_filter() and bnge_hwrm_cfa_l2_set_rx_mask() on the open path by scheduling a retry via netif_rx_mode_schedule_retry() rather than failing the open. Signed-off-by: Vikas Gupta <vikas.gupta@broadcom.com> Reviewed-by: Dharmender Garg <dharmender.garg@broadcom.com> Reviewed-by: Rahul Gupta <rahul-rg.gupta@broadcom.com> Link: https://patch.msgid.link/20260731163712.3463362-3-vikas.gupta@broadcom.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
|
Vikas Gupta
|
d5d7b17b3f |
bnge: refactor rx mode helpers to accept explicit address lists
Rename bnge_cfg_def_vnic() to bnge_cfg_rx_mode() and update bnge_mc_list_updated() and bnge_uc_list_updated() to accept explicit netdev_hw_addr_list pointers rather than deriving them from the netdev. Add a snapshot parameter to bnge_cfg_rx_mode() to skip netif_addr_lock_bh() when the caller provides a pre-snapshotted list. On the open path (snapshot=false), the live netdev UC list is passed and the addr lock is taken as before. Signed-off-by: Vikas Gupta <vikas.gupta@broadcom.com> Reviewed-by: Dharmender Garg <dharmender.garg@broadcom.com> Reviewed-by: Rahul Gupta <rahul-rg.gupta@broadcom.com> Link: https://patch.msgid.link/20260731163712.3463362-2-vikas.gupta@broadcom.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
|
Jakub Kicinski
|
6183a645aa |
Merge branch 'bnxt_en-bug-fixes'
Michael Chan says: ==================== bnxt_en: Bug fixes This series include 3 bug fixes: 1. queue start bug fix on the VNIC's default ring. 2 refactoring patches preceed the actual bug fix. 2. Bug fix for TPA data corruption seen on some ARM systems. 3. PTP PPS setting bug fix. ==================== Link: https://patch.msgid.link/20260731190937.807270-1-michael.chan@broadcom.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
Keegan Freyhof
|
80eaf88efe |
bnxt_en: Fix PTP PPS setting bug
The existing driver logic is always turning on PTP_CLK_REQ_PPS
regardless of the "on" parameter passed to bnxt_ptp_enable().
During shutdown, PTP_CLK_REQ_PPS may be turned off and this
bug will do the opposite and may trigger a PCIe PTM request TLP.
On some systems this can trigger a PCIe AER.
Fix it by properly configuring PTP_CLK_REQ_PPS based on the "on"
parameter.
Fixes:
|
||
Michael Chan
|
c3faf548a0 |
bnxt_en: Disable EOP for TPA on all chips to prevent data corruption
EOP (End of frame padding) on the AGG ring may cause overlapping of
zero padding at the end of one segment with the next segment's data.
If Relaxed Ordering (RO) is enabled, the zero padding may overwrite
valid data in the next segment and corrupt the data. Older chips
(P5 and older) do not automatically disable RO when EOP is enabled.
On some ARM systems, data corruption was reported on 57508 (P5)
chips with RO enabled.
Always disable EOP on all chips on the AGG rings when TPA is enabled
to fix the data corruption.
Fixes:
|
||
Shravya KN
|
0b137529a8 |
bnxt_en: Refresh VNIC default ring on queue restart if needed
When a queue is restarted, refresh VNIC_CFG for all VNICs whose
default RX ring is the restarted ring. This will eliminate this
possible FW warning caused by a stale default ring in the VNIC:
FW reported unknown error type 10
Fixes:
|
||
|
Shravya KN
|
53f01cd594 |
bnxt_en: Determine and store default RX ring in vnic structure
Each VNIC has a default RX ring. The purpose of the default RX ring is to provide a destination for any packets that cannot be parsed by the RSS logic. Up until now, the default RX ring is always Ring 0. We neglected to take care of this default RX ring when adding the queue restart feature. If ring 0 (default ring) is re-started, it may now have a new FW ring ID after freeing the old one and allocating a new one. The VNIC now may have a stale default ring and it may generate an internal exception. This exception may appear in dmesg: FW reported unknown error type 10 The best way to resolve this issue is to use a more appropriate ring for the default ring instead of always ring 0. Ring 0 may not even be in the RSS table, especially on a new RSS context. This patch adds the logic to determine and store the proper default RX ring for a VNIC. For an RSS VNIC, the default ring is the lowest ring number in the RSS table. The next patch will add proper logic to update the VNIC if the default ring changes after queue restart. Reviewed-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com> Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com> Signed-off-by: Shravya KN <shravya.k-n@broadcom.com> Signed-off-by: Michael Chan <michael.chan@broadcom.com> Link: https://patch.msgid.link/20260731190937.807270-3-michael.chan@broadcom.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
|
Shravya KN
|
1d0fc6c7ea |
bnxt_en: Move RSS table fill outside __bnxt_hwrm_vnic_set_rss()
This is a refactor patch with no change in behavior. The caller will now fill the RSS table before calling __bnxt_hwrm_vnic_set_rss(). In the next patch, we'll add code to determine the default ring for the VNIC when we fill the RSS table. Reviewed-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com> Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com> Signed-off-by: Shravya KN <shravya.k-n@broadcom.com> Signed-off-by: Michael Chan <michael.chan@broadcom.com> Link: https://patch.msgid.link/20260731190937.807270-2-michael.chan@broadcom.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
Guopeng Zhang
|
72f522bbf4 |
mm: memcontrol: update state_local when flushing NMI stats
flush_nmi_stats() updates state[] for kmem and slab counters but leaves
the corresponding state_local[] counters unchanged. Local kmem and slab
statistics therefore miss updates collected through the NMI-safe atomic
path.
Update state_local[] together with state[].
Link: https://lore.kernel.org/20260713085053.2916813-1-guopeng.zhang@linux.dev
Fixes:
|
||
Dev Jain
|
04b09ca57d |
mm/rmap: add anon folio unmap dispatcher
Add ttu_anon_folio() as the common entry point for anonymous folio unmapping. It dispatches to the lazyfree or swapbacked helper as appropriate. Keep restoration of cleared PTEs in the caller, where the PTEs were removed. No functional change intended. Link: https://lore.kernel.org/20260730094559.418003-6-dev.jain@arm.com Signed-off-by: Dev Jain <dev.jain@arm.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes (ARM) <ljs@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Rik van Riel <riel@surriel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Dev Jain
|
0caf74274d |
mm/rmap: refactor anon folio unmap in try_to_unmap_one
Refactor anonymous swapbacked folio unmap to ttu_anon_swapbacked_folio(). While at it, rename the remaining subpage variables to page; folios have pages, not subpages. No functional change intended. Link: https://lore.kernel.org/20260730094559.418003-5-dev.jain@arm.com Signed-off-by: Dev Jain <dev.jain@arm.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes (ARM) <ljs@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Rik van Riel <riel@surriel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Dev Jain
|
0955037b98 |
mm/rmap: refactor some code around lazyfree folio unmapping
For lazyfree folio unmapping, after clearing the ptes we must abort the operation if the folio got dirtied or it has unexpected references. Refactor this logic into a function which will return whether we need to abort or not. If we abort, we restore the ptes and bail out of try_to_unmap_one. Otherwise adjust the rss stats of the mm and jump to a label. Also rename that label from "discard" to "finish_unmap"; the former is appropriate in the lazyfree context, but the code following the label is executed for other successful unmap code paths too, so 'discard' does not sound correct for them. No functional change intended. Link: https://lore.kernel.org/20260730094559.418003-4-dev.jain@arm.com Signed-off-by: Dev Jain <dev.jain@arm.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes (ARM) <ljs@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Rik van Riel <riel@surriel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Dev Jain
|
d4ec557282 |
mm/rmap: add try_to_unmap_poisoned_hugetlb_one
Simplify try_to_unmap_one() by separating the hugetlb parts into try_to_unmap_poisoned_hugetlb_one(). To understand the correctness of the refactoring, the following points are noted: 1. try_to_unmap() is called for hugetlb folios only when they are hwpoisoned. 2. A hugetlb VMA cannot be mlocked. 3. page_vma_mapped_walk() returns at most one hugetlb mapping in a VMA, and that mapping points at the head PFN. 4. We won't ever process a softleaf entry that encodes a hugetlb folio; hugetlb folios are never swapped out, migration entries will be skipped (PVMW_MIGRATION not passed), and device-exclusive does not work for hugetlb. 5. The hwpoison entry is constructed from the poisoned folio, just as in the pre-refactor code. Any previous uffd-wp state is deliberately not preserved for the hwpoison entry. 6. TTU_HWPOISON is always present; for it to not be present, either the folio has to be in swapcache, or mapping_can_writeback() is true (see unmap_poisoned_folio), none of which is true for hugetlb folios. 7. Hugetlb uses separate counters from normal rss counters, therefore update_highwater_rss() need not be called. While at it: - Change VM_BUG_* to VM_WARN_*. - Do not declare variables which are only used once. - Constify some variables. - Add some more VM_WARN_* to assert some invariants. Except the above 4 points, no functional change intended. Link: https://lore.kernel.org/20260730094559.418003-3-dev.jain@arm.com Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Suggested-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Rik van Riel <riel@surriel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Dev Jain
|
af56d6acf4 |
mm/rmap: convert page -> folio for hwpoison checks
Patch series "mm/rmap: Refactor try_to_unmap_one", v4. In preparation for batching anonymous large folio unmapping to optimize it, refactor try_to_unmap_one. This series refactors hugetlb, anon-lazyfree and anon-swapbacked logic into their own functions, significantly reducing the length of the huge try_to_unmap_one. This patch (of 5): try_to_unmap() receives hugetlb folios only from the hwpoison path. hugetlb_update_hwpoison() sets the hugetlb folio's head-page hwpoison bit, and page_vma_mapped_walk() reports the hugetlb mapping at the head PFN, so the previous PageHWPoison(subpage) check happened to work for hugetlb. For non-hugetlb folios, unmap_poisoned_folio() currently rejects large folios before calling try_to_unmap(). Hence it is always the case that if try_to_unmap_one() handles an hwpoisoned folio, then the head page is marked with the poison bit. Therefore, convert the poisoned subpage checks to folio_test_hwpoison(). No functional change intended, except that, while at it, convert VM_BUG_* to VM_WARN_*. Link: https://lore.kernel.org/20260730094559.418003-1-dev.jain@arm.com Link: https://lore.kernel.org/20260730094559.418003-2-dev.jain@arm.com Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Rik van Riel <riel@surriel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
Muchun Song
|
1edcd9d7e5 |
selftests/mm: remove obsolete hugetlb vmemmap test
The hugetlb vmemmap selftest was added to check the old HVO layout where tail vmemmap pages reused the head page. That assumption no longer matches the current HVO mapping layout. HVO now keeps a private backing page for the head vmemmap page and remaps redundant tail vmemmap pages to a shared read-only backing page. The old page flag check is therefore testing an obsolete implementation detail rather than the current ABI or behavior. Remove the stale test and its build and run entries. [akpm@linux-foundation.org: don't clean old executables out of .gitignore] Link: https://lore.kernel.org/CAHk-=wiHK5_oBUdUiNAaevmN9f-ORe+QBqbRefAZaw-RbgEn3w@mail.gmail.com Link: https://lore.kernel.org/20260710092427.3459121-1-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: SJ Park <sj@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
Usama Arif
|
ce579dcf73 |
mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd
madvise_free_pte_range() checks pmd_trans_huge(*pmd) unlocked, then
madvise_free_huge_pmd() takes pmd_trans_huge_lock(). pmd_is_huge()
returns true for a device-private PMD, so orig_pmd can be device-private
and enter the !pmd_present() branch.
Skip device-private PMDs in that non-present branch and continue to out
before calling pmd_folio(). Downgrade the check to VM_WARN_ON_ONCE() so
an unexpected PMD softleaf logs a warning rather than panicking. Drop the
thp_migration_supported() guard: it expands to
IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
pmd_is_migration_entry() and pmd_is_device_private_entry() already return
false when that config is not selected, so the guard suppresses only the
case where the warning would already be silent.
Potential trigger: an HMM-based GPU driver races with madvise(MADV_FREE):
migrate_vma_pages() flips the PMD to a device-private entry between the
caller's pmd_trans_huge() check and the callee's pmd_trans_huge_lock().
Link: https://lore.kernel.org/20260710105557.1987433-4-usama.arif@linux.dev
Fixes:
|
||
|
Usama Arif
|
d4b76d0b03 |
mm/madvise: skip device-private PMDs in cold and pageout walks
madvise_cold_or_pageout_pte_range() takes pmd_trans_huge_lock(), whose
pmd_is_huge() check returns true for a device-private PMD. The subsequent
!pmd_present() branch has a VM_BUG_ON() asserting migration is the only
allowed non-present case; a device-private PMD trips it.
Skip device-private PMDs in that non-present branch and continue to
huge_unlock before calling pmd_folio(). Downgrade the check to
VM_WARN_ON_ONCE() so an unexpected PMD softleaf logs a warning rather than
panicking. Drop the thp_migration_supported() guard: it expands to
IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
pmd_is_migration_entry() and pmd_is_device_private_entry() already return
false when that config is not selected, so the guard suppresses only the
case where the warning would already be silent.
Potential trigger: an HMM-based GPU driver races with
madvise(MADV_COLD)/MADV_PAGEOUT: pmd_trans_huge(*pmd) reads true, then
migrate_vma_pages() flips the PMD to a device-private entry before the PMD
lock is acquired.
Link: https://lore.kernel.org/20260710105557.1987433-3-usama.arif@linux.dev
Fixes:
|
||
|
Usama Arif
|
2858b4027f |
mm/mempolicy: skip non-present PMDs when queueing folios
Patch series "mm: handle device-private PMDs in walk callbacks", v3. Since commit |
||
|
Lorenzo Stoakes
|
f1966d954b |
tools/testing/vma: output compared expression on ASSERT_[EQ, NE]()
Update the macros to output the compared values at hex for easier debugging when test asserts fail. We have to be careful not to re-evaluate expressions as they may have side-effects. So update the code to take local copies and use these for both the test and the debug output. Also remove unused IS_SET() macro. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-33-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
fe95ccb1ba |
tools/testing/vma: default VMA, mm flag bits to 64-bit
With all of the sanitisers turned on, setting the VMA and mm flag bits depth to 128 by default results in overly long build times. Reduce this to 64 - we can always manipulate these later for testing of larger bitmaps as needed. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-32-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
9e98b6dd46 |
mm/vma: use guard clauses in can_vma_merge_[before, after]()
Rather than combining a bunch of conditionals in a single expression, simplify by inverting the mergeability requirements into guard clauses. that is - instead of checking what must be true for the conditions to be met, instead check the inverse of the requirements and return false if any are true, defaulting to true. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-31-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
dd7d7d00c0 |
mm/vma: correct incorrect vma.h inclusion
The only files which should be including vma.h are the implementation files for the core VMA logic - vma.c, vma_init.c, and vma_exec.c. This is in order to allow for userland testing of core VMA logic. In this cases, vma_internal.h and vma.h are included, providing both the dependencies upon which the core VMA logic requires and its declarations. Userland testable VMA logic is achieved by having separate vma_internal.h implementations for userland and kernel. Callers other than the core VMA implementation should include internal.h instead. This header does not need to include vma_internal.h as it only contains the vma.h declarations, for which the includes already present suffice. Update code to reflect this, update comments to reflect the fact there are 3 VMA implementation files and document things more clearly. While we're here, slightly improve the language of the comment describing vma_exec.c. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-30-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
613562c29f |
mm/vma: introduce and use vma_set_pgoff()
In order to lay the foundation for work that permits us to track the virtual page offset of MAP_PRIVATE file-backed mappings, we abstract the assignment of vma->vm_pgoff to vma_set_pgoff(). We additionally add a lock check here using the newly introduced vma_assert_can_modify(). This asserts the VMA write lock if the VMA is attached. We also assert that, if this is an anonymous VMA and unfaulted, that its (virtual) page offset is equal to the page offset of the VMA's address. We must be careful about MAP_PRIVATE-/dev/zero which violates fundamental assumptions about anonymous memory, so we check for !vma->vm_file after using vma_is_anonymous() which these mappings satisfy. Additionally, we only perform the assert if CONFIG_MMU is defined, as nommu does not set vma->vm_pgoff = addr >> PAGE_SHIFT. This isn't really relevant to rmap as it has no anon rmap (nor needs it), but we must avoid it asserting falsely. All of this logic is kept in assert_sane_pgoff() to keep things clear. In order to maintain correctness given this assert, we also update __install_special_mapping() to invoke vma_set_range() after it's set vma->vm_ops (which determine whether the VMA is anonymous or not). We do not use vma_set_pgoff() in vm_area_init_from(), as at the point of forking, we don't necessarily have correct locking state. Updating vma_set_range() covers most cases, but in addition to this we also update insert_vm_struct(), compat_set_vma_from_desc() and nommu callers. We also update vma_add_pgoff() and vma_sub_pgoff() to use vma_set_pgoff(). While we're here, we drop a BUG_ON() and update insert_vm_struct()'s comment to reflect the fact anonymous mappings can be added here. Finally, we update the CONFIG_MMU, CONFIG_PER_VMA_LOCK defines in the VMA userland tests so IS_ENABLED() will work correctly with them. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-29-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
bcb52fc56e |
mm/vma: slightly rework the anonymous check in __mmap_new_vma()
Determine if the VMA is anonymous early on, so we separate the logic which invokes callbacks from the rest of the logic. This is in order that a subsequent commit which asserts correct virtual page offset for anonymous mappings correctly asserts this for new anonymous mappings. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-28-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
ce4ec991da |
mm/vma: update vmg_adjust_set_range() to offset pgoff instead
We are calculating the pgoff as an offset, since we have vma_add_pgoff() and vma_sub_pgoff() available, just offset this value directly and use __vma_set_range() for vma->vm_[start, end] values. We take care to update the range before offsetting the page offset, so the adjusted VMA's vm_start and vm_pgoff are mutually consistent at the point the page offset helpers operate - this matters once vma_set_pgoff() comes to assert invariants which relate the two. Doing so lays the foundation for future work which allows for use of virtual page offsets for MAP_PRIVATE-file backed mappings. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-27-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
6387563e6e |
mm/vma: update vma_shrink() to not pass start, pgoff parameters
vma_shrink() is only used by relocate_vma_down() to shrink the tail of a VMA. Therefore neither the start nor the pgoff parameters make any sense. It seemed we were passing the pgoff parameter solely to satisfy vma_set_range()'s requirement for pgoff being specified. Since vma_set_range() is now isolated to vma.c, we can simply introduce __vma_set_range() which sets only vma->vm_[start, end], and invoke this instead, removing start and pgoff from vma_shrink() altogether. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-26-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
775659f8fc |
mm/vma: make vma_set_range() static, drop insert_vm_struct() decl
With __install_special_mapping() moved to vma.c, vma_set_range() can be made into a static function there and is now completely isolated from the rest of mm. While we're here, we can also remove the insert_vm_struct() declaration from mm.h - the function is implemented in vma.c and already declared in vma.h, and has no users outside of mm. Also update the VMA userland tests to reflect this change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-25-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
7578fc092a |
mm/vma: move __install_special_mapping() to vma.c
This function is operating on VMAs and rightly belongs in vma.c, where it can be subject to VMA userland testing and allows us to isolate it from the rest of mm. The _install_special_mapping() function will remain in mmap.c as a wrapper, since this is used by architecture-specific code. Doing so allows us to isolate more functions in vma.c for the same reasons. This forms part of work to allow for tracking MAP_PRIVATE file-backed mappings by their anonymous virtual page offset, as doing so allows us to isolate and keep code that interacts with this together. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-24-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
4ea24922ae |
mm/vma: add and use vma_[add/sub]_pgoff()
Add helpers for adding or subtracting to a VMA's page offset, exposed internally for VMA users within mm in mm/vma.h. This is to lay the foundations for tracking anonymous page offset for MAP_PRIVATE file-backed mappings, where adding and subtracting from this value must be reflected in both the file and anonymous offsets. These are used on VMA split and downward stack expansion. No functional change intended. [akpm@linux-foundation.org: use linear_page_delta() in __split_vma(), per Vlastimal & Lorenzo] Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-23-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
e65f4dfac1 |
mm/vma: introduce vma_assert_can_modify()
vma_assert_write_locked() and vma_assert_attached() are useful for their own purposes, however VMA code absolutely does allow the modification of non-write locked VMAs if they are at that point detached (i.e. unreachable from anywhere). It's therefore useful to be able to assert that a VMA is either detached (modification doesn't matter) or write locked (you're explicitly locked for modification). Therefore introduce vma_assert_can_modify() for this purpose. While we're here, make vma_is_attached() available generally - if !CONFIG_PER_VMA_LOCK, then there's no sense in which a VMA is detached (vma_mark_detached() is a noop), so have this default to true in this case. Also update VMA userland tests to reflect this change, correcting the previously open-coded vma_assert_[attached,detached]() there. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-22-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
93d23dff8e |
mm: use linear_page_[index, delta]() consistently
There are a number of places where we open code what linear_page_index() and linear_page_delta() calculate. Replace this code with the appropriate functions for consistency. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-21-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de> [DRM] Acked-by: Kai Huang <kai.huang@intel.com> # for sgx Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Pedro Falcato <pfalcato@suse.de> # for mm Reviewed-by: Ackerley Tng <ackerleytng@google.com> [guest_memfd] Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
ff8fbc9e08 |
mm/vma: remove duplicative vma_pgoff_offset() helper
This is doing what linear_page_index() does, so eliminate it and replace it with linear_page_index(). Update the VMA userland tests to reflect this change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-20-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
4276358f11 |
mm: prefer vma_[start,end]_pgoff() to vma->vm_pgoff in kernel/
Be consistent in using vma_start_pgoff() and vma_end_pgoff(), which clearly indicates which part of the VMA the page offset refers to and aids greppability. This is part of a broader series laying the ground to provide a virtual page offset for MAP_PRIVATE-file backed anon folios. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-19-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Marek Szyprowski <m.szyprowski@samsung.com> # for kernel/dma Reviewed-by: Gregory Price <gourry@gourry.net> Acked-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
5872168de5 |
mm/vma: use vma_start_pgoff(), linear_page_index() in mm code
There are many instances in which linear_page_index() (as well as linear_page_delta()) is open-coded, which is confusing and inconsistent. Additionally, vma->vm_pgoff doesn't necessarily make it clear that this is the page offset of the start of the VMA range. Doing so also aids greppability. So use vma_start_pgoff() in favour of directly accessing vma->vm_pgoff, and linear_page_index() where we can. This also lays the ground for future changes which will add an anonymous page offset in order to be able to index MAP_PRIVATE-file backed anon folios in terms of their virtual page offset. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-18-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: SJ Park <sj@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
1268e1572b |
mm: introduce and use linear_page_delta()
It's often useful to obtain the number of pages a given address lies at within a VMA. Add linear_page_delta() to determine this and update linear_page_index() to make use of it. Add comments to describe both linear_page_delta() and linear_page_index(). We refer the reader to the comment for vma_start_pgoff() so we don't duplicate information about the meaning of a VMA's page offset, which is also less likely to result in bitrot. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-17-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Ackerley Tng <ackerleytng@google.com> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
68d307442b |
mm/vma: minor cleanup of expand_[upwards, downwards]()
Adjust the stack expansion functions expand_upwards() and expand_downwards() such that they are expressed in terms of named constant values, and make use of vma_start_pgoff(). This clearly documents that we are referencing the page offset of the start of the VMA. Additionally this cleans up the overflow check in expand_upwards(). No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-16-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
a30574d073 |
mm/vma: refactor vmg_adjust_set_range() for clarity
Add comments with ASCII diagrams to describe what we're doing, avoid dubious use of PHYS_PFN(), and use vma_start_pgoff(). The most complicated scenario represented here is vmg->__adjust_next_start - when this is set, vmg->[start, end] actually indicate the range to be retained, so take special care to describe this accurately. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-15-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |