note_page_flush() calls note_page() with addr == 0 and level == -1 to
dump the last row of a ptdump. addr == 0 (1 << 64 wrapped around)
renders a huge region with enormous size for address spaces with
IA bits < 64. For example the stage-2 page tables and the EFI runtime
page table.
More importantly, the last region of the address space and everything
after the address space up to 1 << 64 are merged into one row of
output. If the last region within the address space is valid, it will
appear to remain valid up to 1 << 64 with the same attributes.
Currently only the EFI runtime ptdump is affected by this, but KVM will
soon fix its stage-2 ptdump by using note_page_flush(). Here is an
example of an EFI runtime ptdump (last row):
0x0000008000000000-0x0000000000000000 17179868672G PGD
With this patch:
0x0000008000000000-0x0001000000000000 261632G PGD
To fix this, cache the end address of a ptdump in ptdump_pg_state so
note_page_flush() can call the final note_page() with the correct end
address.
Fixes: 9d80448ac9 ("efi/arm64: Add debugfs node to dump UEFI runtime page tables")
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
* for-next/selftests:
tools: Ensure tools copy of linux/filter.h exports the UAPI
kselftest/arm64: Fix abi test compilation errors
kselftest/arm64: Don't write to P0 in irritator on SME only systems
kselftest/arm64: Add testcase for SECCOMP_RET_TRACE orig_x0 bypass
kselftest/arm64: Add seccomp ptrace x0 bypass test
kselftest/arm64: fp-ptrace: Fix checks for inactive SVE and SSVE regsets
* for-next/sdei:
arm64: escalate smp_send_stop() to an SDEI NMI as a last resort
drivers/firmware: add SDEI cross-CPU NMI service for arm64
firmware: arm_sdei: add SDEI_EVENT_SIGNAL support
firmware: arm_sdei: add sdei_is_present()
* for-next/ptrace:
arm64: syscall: Pass 'orig_x0' as first argument to native system call
arm64: ptrace: Keep 'orig_x0' in-sync with x0 on syscall entry
arm64/fpsimd: ptrace: Fix inactive SVE and SSVE regsets
* for-next/nmi:
arm64: entry: Avoid unnecessary local_irq_disable() on kernel exit
irqchip/gic-v3: make the unmasking of pseudo-NMIs explicit when handling IRQs
arm64: entry: mask DAIF before returning from C EL1 handlers
arm64: suspend: Initialize PMR on resume
arm64: suspend: rely on daif helpers to handle PMR
arm64: hibernate: Restore DAIF state on error
arm64: hibernate: mask DAIF before restoring hibernated kernel
arm64: debug: don't mask DAIF for mdscr_write()
arm64: ptrace: Remove INIT_PSTATE_EL2
* for-next/mm:
arm64/efi: Avoid voluntary preemption with efi_mm installed
arm64: mm: Unmap kernel data/bss entirely from the linear map
arm64: mm: fix accidental linear mapping of no-map reserved memory
arm64: pgtable: convert pte_present() from macro to static inline
arm64: mm: Treat all devices as dma-coherent when CLIDR_EL1.LoC == 0
* for-next/misc:
arm64: Disable KCSAN instrumentation in delay.o
MAINTAINERS: arm64: Add Mark Rutland as an official Reviewer
arm64: smp: Fix IPI teardown for GICv5 flow
arm64: futex: Consolidate 'old == new' check in __lsui_cmpxchg32()
arm64: ftrace: allow DIRECT_CALLS without CALL_OPS
arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS
* for-next/coco:
arm64: RSI: fix field-spanning write warning in attestation token init
virt: arm-cca-guest: Drop unused assignment of platform_device_id driver data
arm64/coco: Add pKVM as a CC platform
arm64/mm: Simplify SWIOTLB setup in arch_mm_preinit()
virt: arm-cca-guest: use migrate_disable() for attestation token requests
Gus reports a bad kernel memory access when using software PAN
(CONFIG_ARM64_SW_TTBR0_PAN=y) on a machine with support for EFI runtime
services:
Unable to handle kernel access to user memory outside uaccess routines
at virtual address 00000000f322ff30
Mem abort info:
ESR = 0x0000000096000004
FSC = 0x04: level 0 translation fault
Internal error: Oops: 0000000096000004 [#1] SMP
Workqueue: efi_rts_wq efi_call_rts
pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : efi_call_rts+0xd8/0x288
Call trace:
efi_call_rts+0xd8/0x288 (P)
process_one_work+0x178/0x4f8
worker_thread+0x194/0x328
This is because the fpsimd context management code called from
__efi_fpsimd_begin() can preempt voluntarily, returning later to the EFI
code with an incorrect value for TTBR0_EL1 thanks to the deferred mm
switching used by the software PAN implementation.
Since EFI runtime services cannot preempt voluntarily and because the
fpsimd switching code does not rely on the TTBR0_EL1 mappings, simply
reorder the fpsimd switch so that it occurs before we change the
page-table.
Cc: Ard Biesheuvel <ardb@kernel.org>
Reported-by: Gus Bourg <gus@bourg.net>
Tested-by: Gus Bourg <gus@bourg.net>
Fixes: a5baf582f4 ("arm64/efi: Call EFI runtime services without disabling preemption")
Link: https://lore.kernel.org/all/20260806000144.3388823-1-gus@bourg.net/
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
The following BTI exception was seen when loading a livepatch module:
Internal error: Oops - BTI: 0000000036000001 [#1] SMP
pstate: 634004c9 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=jc)
pc : kill_orphaned_pgrp+0x0/0x150
lr : do_exit+0x498/0xaf0 [livepatch_combined]
The problem is that the patch module's do_exit() is branching to a
static function in vmlinux using a module PLT veneer (indirect branch),
but the target function doesn't have a BTI landing pad.
Clang 21+ omits the landing pad for static functions which can only be
reached by a direct branch. That's normally fine for ordinary modules
which only branch to global exported functions, but Mark Brown points
out [1] that this isn't guaranteed if the module branches between
sections. Futhermore, livepatch modules use klp relocations to reference
arbitrary kernel symbols, so with CONFIG_RANDOMIZE_MODULE_REGION_FULL
the module is far enough from the kernel that every R_AARCH64_CALL26
needs a PLT.
Put Clang 21+ in the naughty corner alongside GCC, which suffers from
the same issue, by disabling CONFIG_ARM64_BTI_KERNEL until we have a
version of the toolchain with the problem resolved.
Cc: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/da06bbd3-d04b-4d0f-b331-f5b91bc373a5@sirena.org.uk [1]
Fixes: fd1e0fd71f ("arm64: Implement HAVE_LIVEPATCH")
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
[will: Stitched together commit message, diff and bug number]
Signed-off-by: Will Deacon <will@kernel.org>
Currently, when exiting to kernel mode, we attempt involuntary
preemption. The preemption logic expects IRQs to be disabled, which is
why we call local_irq_disable() before attempting preemption.
However, depending on the context, local_irq_disable() may be
unnecessary:
- __el1_irq(), the non-NMI EL1 IRQ path, already has IRQs disabled, so
local_irq_disable() is redundant.
- irqentry_exit_to_kernel_mode_preempt() immediately returns when
exiting from an NMI-like context, so calling local_irq_disable()
beforehand is unnecessary work.
Furthermore, it confuses the pNMI state tracking when we are in a
context with interrupts disabled and the GIC_PRIO_PSR_I_SET bit is set
in the PMR, leading to a warning when
CONFIG_ARM64_DEBUG_PRIORITY_MASKING=y:
WARNING: ./arch/arm64/include/asm/irqflags.h:63 at arm64_exit_to_kernel_mode+0xb8/0xc0, CPU#40: retsnoop/31805
CPU: 40 UID: 0 PID: 31805 Comm: retsnoop Not tainted 7.2.0-rc6-next-20260805 #7 PREEMPTLAZY
pstate: 234013c9 (nzCv DAIF +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
pc : arm64_exit_to_kernel_mode (arch/arm64/kernel/entry-common.c:63)
lr : el1_abort (arch/arm64/kernel/entry-common.c:323)
pmr: 000000f0
Call trace:
arm64_exit_to_kernel_mode (arch/arm64/kernel/entry-common.c:63) (P)
el1_abort (arch/arm64/kernel/entry-common.c:323)
el1h_64_sync_handler (arch/arm64/kernel/entry-common.c:449)
el1h_64_sync (arch/arm64/kernel/entry.S:589)
[...]
Split arm64_exit_to_kernel_mode() into preempt, non-preempt, and
dispatch parts so that we can avoid this extra work where it is not
needed and avoid breaking the pNMI tracking logic.
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: ae654112ea ("arm64: entry: Use split preemption logic")
Link: https://lore.kernel.org/all/20260807-arm64_fix-v1-1-d069ccf9d71b@debian.org/
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
`gic_arch_enable_irqs()` is only used when handling IRQs (which could
be pseudo-NMIs) and unmasking pseudo-NMIs.
The chain of `gic_pmr_mask_irqs()` and `gic_arch_enable_irqs()` for it
is slightly confusing without further explanation.
Remove `gic_arch_enable_irqs()` and instead do the whole pseudo-NMI
umasking in `gic_unmask_pnmis()`, making the operation explicit.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
KCSAN relies on udelay() for injecting delays. To avoid recursively
triggering a watchpoint, where KCSAN sets up watchpoint on an address
that is accessed by udelay() in the same thread, disable instrumentation
in arm64's delay implementation.
Paul found a manifestation of this as follows:
| BUG: KCSAN: data-race in __delay / set_need_resched_current
|
| read (marked) to 0xffff000005899b48 of 8 bytes by interrupt on cpu 8:
| __delay+0xb0/0x378
| __udelay+0x4c/0x60
| kcsan_setup_watchpoint+0x3b4/0x820
| __tsan_unaligned_write4+0x228/0x26c
| set_need_resched_current+0x138/0x1a8
| rcu_exp_handler+0x418/0x4a0
| __flush_smp_call_function_queue+0x36c/0x4a0
| generic_smp_call_function_single_interrupt+0x20/0x30
| ipi_handler+0xec/0x558
| handle_percpu_devid_irq+0x220/0x2a0
| generic_handle_domain_irq+0x84/0xb4
| gic_handle_irq+0x64/0x144
| call_on_irq_stack+0x30/0x48
| do_interrupt_handler+0x80/0xb8
| el1_interrupt+0x3c/0x60
| el1h_64_irq_handler+0x18/0x24
| el1h_64_irq+0x6c/0x70
| smp_call_function_single+0x18c/0x25c
| sync_rcu_exp_select_node_cpus+0x534/0x8bc
| rcu_exp_sel_wait_wake+0x358/0xef4
| wait_rcu_exp_gp+0x30/0x44
| kthread_worker_fn+0x1b4/0x5dc
| kthread+0x1d8/0x204
| ret_from_fork+0x10/0x20
|
| write to 0xffff000005899b4c of 4 bytes by interrupt on cpu 8:
| set_need_resched_current+0x138/0x1a8
| [...]
This matches what is already done in arch/x86/lib/Makefile.
Reported-by: "Paul E. McKenney" <paulmck@kernel.org>
Fixes: dd03762ab6 ("arm64: Enable KCSAN")
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
When a user unbinds an MSC and that MSC is the only MSC left for a
component then the corresponding mpam_component will be freed. If the user
then goes on to read the schemata file in the resctrl filesystem then the
mpam_component will be accessed from resctrl_arch_get_config() leading to a
use after free.
As the MPAM driver is not a module the unbind sysfs interface is the only
way to trigger the remove. Instead of dealing with the complexity of
allowing some unused MSC to unbind just remove the unbind sysfs interface.
Fixes: f04046f257 ("arm_mpam: Add probe/remove for mpam msc driver and kbuild boiler plate")
Signed-off-by: Ben Horgan <ben.horgan@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
If a user unbinds an MSC after mpam_disable() has been run in response
to an error interrupt then a dereference of a NULL pointer occurs as
mpam_disable() sets the drvdata to NULL. Add an early return to the driver
remove callback to avoid this.
Fixes: f04046f257 ("arm_mpam: Add probe/remove for mpam msc driver and kbuild boiler plate")
Signed-off-by: Ben Horgan <ben.horgan@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
PERF_SAMPLE_BRANCH_HW_INDEX is supported by BRBE so hw_id is passed to
userspace, but it's never set by the BRBE driver. Zero initialize it as
it should be according to the docs:
* For the architectures whose raw branch records are
* already stored in age order, the hw_idx should be 0.
It's probably too risky to remove PERF_SAMPLE_BRANCH_HW_INDEX from BRBE
now in case anyone is setting it and reading the value, but zero
initializing the whole struct also protects against the same issue with
new fields that are added in the future.
Fixes: 58074a0fce ("perf: arm_pmuv3: Add support for the Branch Record Buffer Extension (BRBE)")
Signed-off-by: James Clark <james.clark@linaro.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The linear aliases of the kernel text and rodata are also mapped
read-only in the linear map. Given that the contents of these regions
are mostly identical to the version in the loadable image, mapping them
read-only and leaving their contents visible is a reasonable hardening
measure.
Data and bss, however, are now also mapped read-only but the contents of
these regions are more likely to contain data that we'd rather not leak.
So let's unmap these entirely in the linear map when the kernel is
running normally.
When going into hibernation or waking up from it, these regions need to
be mapped, so map the region initially, and toggle the valid bit so
map/unmap the region as needed.
Doing so is required because pages covering the kernel image are marked
as PageReserved, and therefore disregarded for snapshotting by the
hibernate logic unless they are mapped.
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Liz Prucka <lizprucka@google.com>
Cc: Seth Jenkins <sethjenkins@google.com>
Cc: Kees Cook <kees@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Commit 94104e3cfa ("arm64: cpufeature: Rename BBML2_NOABORT as BBML3")
renamed the cpu_supports_bbml2_noabort() helper to cpu_supports_bbml3(),
as the Linux-defined "noabort" semantics have now been incorporated into
the architecture under the BBML3 feature.
Update the caller in the SMMUv3 SVA driver to use the new function.
There is a slightly oddity in that the SMMUv3 architecture already
defined BBML2 in such a way that aborts were prohibited, so we use the
BBML3 feature on the CPU to enable BBML2 in the SMMU.
Fixes: 94104e3cfa ("arm64: cpufeature: Rename BBML2_NOABORT as BBML3")
Signed-off-by: Will Deacon <will@kernel.org>
If you were disappointed at how minimal the initial CMN S3 support
looked compared to previous versions, then oh boy is it time to put your
party hats on... The biggest batch of incompatible changes yet comes not
with a new CMN product, but a point release of an existing one. We've
got new filters, loads of changes to existing events, register fields
moved around for no good reason, and much, much more! On the upside, we
do at least gain a means of working around the isolation feature.
As such, for the sake of sanity in the driver it is easiest to split it
into a distict "model" for our internal abstractions despite it bearing
the same part number as r0/r1.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add the logic to handle events with the upcoming new filter controls.
Since for now we will have the sole invariant of all EVICT_STATE_SEL
events having HBT_LBT_SEL as a secondary filter, for the sake of
simplicity we can just special-case that, and save the complication
of a full multi-filter abstraction until unavoidably necessary.
Reviewed-by: Leo Yan <leo.yan@arm.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
The ABI hole I have dug myself into requires the driver to know which
event encodings are associated with which particular filter control.
Since we will soon have a notion of multiple filters per event, refactor
the event data to encapsulate filters in an explicit structure, which
can then more easily scale as an array in future.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
We're soon going to need to cope with events having multiple filters,
plus the filter fields themselves moving around, wherein any more inline
if/else logic will struggle to scale. Add a more general abstraction for
the node-specific filter controls, and rejig the pmu_event_sel filter
programming around it in a more extensible manner.
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
CMN has already grown many more event-specific filters than the original
Occupancy ID, but since they are all independent of each other we've
just overloaded them onto the same name. Before we add yet more, and
they begin to overlap, rename all our "occupid" variables to "filter" so
that things can be a bit clearer and more consistent (but leaving the
format attribute itself, to avoid UAPI concerns).
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
When reserved-memory regions with the "no-map" property are not
page-aligned, the kernel may accidentally map them into the linear
mapping, contradicting the no-map semantics.
The root cause is a mismatch between /proc/iomem's address boundaries
and the actual page table mapping boundaries:
1. /proc/iomem derives its ranges from memblock via
memblock_region_reserved_base_pfn/memblock_region_reserved_end_pfn,
which perform PFN rounding so the displayed boundaries are
page-aligned. This gives the impression that the no-map region
occupies whole pages.
2. However, memblock_mark_nomap() splits memblock.memory regions at
exact byte boundaries (memblock_isolate_range preserves raw DT
base/size with no alignment). When for_each_mem_range iterates the
non-NOMAP regions adjacent to a no-map region, it returns start/end
values that are NOT page-aligned — they are the precise byte
boundaries from the memblock split.
3. These sub-page-aligned values are passed to
__create_pgd_mapping_locked(), which does:
phys &= PAGE_MASK;
addr = virt & PAGE_MASK;
end = PAGE_ALIGN(virt + size);
The downward rounding of phys via PAGE_MASK extends the mapped
range backward into the adjacent no-map region, effectively
including no-map memory in the linear mapping.
For example, with 64K pages, reserved_region@A2000000 (base=0xA2000000,
size=0x8000, no-map) causes for_each_mem_range to return
start=0xA2008000 for the next mappable region. After phys &= PAGE_MASK,
the actual mapping starts at 0xA2000000 — the entire no-map region is
incorrectly mapped.
Fix this by rounding the mappable range inward to PAGE_SIZE boundaries
before passing it to __map_memblock: start is rounded UP and end is
rounded DOWN. This ensures the mapped area never overlaps with adjacent
no-map regions. The cost is at most one page of unmapped gap at each
boundary, which is preferable to violating no-map semantics.
Signed-off-by: liulhong617 <liulhong617@gmail.com>
Signed-off-by: Will Deacon <will@kernel.org>
Normally when there is an include/foo.h and an include/uapi/foo.h the
non-UAPI copy includes the UAPI copy. This is the case for the in
kernel copy of linux/filter.h but not for the copy in tools/ which
results in build breaks for the newly added arm64
seccomp_ptrace_x0_bypass selftest. Add an explicit include of the uapi
to fix the test and avoid future surprises.
Fixes: 2fcbc4adf9 ("kselftest/arm64: Add seccomp ptrace x0 bypass test")
Fixes: f143c11bb7 ("tools: bpf: Use local copy of headers including uapi/linux/filter.h")
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
The arm64 ABI selftests fail to compile due to missing include paths
for kernel headers, causing errors like incomplete type struct sock_filter
and implicit BPF macro declarations.
Add $(KHDR_INCLUDES) and -I$(top_srcdir)/tools/include to CFLAGS
to resolve the header search path. Also remove the hardcoded __NR_write
macro and include <asm/unistd.h> to obtain the correct syscall number.
Fixes: 21e37da120 ("kselftest/arm64: Add testcase for SECCOMP_RET_TRACE orig_x0 bypass")
Fixes: 2fcbc4adf9 ("kselftest/arm64: Add seccomp ptrace x0 bypass test")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/r/202608021842.jp6IBrFi-lkp@intel.com/
Suggested-by: Mark Brown <broonie@kernel.org>
Reviewed-by: Mark Brown <broonie@kernel.org>
Tested-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
Overriding the idle mechanism might be useful for debugging and performance
testing. Add a cmdline parameter for it, similar to the existing idle=
parameter already present for the x86 and ppc architectures.
It is also useful on platforms where the WFI instruction misbehaves,
such as Apple Silicon SoCs. Generally, a misbehaving instruction should
be treated as an erratum and patched using the alternatives framework.
However, in the Apple Silicon case we need more flexibility because it is
difficult to detect whether the erratum applies. For example, Linux VMs
inside macOS have the same MIDR and may even seem like they're running
in EL2 in the case of NV, but should continue using WFI (it's trapped and
handled correctly by the hypervisor there). Thus, we prefer to
let the m1n1 bootloader add the idle=nop parameter[1].
Link[1]: https://lore.kernel.org/all/99b69262-e54b-424e-baa2-96ef7013b87a@kernel.org/
Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Yureka Lilian <yureka@cyberchaos.dev>
Signed-off-by: Will Deacon <will@kernel.org>
Most EL1 exceptions already call local_daif_mask() before returning,
with the exception of debug exception handlers which do not change
DAIF, and the IRQ/FIQ/Error handlers.
However, DAIF get masked in kernel_exit() in all cases when returning
from EL1 C handlers anyway.
Move this masking from assembly to C by calling local_daif_mask()
before irqentry_nmi_exit(). Unlike the raw DAIF masking helper,
local_daif_mask() invokes trace_hardirqs_off(), so it must execute
while RCU is still watching.
Remove the disable_daif assembly macro, as this was its only use.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
When we resume from cpu_suspend() context tracking, specially,
ct_idle_exit() performs IRQ save/restore sequence. It doesn't cause
any functional issues since we have masked all exceptions prior
suspend and have not restored them. However, in case of pseudo-NMI PMR
can be set by firmware to arbitrary value, thus IRQ save/restore
routines manipulates this arbitrary value. Again, it doesn't cause any
issues since PMR variant of IRQ save helper carries a
__pmr_irqs_disabled_flags() guard.
Going forward __pmr_irqs_disabled_flags() guard will be gone and we
will call __pmr_local_irq_disable() unconditionally - that would cause
warning in case CONFIG_ARM64_DEBUG_PRIORITY_MASKING is set.
Initialize PMR to a value known to Linux on resume until the normal
exception restore path restores the saved DAIF and PMR state.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Commit 77345ef704 ("arm64: suspend: Use cpuidle context helpers
in cpu_suspend()") added cpuidle helpers to handle PMR manipulation
and restoration to ensure that the CPU receives interrupts when suspended
and pseudo-NMIs are enabled.
However, those helpers are called in between a pair of `local_daif_save()`
and `local_daif_restore()`, which already configure the PMR as expected.
Effectively, `arm_cpuidle_save_irq_context()` is a no-op here,
even when using pseudo-NMIs, and `arm_cpuidle_restore_irq_context()`
would not restore proper interrupt masking configuration early enough
if there were unexpected changes during suspend or resume.
(This can be observed with Trusted Firmware A (TF-A) at EL3 handling
suspend through PSCI. Even though it should not be the case, TF-A can
reset `ICC_PMR_EL1` during CPU_SUSPEND, thus resuming the kernel
with an inconsistent priority mask value on hardware implementing
more than the minimum number of priority levels, such as Morello.)
Thus : remove the cpuidle context helpers as they do not do anything,
but keep the comment mentioning the need for interrupts to reach the CPU
if we are using pseudo-NMIs.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
Sashiko AI has reported that if swsusp_mte_save_tags() for some reason
fails we return from swsusp_arch_suspend() with DAIF being masked -
that is not what we'd expect. Restore the saved DAIF state before
returning from the error path.
Fixes: ee11f332af ("arm64: mte: Save tags when hibernating")
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
The arm64 hibernate code manages the exception masking in an unsound
way, leading to potential crashes and/or warnings during resume.
When a hibernation image is saved in `swsusp_arch_suspend()`, all DAIF
exceptions are masked (by virtue of `local_daif_save()`), and the
suspended image is saved assuming that all DAIF exceptions will remain
masked when the image is restored.
When a hibernation image is resumed by `swsusp_arch_resume()`, only
interrupts are masked (by virtue of `local_irq_disable()` in
`resume_target_kernel()`). When pseudo-NMI is enabled the DAIF.IF bits
will be clear, and regardless of pseudo-NMI the DAIF.DA bits will be
clear.
This means that there are two problems:
(1) It is possible to take Debug, SError, or pseudo-NMI exceptions
during the resume process. This is unsafe, as during the resume
process both the old ane new kernels will tranisently be in an
inconsistent state, and swsusp_arch_suspend_exit() won't retain
an executable mapping of any exception vectors.
Any exception taken here will be fatal and silent.
(2) When re-entering the resumed kernel, some DAIF bits will be clear
unexpectedly. This permits Debug, SError, or pseudo-NMI exceptions
to be taken for a short period while the resumed kernel is not yet
in a consistent state.
This is detected by CONFIG_ARM64_DEBUG_PRIORITY_MASKING.
Avoid these issues by masking all DAIF exceptions during resume.
Fixes: 82869ac57b ("arm64: kernel: Add support for hibernate/suspend-to-disk")
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
Masking DAIF around the write to MDSCR_EL1 doesn't do anything: we can
write to sysregs with interrupts unmasked, and writing to PSTATE is
not a context synchronization event so it does not synchronize it.
This is done in the context of a general interrupt handling cleanup,
so it does not address the missing context synchronization for the
MDSCR_EL1 write, staying consistent with the current state. This
should be addressed in a future patch.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Last user of INIT_PSTATE_EL2 has gone with ae4b7e38e9 ("arm64: Allow
sticky E2H when entering EL1"), so remove it.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add ID_AA64MMFR2_EL1.BBM based BBML3 feature detection in
cpu_supports_bbml3() so that cpus with the feature would
not have to be added into MIDR based supports_bbml3_list.
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
[will: Tidy up cpu_supports_bbml3()]
Signed-off-by: Will Deacon <will@kernel.org>
- As bbml2_noabort is functionally equivalent to bbml3,
rename cpu/system_supports_bbml2_noabort to
cpu/system_supports_bbml3.
The ARM64 capability name is also renamed accordingly.
- As BBML2_NOABORT or the equivalent BBML3 is the
kernel requirement for setting up linear map with
block/contpte mappings and not BBML2, replace all
bbml2 references with bbml3.
FEAT_BBML3, is introduced as part of 2025 Architecture Extensions.
https://developer.arm.com/documentation/109697/2026_03/2025-Architecture-Extensions
No functional changes are introduced with this patch.
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add below cpus to the midr list, which supports
BBML2_NOABORT.
Cortex A520(AE)
Cortex A715
Cortex A720(AE)
Cortex A725
Neoverse N3
C1-Nano
C1-Pro
C1-Ultra
C1-Premium
C1-Ultra and C1-Premium both suffer from erratum 3683289,
where Break-Before-Make must be followed to avoid a livelock.
For both CPUs, the erratum is fixed from r1p1.
Hence we do not enable BBML2_NOABORT for CPU revisions <= r1p0.
The relevant SDENs are:
* C1-Ultra: https://developer.arm.com/documentation/111077/9-00/
* C1-Premium: https://developer.arm.com/documentation/111078/9-00/
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add cputype definitions for C1-Nano.
The definition can be found in C1-Nano TRM,
https://developer.arm.com/documentation/107753/0002
as part of MIDR_EL1 bit descriptions.
This is going to be used in the bbml3 support list.
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add cputype definitions for Cortex-A520AE.
The definition can be found in Cortex-A520AE TRM,
https://developer.arm.com/documentation/107726/0001/
as part of MIDR_EL1 bit descriptions.
This is going to be used in the bbml3 support list.
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
pte_present() is used as the basis for both pmd_present() and pud_present().
It is currently implemented as a macro composed of pte_val() and
pte_present_invalid().
When pte_present() or its higher-level variants are used directly with
ptep_get() or pXdp_get(), for example:
pte_present(ptep_get(pte));
pmd_present(pmdp_get(pmd));
pud_present(pudp_get(pud));
the macro expansion causes the compiler to evaluate the argument twice,
resulting in redundant loads. For example, pte_present() expands to:
!pte_val(READ_ONCE(*pte) || pte_present_invalid(READ_ONCE(*pte))
A typical example is pud_free_pmd_page(), where the expansion of
pmd_present() generates:
...
/* pmd_present() (x20 = pmdp) */
1b88: f9400288 ldr x8, [x20] // read pmdp.
1b8c: f9000fa8 str x8, [x29, #0x18]
1b90: 3707fec8 tbnz w8, #0x0, 0x1b68 <pud_free_pmd_page+0xd0>
1b94: f9400288 ldr x8, [x20] // redundant read of pmdp.
1b98: 8a170109 and x9, x8, x23
1b9c: f9000fa8 str x8, [x29, #0x18]
1ba0: f120013f cmp x9, #0x800
1ba4: 54fffe20 b.eq 0x1b68 <pud_free_pmd_page+0xd0>
1ba8: 17fffff4 b 0x1b78 <pud_free_pmd_page+0xe0>
...
Convert pte_present() to static inline function so that prevent the
generation of redundant code and move pte_valid() and
pte_present_invalid() further up so the inline function can use them.
After this change, the generated code becomes:
...
/* pmd_present() (x20 = pmdp) */
1a30: f9400288 ldr x8, [x20]
1a34: 8a170109 and x9, x8, x23
1a38: f9000fa8 str x8, [x29, #0x18]
1a3c: f120013f cmp x9, #0x800
1a40: 54fffe80 b.eq 0x1a10 <pud_free_pmd_page+0xd0>
1a44: 3607fee8 tbz w8, #0x0, 0x1a20 <pud_free_pmd_page+0xe0>
1a48: 17fffff2 b 0x1a10 <pud_free_pmd_page+0xd0>
...
This eliminates the redundant load and also reduces code size at
call sites using this pattern. For example, pud_free_pmd_page() shrinks
from 7,500 bytes to 7,148 bytes, a reduction of approximately 4.7%.
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The 64-bit Programmers' model extension, now named FEAT_CSPMU_EXT64,
makes all the non-counter registers 64-bit as well, of which we most
need to care PMEVFILTR<n>, PMEVTYPER<n>, and PMCR since it changes
location. Our event config fields are ready for this internally, but
we need a few more tweaks to propagate 64-bit values properly from
end to end.
Since 64-bit events leave no room for our existing "cycles" encoding,
and we don't really expect to see implementations of it anyway, we
deliberately omit generic support for the architectural fixed cycle
counter in this case to save some bother. At worst, vendor modules
can still implement their own encoding if they really want to.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Ilkka Koskinen <ilkka@os.amperecomputing.com>
Signed-off-by: Will Deacon <will@kernel.org>
Mark has been part of the arm64 furniture for as long as the code has
existed in mainline and is consistently one of the most active and
knowledgeable reviewers that we have.
Make it official.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Since commit 55b48e23f5 ("genirq/devres: Add error handling in
devm_request_*_irq()"), devm_request_irq() automatically logs
detailed error messages on failure. Remove the now-redundant
driver-specific dev_err() and dev_err_probe() calls.
Signed-off-by: Pan Chuang <panchuang@vivo.com>
Acked-by: Yushan Wang <wangyushan12@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
