- Take a reference on the last IRQ loaded into an LR to prevent it
from being freed while running the guest (Marc Zyngier)
- Ensure that the ITS MOVALL command only affects LPIs that were
previously affined to the source redistributor (Marc Zyngier)
- Fix + test for honoring the host's trap configuration when running
non-protected VMs while KVM is in protected mode (Fuad Tabba)
- Use the host stage-1 mapping granularity for VM_PFNMAP mappings at
stage-2 (Mostafa Saleh)
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQSNXHjWXuzMZutrKNKivnWIJHzdFgUCar6KFRccb2xpdmVyLnVw
dG9uQGxpbnV4LmRldgAKCRCivnWIJHzdFoQxAP4lpoLN5+XlRshepgo+t/hZnhqj
4X6A8t7gtKv1iSp9WgD9FOeekzRH8HunEyjTyez/SmIb4ZZ9yYSMd5wIXX7uqgE=
=FD18
-----END PGP SIGNATURE-----
Merge tag 'kvmarm-fixes-7.3-2' of https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 fixes for 7.3, round #2
- Take a reference on the last IRQ loaded into an LR to prevent it
from being freed while running the guest (Marc Zyngier)
- Ensure that the ITS MOVALL command only affects LPIs that were
previously affined to the source redistributor (Marc Zyngier)
- Fix + test for honoring the host's trap configuration when running
non-protected VMs while KVM is in protected mode (Fuad Tabba)
- Use the host stage-1 mapping granularity for VM_PFNMAP mappings at
stage-2 (Mostafa Saleh)
When commit 2aa53d68ce ("KVM: arm64: Try stage2 block mapping for
host device MMIO") added VM_PFNMAP support to get_vma_page_shift(),
stage-1 page tables did not support huge PFNMAP (as in VFIO-PCI
vfio_pci_mmap_huge_fault()) and transparent_hugepage_adjust() was
unsafe for MMIO as it dereferenced struct page.
Since commit 6011cf68c8 ("KVM: arm64: Walk userspace page tables to
compute the THP mapping size"), transparent_hugepage_adjust() instead
walks the host stage-1 page tables via get_user_mapping_size() without
touching struct page. Meanwhile, commit 3e509c9b03 ("mm/arm64:
support large pfn mappings") enabled stage-1 huge PFNMAP.
So. we can drop the VMA-based VM_PFNMAP size calculation in
get_vma_page_shift() and let transparent_hugepage_adjust() derive
the stage-2 mapping size directly from the populated stage-1 leaf
for non-cacheable mappings as well.
Assisted-by: LLM
Cc: stable@vger.kernel.org
Fixes: 2aa53d68ce ("KVM: arm64: Try stage2 block mapping for host device MMIO")
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Oliver Upton <oupton@kernel.org>
Userspace can hide a feature from a guest by clearing its field in a
writable ID register, and KVM then makes the feature's instructions
UNDEFINED in the guest by trapping or disabling them. No selftest checks
that.
Add a test that runs the instruction of each of TLBI OS, MOPS, TCR2_EL1
and FPMR once with its field as advertised and once with it cleared, and
expects an UNDEF only when cleared. A feature the vCPU doesn't advertise
is skipped, as is hidden TLBI OS on a CPU with neither FGT nor
FEAT_EVT2, where KVM can't trap it.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260929090031.3829185-5-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
For each vCPU, pKVM's EL2 builds its own HCR_EL2 and takes only TWI, TWE
and VSE from the host's value on each entry. For a non-protected VM it
has fallen behind the host's: on a CPU with MTE the guest can read
GMID_EL1, on one with FEAT_EVT2 but no FGT it can execute a TLBI OS its
ID registers hide, an interrupt injected without a vGIC (VI, VF) never
arrives, and set/way emulation loses the TVM trap it relies on.
For a non-protected VM, also take the bits the host varies with the VM's
configuration or at runtime: VI, VF, TVM, TID2, TID4, TID5 and TTLBOS.
They only pend interrupts for, or add traps to, a VM the host controls.
The traps exit to the host, which handles them as it does without pKVM.
The bits that change what EL2 does on entry and exit (E2H, RW, API/APK)
or depend only on the CPU (TEA, TERR, FWB) stay EL2's. A protected VM
still takes only TWI, TWE and VSE.
EL2 now sets TID2 or TID4 only for a protected VM, and never sets ATA,
as the host rejects KVM_CAP_ARM_MTE in pKVM.
Fixes: b56680de9c ("KVM: arm64: Initialize trap register values in hyp in pKVM")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260929090031.3829185-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
pKVM keeps its own copy of each vCPU's HCR_EL2 at EL2, initialised with
RW set unconditionally. Nothing stops userspace from creating a
non-protected AArch32 VM, and with RW set, entering one of its vCPUs is
an illegal exception return: KVM_RUN fails with KVM_EXIT_FAIL_ENTRY.
Clear RW for a vCPU that is AArch32 at EL1, when the CPU has AArch32
EL1. On a CPU without it, RW stays set and the entry still fails, rather
than EL2 switching *32_EL2 registers that are UNDEFINED there. The host
already rejects such a vCPU, but EL2 takes the vCPU's features from the
host and doesn't rely on that.
Fixes: b56680de9c ("KVM: arm64: Initialize trap register values in hyp in pKVM")
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260929090031.3829185-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
FGUs don't actually require hardware support, so don't gate fgt trap
config insertion on FGT. This is the only action required to allow FGUs
always, as kvm_calculate_traps() already computes the FGU bits with or
without FEAT_FGT.
This also fixes a spurious WARN when a TLBI OS is run on a guest in vEL1
without FEAT_TLBIOS on non FEAT_FGT hardware. In this configuration the
course-grained HCR_EL2.TTLBOS trap reaches the handler handle_tlbi_el1(),
which expects an EL1 TLBI only from vEL2. With FGUs enabled the UNDEF
will be injected without needing the specific handler.
Fixes: f5a5a406b4 ("KVM: arm64: Propagate and handle Fine-Grained UNDEF bits")
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260929090031.3829185-2-fuad.tabba@linux.dev
[oliver: Take Wei-Lin's suggested changelog]
Signed-off-by: Oliver Upton <oupton@kernel.org>
The MOVALL command moves the pending state of all LPIs targeting a given
redistributor to another one. However, we seem to have lost the
filtering on the source RD, which means we move all LPIs to the target.
Not quite what the spec mandates.
Hack update_affinity() to take an optional source vcpu that is used as a
filter when non-NULL, restoring the filtering that was performed by
vgic_copy_lpi_list() back in the days.
Fixes: 11f4f8f3e6 ("KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_cmd_handle_movall()")
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260929093548.3598547-7-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
Referencing the last interrupt inserted in an LR is rather fragile, as
this interrupt can vanish if a concurrently unmapped LPI.
Solve this by bumping up the refcount on the interrupt when populating
last_lr_irq, and drop it at vgic_prune_ap_list() time, when LPIs are
being reclaimed.
Fixes: 6da5e537f5 ("KVM: arm64: vgic: Pick EOIcount deactivations from AP-list tail")
Reported-by: Yuchao Zhang <ndaugoing@gmail.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260929093548.3598547-5-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
last_lr_irq is always populated when there is any interrupt populated in
the AP list. Not only this is not necessary (it is only useful when we
completely fill the LRs), but this is in the way of further fixes.
Make sure last_lr_irq is kept to NULL when we LRs are not completely
full.
Fixes: 6da5e537f5 ("KVM: arm64: vgic: Pick EOIcount deactivations from AP-list tail")
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260929093548.3598547-4-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
Explicitly flush caches after intra-host migration before clearing "SEV
active" on the source VM, as doing cache maintenance afterwards creates a
tiny window where memory reclaim could return memory to the host without
performing a cache flush, e.g. as pointed out by Sashiko:
CPU1 in sev_migrate_from():
src->active = false;
CPU2 running concurrent unmap:
Since active is false, the automatic cache flush in
sev_guest_memory_reclaimed is skipped.
The host frees and reallocates the page.
CPU1 in sev_migrate_from():
sev_writeback_caches(src_kvm);
Executes a hardware cache flush (wbnoinvd), which writes the guest's old
dirty ciphertext over the new page owner's data.
Fixes: 93de2a6a4b ("KVM: SEV: Do cache maintenance on the source VM during intra-host migration")
Cc: stable@vger.kernel.org
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260923165304.1662E1F000FF@smtp.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260928154644.2559454-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Nullify have_run_cpus when freeing the mask, particularly in the error path
of __sev_guest_init(), so that KVM doesn't have to subtly use sev->active
to track whether or not the mask has been freed. As pointed out by Sashiko,
blindly freeing the mask in sev_vm_destroy() results in a double-free if
the mask is freed if __sev_guest_init() fails.
Throw the logic in a helper as nullifying the pointer is frustratingly
difficult and weird due to have_run_cpus being a single-entry array when
CPUMASK_OFFSTACK=n. Deliberately don't use CPUMASK_VAR_NULL, as it's not
directly assignable when the cpumask is on-stack, e.g. requires using a
local variable and a memcpy(), which is beyond ridiculous. Furthermore,
while clearing the on-stack bitmask is an unnecessary and arguably unwanted
side effect, KVM absolutely relies on '0' being the "null" value given that
the struct is zero-allocated.
Opportunistically add an alloc() helper to pair with free(); there are just
enough call sites to make doing so worthwhile.
Fixes: 12c1f6e03f ("KVM: SEV: Free have_run_cpus during VM destruction even if VM is no longer SEV")
Cc: stable@vger.kernel.org
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260923165349.CAAF01F000FF@smtp.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260928154644.2559454-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260813223610.2043560-7-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260813223610.2043560-6-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260813223610.2043560-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260813223610.2043560-4-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Add a selftest to verify that KVM intercepts x2APIC MSR accesses for L1
after APICv is inhibited while L2 is active. This is a regression test for
an AVIC bug where KVM would skip updating x2APIC MSR intercepts while L2
is active, thus giving L1 access to a wide swath of L0's x2APIC surface.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260710162052.2188574-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Force a refresh of the vmcs02 MSR bitmap during nested VM-Enter if the
runtime eVMCS controls (pin, primary, secondary, etc.) are being updated.
If L1 isn't intercepting TPR writes, runs L2 with TPR virtualization, and
then runs the same L2 with TPR virtualization disabled, KVM will fail to
refresh msr_bitmap02 and leave TPR in passthrough mode even though TPR
virtualization is disabled. I.e. failure to refresh the bitmap lets L2 (or
L1 by proxy) read and write L0's TPR.
Fixes: 502d2bf5f2 ("KVM: nVMX: Implement Enlightened MSR Bitmap feature")
Cc: stable@vger.kernel.org
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Use the MSR permission bitmap of the active VMCB instead of assuming that
KVM is always using vmcb02's bitmap when L2 is active, as KVM uses msrpm02
if and only if L1 wants to intercept MSR accesses, i.e. if and only if KVM
needs to merge msprm01 with msrpm12.
Don't bother tracking the virtual address of the bitmap that's being used,
as __va() is cheap on x86, and caching the virtual address would introduce
yet another source of potentially stale information.
Fixes: b2ac58f905 ("KVM/SVM: Allow direct access to MSR_IA32_SPEC_CTRL")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260826195833.844526-1-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Don't (re)read PERF_CNTR_GLOBAL_CTL from hardware on a failed VMRUN, as the
purpose of the read is to synchronize KVM's cache with any writes done by
the guest, and the guest can't possibly have modified the MSR if it never
got a chance to run.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260904170642.3291466-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Don't mark the ASID as dirty in the VMCB when requesting a TLB flush via
control.tlb_ctl. Per "15.15.3 VMCB Clean Field" of the July 2026, Revision
3.45 version of the APM:
The following are explicitly not cached and not represented by Clean bits:
* TLB_Control
Fixes: 7e8e6eed75 ("KVM: SVM: Move asid to vcpu_svm")
Suggested-by: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260904170642.3291466-4-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Leave control.erap_ctl and control.clean as-is in the VMCS if VMRUN fails,
because as per AMD:
there's no explicit architectural guarantee about the behavior in the
presence of VMRUN failures. So the best thing to do would be to assume
that if VMRUN fails, the actions requested in the control fields may not
have been performed.
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260904170642.3291466-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Don't reset the VMCB's TLB control back to "do nothing" on a failed VMRUN,
as empirical testing shows that the CPU performs the requested TLB flush if
and only if VMRUN is successful, i.e. clearing TLB control on a failed
VMRUN effectively drops a TLB flush.
Explicitly track the need to flush all ASIDs on a per-CPU basis, as the
ASID reuse condition is tied to the pCPU, not to the vCPU. As a bonus,
this also obviates the need to avoid clobbering FLUSH_ALL_ASID with
TLB_CONTROL_FLUSH_ASID, e.g. in svm_flush_tlb_asid().
Deliberately don't bother saving/restoring the "old" tlb_ctl on failure,
in quotes because it's not exactly the old tlb_ctl, it's the tlb_ctl from
after pre_svm_run(), but before updating tlb_ctl for flush_all_asids. If
VMRUN fails and TLB_CONTROL_FLUSH_ALL_ASID is forced, then the next
successful run of the VMCB *may* unnecessarily flush all ASIDs, which
strictly speaking could result in noisy neighbor issues. However, the
fact that new_asid() is already guest-triggerable, because of KVM's flawed
behavior of clearing the ASID on emulated INIT, means that a guest can
already trigger a flush of all ASIDs at roughly the same rate. And once
KVM stops clobbering the ASID on emulated INIT, *or* assigns a static ASID
to each vCPU, this flaw goes away.
Fixes: 38e5e92fe8 ("KVM: SVM: Implement Flush-By-Asid feature")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Suggested-by: Yosry Ahmed <yosry@kernel.org>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Jim Mattson <jmattson@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260904170642.3291466-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
When walking shadow page tables, immediately terminate the walk if the root
is a "dummy" root, i.e. a root whose top-level page table is backed by the
zero page, but otherwise doesn't exist. If memslot creation races with a
stage-2 page fault (EPT violation or #NPF) from L2, then if the stars align,
KVM will attempt to walk shadow page tables using the zero page and hit a
NULL pointer deref.
BUG: kernel NULL pointer dereference, address: 0000000000000021
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP
CPU: 30 UID: 1000 PID: 941 Comm: qemu Not tainted 7.2.0-rc2-1b731e5ded48-next-vm #1741 PREEMPTLAZY
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:__kvm_mmu_invalidate_addr+0xea/0x210 [kvm]
Call Trace:
<TASK>
kvm_mmu_invalidate_addr+0x92/0xe0 [kvm]
__kvm_inject_emulated_page_fault+0x67/0x80 [kvm]
ept_page_fault+0x160/0x850 [kvm]
kvm_mmu_do_page_fault+0x102/0x1f0 [kvm]
kvm_mmu_page_fault+0x8e/0x6b0 [kvm]
vmx_handle_exit+0x163/0x640 [kvm_intel]
kvm_arch_vcpu_ioctl_run+0x960/0x2120 [kvm]
kvm_vcpu_ioctl+0x2c7/0x970 [kvm]
__x64_sys_ioctl+0x90/0xd0
do_syscall_64+0x67/0x5f0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f606d0b53bb
Opportunistically harden the shadow walks against fully invalid roots, but
WARN, as all callers are expected/required to pre-check for a valid root.
Don't WARN in the dummy root case as the whole point of using a dummy root
is to provide a root that's valid enough to enter the guest, i.e. it should
Just Work for all flows except those that *need* to know about dummy roots.
Alternatively, KVM could allocate a dedicated page and associated shadow
page structure for the dummy root, which is very tempting as it such an
approach should be more resilient against unexpected behavior. But that
would be a much larger and thus riskier change than simply terminating
walks of dummy roots.
Fixes: 0e3223d8d0 ("KVM: x86/mmu: Use dummy root, backed by zero page, for !visible guest roots")
Reported-by: Gabriel Schneider <gbrls@osec.io>
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260925234256.2384816-1-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Now that KVM holds kvm->lock for the entirety of vCPU creation, from when
created_vcpus is incremented until the new vCPU is fully onlined, WARN if
the impossible happens and KVM somehow sees a discrepancy between the
number of vCPUs "created" and "onlined".
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-8-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Now that KVM uses kvm_get_vcpu_by_id() to check for an existing vCPU ID
before doing any meaningful work, which was made possible by holding
kvm->lock for the entirety of vCPU creation, revert the now-redundant
"early" vCPU ID tracking. The claims about the impact of kvm->vcpu_ids on
the memory footprint were a wee bit wrong: the worst case scenario isn't
256 bytes per VM, it's 256 "unsigned longs" per VM, i.e. 2048 bytes per VM.
Increasing the size of "struct kvm" by 2048 nearly doubled the total size
on many architectures, and tripped x86's KVM_SANITY_CHECK_VM_STRUCT_SIZE,
which was added to detect this *exact* scenario, where a single change
significantly increased the size of "struct kvm". I.e. attempting to build
KVM with CONFIG_DEBUG_KERNEL=n fails on x86 (the build failures got missed
because all build bots apparently test only CONFIG_DEBUG_KERNEL=y kernels,
and maintainers' test flows were similarly lacking).
This reverts commit 97d65b544f.
Fixes: 97d65b544f ("KVM: Check for duplicate vcpu_id as early as possible")
Reported-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Closes: https://lore.kernel.org/all/56a4bc35ee605588b7cc36c8e45c12b5f3b506cb.camel@guillain.net
Reported-by: Paweł S <spawel523@gmail.com>
Closes: https://lore.kernel.org/all/CABD%3DWFOS4j4hDv%2BpW-eEM9HAM2q2GY_iYdAG%2BqvYcUEinUrcQQ@mail.gmail.com
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-7-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Now that kvm->lock is held for the entirety of vCPU creation, check for a
conflicting vCPU ID at the begnning of vCPU creation, before the arch
precreate() hook is invoked. This will allow reverting commit 97d65b544f
("KVM: Check for duplicate vcpu_id as early as possible"). For now, keep
the redundant vcpu_ids tracking as a sanity check.
No functional change intended (absent KVM bugs, checking vcpu_ids and
walking kvm_get_vcpu_by_id() should yield the same result).
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-6-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
When creating a vCPU, don't drop kvm->lock to when doing the bulk of actual
vCPU creation, as allowing multiple vCPUs to be created in parallel adds
significant complexity in KVM (as evidenced by the many related bugs), and
all known VMMs fully serialize vCPU creation. Remove all manually locking
of kvm->lock from kvm_arch_vcpu_{post,}create() for obvious reasons.
For many years, "everyone" has assumed that dropping kvm->lock was done for
performance reasons optimization, e.g. to allow userspace to create all
vCPUs concurrently for latency purposes. But as above, no known VMM does
that. Looking at the history of this code, before commit 11ec280471
("KVM: Convert vm lock to a mutex"), kvm->lock was a spinlock. I.e. KVM
*had* to drop kvm->lock when doing the bulk of vCPU creation, otherwise KVM
couldn't do normal memory allocations. When kvm->lock got turned into a
mutex for unrelated reasons, no one took advantage updated of the change to
simplify vCPU creation. And 19 years later, everyone just assumed that KVM
continued to deal with the complexity for performance reasons.
Furthermore, naively parallelizing vCPU creation in userspace is likely a
net negative due to the overheads of task creation. Unless a VMM carefully
avoids the extra overhead related to parallelization, e.g. spawns each
vCPU's thread before creating the vCPU, creating vCPUs concurrently is a
net *negative* up until about ~64 vCPUs, after which the times are a wash.
The absolute speed of light _is_ faster if KVM doesn't hold kvm-lock, but
at vCPU counts of ~16 or less, it's probably in the noise when considering
total VM creation time, as the added latency is less than 1ms up until 16
or so vCPUs.
On top of all that, KVM has had a *lot* of fatal bugs (most often found by
syzkaller) related to vCPUs being created while trying to do per-VM
operations (basically, see every flow that locks all vCPUs). I.e. the
parallel vCPU creation "support" is actively harmful as the only "use case"
is for misbehaving userspace to exploit KVM bugs.
Serializing vCPU creation will allow reverting commit 97d65b544f ("KVM:
Check for duplicate vcpu_id as early as possible"), which had "minor" math
error: the worst case scenario isn't "256 bytes per VM", it's "256 unsigned
longs per VM", i.e. 2048 bytes per VM, which doubles the size of each VM
and pushes several architectures into order-1 allocations.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Use kvm_is_vcpu_creation_in_progress() instead of an open-coded equivalent
during AIA initialization. Unlike similar vGIC code in arm64, the relevant
RISC-V code runs under kvm->lock, i.e. can use the standard API without
hitting lockdep false positive.
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-4-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Now that KVM's APIs for locking all vCPUs return -EBUSY if vCPU creation is
in-progress, drop the manual check for the same from vGIC creation, and
update the comments accordingly.
Note, while KVM arm64 guards many vGIC operations with its arch-specific
config_lock, holding kvm->lock is sufficient to guarantee a stable result
for "is vCPU creation in-progress". So, no functional change intended.
Note #2, the open coded check in vgic_init() is racy when called without
kvm->lock held, e.g. via vgic_lazy_init(). I.e. that check needs to stay
open coded to avoid triggering a lockdep assert. Whether or not the race
is "fine" is a problem for a different day.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reject locking of all vCPUs if vCPU creation is in-progress, i.e. if the
number of "created" vCPUs doesn't match the number of "onlined" vCPUs.
It's simply not possible to guarantee that KVM has truly locked all vCPUs
if one or more vCPUs are actively being created. Holding kvm->lock does
prevent in-flight vCPUs from being fully onlined, but it's infeasible for
common KVM to know whether or not that provides sufficient protection.
In practice, this is likely a minor bug fix for the ARM and RISC-V usage of
kvm_trylock_all_vcpus(), and a glorified nop for everything else. E.g.
ARM's kvm_timer_vcpu_init() can race kvm_vm_ioctl_set_counter_offset() with
respect to observing KVM_ARCH_FLAG_VM_COUNTER_OFFSET.
Opportunistically drop x86's existing manual checks on vCPU creation being
in-progress as all of x86's checks immediately precede or follow locking of
all vCPUs. Leave arm64 and RISC-V alone for the moment, as their checks
aren't as obviously redundant/equivalent.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- Fix a brown paper bag bug where KVM would incorrectly treat Intel PMU MSRs
as valid on AMD.
- Fix a regression in the hardware disable selftest where it checked the wrong
macro when detecting glibc support (breaks at least musl).
- Never clear KVM_REQ_VM_DEAD so that dead VMs stay dead, which is especially
important for KVM_BUG_ON() flows, which often guard more dangerous bugs.
- Re-pend GET_NESTED_STATE_PAGES if getting the pages fails, to fix a bug
where KVM would let userspace run a broken setup with stale vmcs12 pages.
- Fix a class of bugs where KVM would fail to fill kvm_run exit fields if
getting nested pages failed.
- Treat reserved entries in the memory attributes xarray as "no attributes",
to fix false positives when checking for mixed attributes.
- Fix memcg accounting for the memory attributes xarray (the xarray library
subtly requires the xarray to be configured for accounting upfront; the gfp
flags taken at runtime are used only rarely).
- Don't pre-reserve xarray entries when storing empty attributes, as storing
NULL must not require memory allocation (KVM and other subsystems heavily
rely on this behavior).
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmq3A78ACgkQOlYIJqCj
N/1uVA//TVDznH49htPNV4dh7vfiE8oF9ka4e9KTPA2eMdDeNVInNXBz0ZbJCdNN
Hm7iZZfk8yx0JCCGXdp8EoYg66iwulBr0KZXRP2U2LoyiRELA1VtGGOLUVP3PXjF
hElXD56+RrkJJSucINlFU7kSfl2hY/Jbaq87HVBLpXp8iY3stBbLdgMT1ReLdorO
s9i2uJRqHKjtSG330qDYEG1TtRAdUehgRbxbbJVuOEYtU37tCvX6ai8dEjZAvxYo
1VZOD05ITo12HUXPiZjIMoNTujQnZKmGYlBYczWlfOmkY/tN6SV8M41srV3ZcK0k
XGOQM6tOzbVTWaSH/pW1krklnOFAbnL6QzHzb3enHZfUN8AikHPDyfsJWzt/s0f/
ATTQk03OYL4fJMQSQ778EI6S4PcOzY9HiBxDXM5ySpUv1uyXw8ZFsQSdH/TYTMhi
Dce42r5rlGa1vAKHwI1Xya2su3REhWzxYxGpUOg/PtRRqWvL2RMrHArzEd6zu+Ti
jPinuQuXYcT1FTCa5S39NWhU4gWpjw8Pvs9VuUYSCDYxZCaqQ2YheYgsAcgPqgfE
hS43XVmmOB/tXGsvu+/YuEfbzNV/mHOuJBcjBnnsBP3d6EFB02FPKg4/LYiXujYo
OocUbSB7bKd87W3B+Ys3QmgOtZh4qX/aLYusa2vQ4Dp9ujXqcWs=
=LG0B
-----END PGP SIGNATURE-----
Merge tag 'kvm-x86-fixes-7.3-rc5' of https://github.com/kvm-x86/linux into HEAD
KVM fixes for 7.3-rcN
- Fix a brown paper bag bug where KVM would incorrectly treat Intel PMU MSRs
as valid on AMD.
- Fix a regression in the hardware disable selftest where it checked the wrong
macro when detecting glibc support (breaks at least musl).
- Never clear KVM_REQ_VM_DEAD so that dead VMs stay dead, which is especially
important for KVM_BUG_ON() flows, which often guard more dangerous bugs.
- Re-pend GET_NESTED_STATE_PAGES if getting the pages fails, to fix a bug
where KVM would let userspace run a broken setup with stale vmcs12 pages.
- Fix a class of bugs where KVM would fail to fill kvm_run exit fields if
getting nested pages failed.
- Treat reserved entries in the memory attributes xarray as "no attributes",
to fix false positives when checking for mixed attributes.
- Fix memcg accounting for the memory attributes xarray (the xarray library
subtly requires the xarray to be configured for accounting upfront; the gfp
flags taken at runtime are used only rarely).
- Don't pre-reserve xarray entries when storing empty attributes, as storing
NULL must not require memory allocation (KVM and other subsystems heavily
rely on this behavior).
Manually perform cache maintenance on the source VM during intra-host
migration to ensure no stale data is left in CPU caches after the VM is
destroyed. Because the source VM is "converted" to a non-SEV VM, KVM's
memory reclaim flows won't trigger cache maintenance, e.g. when all guest
memory is reclaimed in response to detaching from the mmu_notifier.
Note, relying on the destination VM to do cache maintenance isn't an option
as KVM doesn't require identical guest memory configurations, i.e. the
source VM may have access to memory that the destination VM does not.
Enforcing equivalent memory configurations is infeasible, as it would
require a *deep* comparison of memslots, e.g. to verify that not only are
the memslot identical, but what the memslots point at is also identical.
Fixes: b56639318b ("KVM: SEV: Add support for SEV intra host migration")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260923163721.1584779-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Unconditionally free SEV's "have run CPUs" cpumask in the VM destroy path,
i.e. even for what appear to be non-SEV VMs, as an SEV VM becomes a non-SEV
VM if its state is intra-host migrated. Alternatively, the mask could be
freed in sev_migrate_from() when "converting" the source VM, but that gets
annoying because ideally KVM would nullify the mask to guard against UAF,
and nullifying the mask would need be conditioned on CPUMASK_OFFSTACK=y.
Freeing the mask during sev_migrate_from() is also not robust against other
KVM bugs, though that's kind of a moot point since any such bugs would show
up even if sev->active is never set. I.e. KVM must get that side of things
correct. But, that's not a great reason to add more code just to make
things marginally less robust.
Fixes: 6f38f8c574 ("KVM: SVM: Flush cache only on CPUs running SEV guest")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260923163721.1584779-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Skip the xarray reservation loop when clearing all memory attributes, as
storing NULL only erases the entry and never needs to allocate, so no
reservation (and no cleanup of a failed one) is required in that case.
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: David Ballesteros <davimaba.v@proton.me>
Signed-off-by: Zeng Chi <zengchi@kylinos.cn>
Link: https://patch.msgid.link/20260921102442.1232375-1-zeng_chi911@163.com
[sean: split to separate patch]
Signed-off-by: Sean Christopherson <seanjc@google.com>
Explicitly instantiate the memory attributes xarray with XA_FLAGS_ACCOUNT
to ensure that all allocations are accounted to the memcg. Frustratingly,
memory allocations done in the "fastpath" do not honor the passed in gfp,
even for an explicit xa_reserve(). Only the rare, slow path __xas_nomem()
honors the original gfp. E.g.
xa_reserve(..., GFP_KERNEL_ACCOUNT)
|
-> ...
|
-> __xa_cmpxchg_raw()
|
-> xas_store() <== does not take @gfp
|
-> xas_create()
|
-> xas_alloc()
The bug was confirmed by observing that a process in a cgroup limited to
256 MiB grew radix_tree_node slab by ~512 MiB while its memory.current
stayed near 0.
Fixes: 5a475554db ("KVM: Introduce per-page memory attributes")
Cc: stable@vger.kernel.org
Assisted-by: Claude-Code:claude-opus-5
Signed-off-by: David Ballesteros <davimaba.v@proton.me>
Link: https://patch.msgid.link/20260915175335.138547-4-davimaba.v@proton.me
[sean: rewrite changelog, tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
kvm_vm_set_mem_attributes() reserves an xarray entry for every gfn in
the range before storing the new attributes, so that the store loop
can't fail partway through. If one of the reservations fails, e.g. with
-ENOMEM, the entries that were already reserved are left in the array.
That is harmless as far as xa_reserve() is concerned, as the reserved
entries read back as NULL via xa_load(), but it confuses the "does this
range have no attributes at all" check:
if (!attrs)
return !xas_find(&xas, end - 1);
A reserved entry is XA_ZERO_ENTRY, not NULL, and xas_find() returns it
as present. So a leftover reservation makes KVM report that a fully
shared range has attributes even though kvm_get_memory_attributes()
returns none for every gfn in the range. On x86, the next time
mixed-attribute tracking is recomputed for the range (memslot creation,
or a later attribute change that straddles the 2MiB page),
hugepage_has_attrs() treats a fully shared 2MiB range as mixed and
refuses to map it with a hugepage, until userspace happens to set
attributes on the range again.
Drop the shortcut and handle the !attrs case in the per-index loop,
using xas_next_entry() to find the next non-NULL entry. xas_next_entry()
is essentially an optimized xas_find(), so the effective change is that
the !attrs lookup now goes through xas_retry() like the attrs != 0 case,
i.e. reserved entries are skipped and retry entries restart the walk.
Don't check the index when no entry is found, as the xarray leaves the
xas index in a bogus state in that case; no entry simply means the rest
of the range has no attributes.
KVM never stores a non-NULL entry with a value of zero (clearing stores
NULL), but such an entry would be returned by xas_next_entry() and trip
the index check, so WARN if one is ever seen.
Fixes: 5a475554db ("KVM: Introduce per-page memory attributes")
Cc: stable@vger.kernel.org
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: David Ballesteros <davimaba.v@proton.me>
Signed-off-by: Zeng Chi <zengchi@kylinos.cn>
Link: https://patch.msgid.link/20260921102442.1232375-1-zeng_chi911@163.com
[sean: expand comment to elaborate on xarray APIs, split optimization out]
Signed-off-by: Sean Christopherson <seanjc@google.com>
Fill kvm_run with "internal error, emulation" in the common error handling
paths for getting nested state pages, as requiring each check to manually
fill kvm_run is error prone and requires a non-trivial amount of copy+paste.
Specifically, both SVM and VMX fail to fill kvm_run if load_pdptrs() fails,
and SVM fails to fill kvm_run if kvm_hv_verify_vp_assist() fails. If those
flows fail, the *best* case scenario is that KVM will exit to userspace
with KVM_EXIT_UNKNOWN. The worst case scenario is that KVM exits with a
stale exit_reason and confuses userspace.
Note, SVM never exits to userspace if something goes sideways when dealing
with vmcb12 assets while emulating VMRUN, i.e. lack of SVM-specific code is
not a bug.
Fixes: 0f85722341 ("KVM: nVMX: delay loading of PDPTRs to KVM_REQ_GET_NESTED_STATE_PAGES")
Fixes: 232f75d3b4 ("KVM: nSVM: call nested_svm_load_cr3 on nested state load")
Fixes: 3f4a812edf ("KVM: nSVM: hyper-v: Enable L2 TLB flush")
Cc: stable@vger.kernel.org
Reported-by: Jinwoo Lee <rkskek9254@gmail.com>
Closes: https://lore.kernel.org/all/20260813043932.3214460-1-rkskek9254@gmail.com
Reported-by: Stefan Teodorescu <fane@google.com>
Link: https://patch.msgid.link/20260921211608.1030158-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Re-pend GET_NESTED_STATE_PAGES before exiting to userspace if getting the
nested pages fails in the KVM_RUN path. If userspace re-runs the vCPU, and
vmcs02 holds valid PFNs from the *previous* run of L2, then KVM could
re-enter L2 with stale, unpinned PFNs mapped into e.g. the vAPIC page.
Note, both SVM and VMX (as of commit 11722439fb ("KVM: nVMX: Ensure
KVM_REQ_GET_NESTED_STATE_PAGES is cleared on VM-Exit") ensure the request
is cleared on VM-Exit (including the "forced" case), i.e. there is no risk
of double-mapping due to emulated VMLAUNCH/VMRESUME/VMRUN *and* the request
trying to map the nested pages.
Fixes: 671ddc700f ("KVM: nVMX: Don't leak L1 MMIO regions to L2")
Cc: stable@vger.kernel.org
Reported-by: Jinwoo Lee <rkskek9254@gmail.com>
Closes: https://lore.kernel.org/all/20260813043932.3214460-1-rkskek9254@gmail.com
Reported-by: Stefan Teodorescu <fane@google.com>
Link: https://patch.msgid.link/20260921211608.1030158-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
- Synchronize hrtimer during VCPU teardown
- Fix the conversion between vsip and hvip values
- Serialize IMSIC attributes with vCPU migration
- Release unused page after MMU invalidation
- Propagate interrupted G-stage faults to KVM user-space
- Fix NACL hfence entry update order
- Fix sdata leak and stale snapshot_addr in snapshot_set_shmem
- Preserve firmware counter value across PMU counter stop/start
- Report PMU snapshot write failure to the guest
- Fix perf-backed counter accounting across PMU stop and read
- Fix HSM hart status error propagation
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEZdn75s5e6LHDQ+f/rUjsVaLHLAcFAmqvlhYACgkQrUjsVaLH
LAeXtw//W/aV2pVW3LD6Ym3l25/7uDDSCwAp8QyzNxp4v9FoSLL/1czP5onXEbHo
xroVQcPx+2ezw70CNDtPxMpnaZi1zE9GkCYy+ahtWjRbBOYHWBJdSIRbL0niw6h7
iwfItAa5q5K/tcbSPf0GgNPowCQvV2Sl1t2xBOU0YyJckr8MsQjrTz71klAAbV7d
tq4FO3RLDI0HO8HZ3ZomJSBLmZ6exMZJIumORWja59niWKj0qREwnCEs5ZgnROYz
5JqcBXqPX8z6U7e8D7qvpHTDDijvAyKqyg3are0xkREDlrk1XncAar99YAAgSFjF
sTP9HF741IAwO4tW4ZwC5PHkvEV1aSPobZvLQWqjAKL84PihUF5s37TjEN4Ety4P
lH/bFRNnBKrCjiUIE3oxktDF2EuEl3zcpNeoVDogQ3X7HZM+BH+xRxfMRuiIdeUh
wgx5Ws/byp2S/pecd5tAim4DFQNuJx5Jyz1R3s9KEjoOA+j2f5gsB75Xt6oYWo6/
4CAmq3JcsUgIiisKHYDOP52SNWl7iBcZE9AHMcqNWra+il9ciNDmuzWc4BrxCAQ3
HiSIQQPqwxGRy6+h6LKhuQBDimj9HoMt5tV0/021KdFYMnW963IOS6luwXgjX62C
CiOGixBXCgXBjK57TUkT0KZZFiE2I4YTqDyGyAgc/e7R23XVw1Q=
=riAw
-----END PGP SIGNATURE-----
Merge tag 'kvm-riscv-fixes-7.3-1' of https://github.com/kvm-riscv/linux into HEAD
KVM/riscv fixes for 7.3, take #1
- Synchronize hrtimer during VCPU teardown
- Fix the conversion between vsip and hvip values
- Serialize IMSIC attributes with vCPU migration
- Release unused page after MMU invalidation
- Propagate interrupted G-stage faults to KVM user-space
- Fix NACL hfence entry update order
- Fix sdata leak and stale snapshot_addr in snapshot_set_shmem
- Preserve firmware counter value across PMU counter stop/start
- Report PMU snapshot write failure to the guest
- Fix perf-backed counter accounting across PMU stop and read
- Fix HSM hart status error propagation
- Invalidate the ITS translation cache when the guest changes the
base address of the ITS tables (Fuad Tabba)
- Skip saving ITS devices with device IDs that are out-of-bounds
rather than failing the entire ITS save ioctl (Fuad Tabba)
- Close race between VM teardown and invalidations of nested MMUs
when handling MMU operations that are allowed to block
(Lorenzo Stoakes)
- Various fixes for the handling of the host's untrusted SVE
configuration in pKVM (Fuad Tabba)
- Make sure that empty SMCCC ranges based at 0 are rejected by the
kvm_smccc_set_filter() (Karl Mehltretter)
- Revoke the host mapping for pKVM's private stack pages, along
with a new sanity check that all mappings in the hyp's private
VA range have been correctly marked as hyp-owned (Fuad Tabba)
- Lifetime fixes for the array of shadow stage-2 MMUs, ensuring that
concurrent vCPU initialization cannot relocate in-use MMUs. Defer
the freeing of shadow stage-2 MMUs to the point that no other users
(e.g. MMU notifier) could reference them (Marc Zyngier)
- Drop useless WARN when rejecting an unsupported ioctl for pKVM
(Fuad Tabba)
- Fix the steal_time selftest to install correctly-sized mappings for
non-4K hosts (Sebastian Ott)
- Correct mapping of fine-grained trap for GCSPOPX instruction
(Mark Brown)
- Fix KVM_BUG_ON() due to missing handling of DBGBXVR<n> from 32-bit
guests (Karl Mehltretter)
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQSNXHjWXuzMZutrKNKivnWIJHzdFgUCaq7S2hccb2xpdmVyLnVw
dG9uQGxpbnV4LmRldgAKCRCivnWIJHzdFuyFAPwMTw52duiQXiJd2MLUe7qndEWW
s5vcAPDZIdDOWY4HewD/Xh7j5EwanNDM2DDXvdrgjnsTrwsJBW2ULIBKS6lvOA4=
=fF21
-----END PGP SIGNATURE-----
Merge tag 'kvmarm-fixes-7.3-1' of https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 changes for 7.3, take #2
- Invalidate the ITS translation cache when the guest changes the
base address of the ITS tables (Fuad Tabba)
- Skip saving ITS devices with device IDs that are out-of-bounds
rather than failing the entire ITS save ioctl (Fuad Tabba)
- Close race between VM teardown and invalidations of nested MMUs
when handling MMU operations that are allowed to block
(Lorenzo Stoakes)
- Various fixes for the handling of the host's untrusted SVE
configuration in pKVM (Fuad Tabba)
- Make sure that empty SMCCC ranges based at 0 are rejected by the
kvm_smccc_set_filter() (Karl Mehltretter)
- Revoke the host mapping for pKVM's private stack pages, along
with a new sanity check that all mappings in the hyp's private
VA range have been correctly marked as hyp-owned (Fuad Tabba)
- Lifetime fixes for the array of shadow stage-2 MMUs, ensuring that
concurrent vCPU initialization cannot relocate in-use MMUs. Defer
the freeing of shadow stage-2 MMUs to the point that no other users
(e.g. MMU notifier) could reference them (Marc Zyngier)
- Drop useless WARN when rejecting an unsupported ioctl for pKVM
(Fuad Tabba)
- Fix the steal_time selftest to install correctly-sized mappings for
non-4K hosts (Sebastian Ott)
- Correct mapping of fine-grained trap for GCSPOPX instruction
(Mark Brown)
- Fix KVM_BUG_ON() due to missing handling of DBGBXVR<n> from 32-bit
guests (Karl Mehltretter)
The consolidation of the breakpoint and watchpoint register accessors
switched DBGBXVR<n> from trap_bvr() to trap_dbg_wb_reg(). The latter
selects backing storage with demux_wb_reg(), which only handles Op2 values
4 through 7. Since DBGBXVR<n> uses Op2 1, an AArch32 guest access hits
KVM_BUG_ON() and marks the VM dead.
DBGBXVR<n> aliases DBGBVR<n>_EL1[63:32], and its AA32(HI) descriptor
already selects the upper half. Map Op2 1 to dbg_bvr[] alongside Op2 4,
restoring the pre-regression behavior.
Fixes: 3ce9f3357e ("KVM: arm64: Fold DBGxVR/DBGxCR accessors into common set")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-5
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260810005616.13227-1-kmehltretter@gmail.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
The encoding to trap mapping currently maps a FGT on OP_GCSPOPX to
HFGITR_EL2.nGCSEPP but as per DDI0601 2026-06 this FGT controls trapping
of GCSPUSHX and GCSPOPCX, and not the separate GCSPOPX instruction.
Update the mapping to reflect the architecture.
Fixes: 863ac38984 ("KVM: arm64: Add missing HFGITR_EL2 FGT entries to nested virt")
Reviewed-by: Leonardo Bras <leo.bras@arm.com>
Signed-off-by: Mark Brown <broonie@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://patch.msgid.link/20260901-arm64-gcs-v20-2-f31750bdfadb@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
Fix the following failure when running with 16K host page size:
==== Test Assertion Failure ====
lib/kvm_util.c:991: vm_adjust_num_guest_pages(vm->mode, npages) == npages
pid=873 tid=873 errno=0 - Success
1 0x0000000000405a27: vm_mem_add at kvm_util.c:991
2 0x000000000040241f: check_steal_time_uapi at steal_time.c:223 (discriminator 7)
3 (inlined by) main at steal_time.c:539 (discriminator 7)
4 0x00007fff8b57af3b: ?? ??:0
5 0x00007fff8b57b007: ?? ??:0
6 0x0000000000402b6f: _start at ??:?
Number of guest pages is not compatible with the host. Try npages=4
Fixes: fc240715fc ("KVM: selftests: arm64: Fix steal_time test after UAPI refactoring")
Reported-by: Zenghui Yu <zenghui.yu@linux.dev>
Link: https://lore.kernel.org/kvmarm/7575a845-a542-4b16-b512-aec3126f97f3@linux.dev/T/#u
Signed-off-by: Sebastian Ott <sebott@redhat.com>
Reviewed-by: Zenghui Yu (Huawei) <zenghui.yu@linux.dev>
Link: https://patch.msgid.link/20260914131013.60334-1-sebott@redhat.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
kvm_pkvm_ioctl_allowed() WARNs when kvm_get_cap_for_kvm_ioctl() doesn't
find the ioctl number in vm_ioctl_caps[], and kvm_arch_vm_ioctl() calls
it for every number the generic code doesn't handle, so
ioctl(vm_fd, 0xdeadbeef) from userspace taints a pKVM host and panics it
under panic_on_warn. The lookup is fed userspace input: return false,
and userspace gets the -EINVAL kvm_arch_vm_ioctl() returns for that
number on a host without pKVM.
Fixes: b12b3b04f6 ("KVM: arm64: Check whether a VM IOCTL is allowed in pKVM")
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Link: https://patch.msgid.link/20260914093838.1082637-1-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
We free the shadow S2 structures from kvm_arch_flush_shadow_all(), which
is a Bad Idea(tm). Freeing the page tables is fair game (this is what
this callback is for), but freeing the container that could still be
referenced by another part of the system is not great.
Instead, grow separate destructors that gets called when we tear the VM
down for good. From there, we can nuke both the individual MMUs as well
as the global array that points to them, safe in the knowledge that the
vcpus themselves have been destroyed already.
Fixes: 4f128f8e1a ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures")
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260911162203.1919330-3-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
The nested_mmus array holds the shadow page tables that are used when
a guest is running a nested context. These structures are allocated on
VCPU_INIT for whole guest, which implies that they may have to be
relocated as the array grows.
Should a VCPU_INIT occur whilst a vcpu is actively running an L2 and
that the allocation requires relocation, that vcpu will still be
running with a pointer to the previous structure, which will have been
freed.
Fix this by turning the array of structures to an array of pointers,
which is now allocated at VM creation, sized to the absolute maximum
that KVM can handle.
In turn, each VCPU_INIT contributes S2_MMU_PER_VCPU to the pool. No
reallocation is ever performed, and the life cycle of each object is
much clearer:
- the nested_mmus array is allocated in kvm_init_nested(), and freed
in kvm_arch_destroy_vm()
- s2_mmu structures are allocated in kvm_vcpu_init_nested(), and freed
on kvm_arch_flush_shadow_all()
Finally, the freeing of vcpu->arch.vncr_array is made consistent
rather than being done on some failure paths, but not others.
Fixes: 4f128f8e1a ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures")
Reported-by: Shen Yongchao <grayhat@foxmail.com>
Reported-by: Karl Mehltretter <kmehltretter@gmail.com>
Suggested-by: Karl Mehltretter <kmehltretter@gmail.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://lore.kernel.org/r/20260803224405.41468-1-kmehltretter@gmail.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260911162203.1919330-2-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
fix_host_ownership() transfers only what it walks, so a hyp mapping
outside the linear map is not manipulated by the walk.
Walk the quarter of the VA space holding the private range and the
vmemmap once the transfer is done, and fail init unless every valid
leaf is hyp-owned: in the vmemmap when the page is memory, and in the
host stage-2, where hyp text may instead be mapped without write
access. A leaf that is not memory has no vmemmap entry and is checked
against the host stage-2 alone. Hyp text is matched by physical
address, since the only executable mapping in the range is the
Spectre-v3a vectors, whose VA is a private allocation, and an
executable mapping of anything else must not be host-readable. The
vmemmap can be block-mapped, so the walker checks each page of a leaf.
Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-5-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
__io_map_base is the start of the private VA range only until the first
allocation from it, after which it is the allocation cursor. Keep it as
the start and move the cursor to __io_map_next, for the walk of the
range the next patch adds.
No functional change intended.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
On a non-hVHE host, fix_host_ownership_walker()'s test for PAGE_HYP_EXEC
never matches: KVM_PGTABLE_PROT_UX is cleared at map time and only PX is
reported on read-back. Hyp text is therefore donated rather than left
read-only in the host stage-2, and the instruction dump in
nvhe_hyp_panic_handler() reads a page the host has no access to.
Match the text by physical address instead, in a helper a later patch
reuses. A test on the permissions would leave any other executable
mapping host-readable too.
Fixes: 80cbfd7174 ("KVM: arm64: Honor UX/PX attributes for EL2 S1 mappings")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
fix_host_ownership() walks only the linear-map alias of each memblock
region, and the per-CPU hyp stack, mapped in the private VA range for
its guard page, has none.
Walk each stack's VA range with the same walker.
Fixes: 1a919b17ef ("KVM: arm64: Add guard pages for pKVM (protected nVHE) hypervisor stack")
Reported-by: Hiroyuki Katsura <hk590@cam.ac.uk>
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
