Commit graph

1,649 commits

Author SHA1 Message Date
Nikunj A Dadhania
25f744ffa0 KVM: SVM: Bump asid_generation on CPU online to avoid ASID collision after hotplug
If a vCPU stays scheduled out (or blocked) while the last pCPU it ran
on goes through a hotplug cycle (online->offline->online), and the vCPU
then resumes execution on the same pCPU, then it is possible for it to
run with an ASID that has now been assigned to a different vCPU,
resulting in stale TLB translations being used.

svm_enable_virtualization_cpu() resets asid_generation to 1 and sets
next_asid to max_asid + 1 on every CPU online event, including hotplug
cycles.  Because next_asid starts beyond the pool boundary, the first
call to new_asid() after an online event always wraps the pool,
incrementing asid_generation to 2 and assigning ASIDs starting from
min_asid.

Consider two vCPUs from different VMs, vCPU-A pinned to CPU-X holding
asid_generation=2 and ASID=N from before the hotplug event:

  1. CPU-X goes offline and back online: asid_generation resets to 1,
     next_asid = max_asid + 1.

  2. One or more vCPUs migrate to CPU-X and call new_asid(), wrapping
     the pool and consuming ASIDs starting from min_asid.  Eventually
     vCPU-B from a different VM is assigned asid_generation=2, ASID=N
     — the same ASID that vCPU-A held before the hotplug.

  3. vCPU-A enters pre_svm_run() on CPU-X: current_vmcb->cpu is
     unchanged so the migration branch is skipped.  Its saved
     asid_generation=2 matches sd->asid_generation=2, so the generation
     check silently passes and vCPU-A continues running with ASID=N —
     the same ASID just freshly assigned to vCPU-B.

Both vCPUs from different VMs now run on CPU-X with the same ASID,
causing them to share NPT TLB entries and producing stale translations.

The collision manifests as a KVM internal error (Suberror: 1, emulation
failure).  The NPT page fault reports a faulting GPA far outside the
VM's physical memory range — a sign of stale TLB translations being
used.  KVM falls back to instruction emulation, which fails on
FPU/XSave instructions (XRSTOR, STMXCSR) that the emulator does not
implement.

Fix this by incrementing asid_generation instead of resetting it to 1
in svm_enable_virtualization_cpu().  On module load, asid_generation
starts at 0 (memset) and the increment produces 1, identical to the
old behaviour.  On subsequent hotplug cycles the generation advances
beyond any value a vCPU previously observed on this CPU, so the
generation check in pre_svm_run() reliably forces new_asid() on every
vCPU after every hotplug cycle.

Fixes: 774c47f1d7 ("[PATCH] KVM: cpu hotplug support")
Reported-by: Chandrakanth Silveru <Chandrakanth.Silveru@amd.com>
Tested-by: Srikanth Aithal <Srikanth.Aithal@amd.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Nikunj A Dadhania <nikunj@amd.com>
Message-ID: <20260715063506.672432-1-nikunj@amd.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-07-16 08:55:03 +02:00
Atish Patra
6ee4140788 KVM: SEV: Do not allow intra-host migration/mirroring of SNP VMs
The intra-host migration/mirroring feature is not fully implemented for
SEV-SNP VMs. The proper migration requires additional SNP-specific
state such as guest_req_mutex, guest_req_buf, and guest_resp_buf to be
transferred or initialized on the destination.

The SNP VM mirroring requires vmsa features to be copied as well otherwise
ASID would be bound to SNP range while VM is detected as a SEV VM.

Reject SNP source VMs in migration/mirroring until proper SNP state
transfer is implemented.

Fixes: 1dfe571c12 ("KVM: SEV: Add initial SEV-SNP support")

Reported-by: Chris Mason <clm@meta.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Atish Patra <atishp@meta.com>
Link: https://patch.msgid.link/20260602-sev_snp_fixes-v3-1-24bfd3ae047c@meta.com
Cc: stable@vger.kernel.org
[sean: let lines poke past 80 chars, tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-07-10 08:55:40 -07:00
Uwe Kleine-König (The Capable Hub)
995832b2ce Replace <linux/mod_devicetable.h> by more specific <linux/device-id/*.h> (c files)
Replace the #include of <linux/mod_devicetable.h> by the more specific
<linux/device-id/*.h> where applicable. For most cases the include
can be dropped completely, only a few drivers need one or two headers
added.

Acked-by: Danilo Krummrich <dakr@kernel.org>
Acked-by: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://patch.msgid.link/1a3f2007c5c5dcf555c09a4035ce3ae8ef1b6c49.1782808461.git.u.kleine-koenig@baylibre.com
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
2026-07-03 07:38:17 +02:00
Linus Torvalds
c75597caad s390:
* Fix S390_USER_OPEREXEC so it can now be enabled regardless of other
   unrelated capabilities
 
 * Fix handling of the _PAGE_UNUSED pte bit that could lead to guest
   memory corruption in some scenarios
 
 * A bunch of misc gmap fixes (locking, behaviour under memory pressure)
 
 * Fix CMMA dirty tracking
 
 x86:
 
 * Tidy up some WARN_ON() and BUG_ON(), replacing them with WARN_ON_ONCE()
   or KVM_BUG_ON().  All of these have obviously never triggered, or
   somebody would have been annoyed earlier, but still.
 
 * Fix missing interrupt due to stale CR8 intercept
 
 * Add a statistic that can come in handy to debug leaks as well as the
   vulnerability to a class of recently-discovered issues.
 
 * Do not ask arch/x86/kernel to export default_cpu_present_to_apicid()
   just for KVM.
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCgAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmo7xaAUHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroN7bwf/Ua6PrRrgdzfUGiIruBDnFGtXnaKf
 UPGpgl35Kl4ntPlmH6wRG6N0jP9sloa17Qj/Y33O1Um3Mi/eDV0UESW8o8tfFE2J
 CWUPc3/adw2urhnkXsoleSFy21a89TgLd0p4tot832p+IHi3iNJ45fx7XxftBeGt
 fa/liscncEX5tGwie0ZVkIEE/ob3+eX4ZYsTLXMi6FxAlyGPYNrp4jwJeYH9AFSd
 ZYGqVbDjDq14vhnppKXM8DMfHTXN4OVYw2RJZ9Y7u2h7ku3sN0XakzVTAUz5gtoM
 h+iu8dn5yslODBZiNGlls8UYm2bweOBCumH1ZITe/eTNuEdUOJGILorKRw==
 =QzfV
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm fixes from Paolo Bonzini:
 "s390:

   - Fix S390_USER_OPEREXEC so it can now be enabled regardless of other
     unrelated capabilities

   - Fix handling of the _PAGE_UNUSED pte bit that could lead to guest
     memory corruption in some scenarios

   - A bunch of misc gmap fixes (locking, behaviour under memory
     pressure)

   - Fix CMMA dirty tracking

  x86:

   - Tidy up some WARN_ON() and BUG_ON(), replacing them with
     WARN_ON_ONCE() or KVM_BUG_ON(). All of these have obviously never
     triggered, or somebody would have been annoyed earlier, but still...

   - Fix missing interrupt due to stale CR8 intercept

   - Add a statistic that can come in handy to debug leaks as well as
     the vulnerability to a class of recently-discovered issues

   - Do not ask arch/x86/kernel to export
     default_cpu_present_to_apicid() just for KVM"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (22 commits)
  x86/apic: KVM: Use cpu_physical_id() to get APIC ID of running vCPU for AVIC
  KVM: x86/mmu: Expose number of shadow MMU shadow pages as a stat
  KVM: x86: Unconditionally recompute CR8 intercept on PPR update
  KVM: VMX: Grab vmcs12 on CR8 interception update iff vCPU is in guest mode
  KVM: x86: WARN (once) if RTC pending EOI tracking goes off the rails
  KVM: x86: WARN and fail kvm_set_irq() if a PIC or I/O APIC vector is invalid
  KVM: x86: Bug the VM, not the kernel, if the ISR count {under,over}flows
  KVM: x86/mmu: Bug the VM, not the host kernel, if KVM write-protects upper SPTEs
  KVM: x86: Replace BUG_ON() with WARN_ON_ONCE() on "bad" nested GPA translation
  KVM: Replace guest-triggerable BUG_ON() in ioeventfd datamatch with get_unaligned()
  KVM: s390: Return failure in case of failure in kvm_s390_set_cmma_bits()
  KVM: s390: selftests: Fix cmma selftest
  KVM: s390: Fix cmma dirty tracking
  KVM: s390: Fix locking in kvm_s390_set_mem_control()
  KVM: s390: Fix handle_{sske,pfmf} under memory pressure
  KVM: s390: Fix code typo in gmap_protect_asce_top_level()
  KVM: s390: Do not set special large pages dirty
  KVM: s390: Fix dat_peek_cmma() overflow
  s390/mm: Fix handling of _PAGE_UNUSED pte bit
  KVM: s390: Fix typo in UCONTROL documentation
  ...
2026-06-25 10:21:13 -07:00
Sean Christopherson
098e32cba3 x86/apic: KVM: Use cpu_physical_id() to get APIC ID of running vCPU for AVIC
Use cpu_physical_id() instead of default_cpu_present_to_apicid() when
getting the APIC ID of the pCPU on which a vCPU is running/loaded, as the
kernel has gone way off the rails if a vCPU is loaded on a pCPU that has
been physically removed from the system.  Even if the impossible were to
happen, the absolutely worst case scenario is that hardware will ring the
AIVC doorbell on the wrong pCPU, i.e. a severely broken system will
experience mild performance issues.

Kill off KVM's superfluous kvm_cpu_get_apicid() wrapper along with the
for-KVM export of default_cpu_present_to_apicid(), as they existed purely
for the wonky AVIC usage.

Cc: Kai Huang <kai.huang@intel.com>
Cc: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Acked-by: Naveen N Rao (AMD) <naveen@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Message-ID: <20260612185459.591892-1-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-24 07:52:24 -04:00
Sean Christopherson
39e9c35e44 KVM: x86: Replace BUG_ON() with WARN_ON_ONCE() on "bad" nested GPA translation
If KVM attempts to translate what it thinks is an L2 GPA with a non-nested
MMU, simply WARN and return the GPA, i.e. trust the MMU more than the
caller, as there is zero reason to potentially panic the host kernel just
because KVM misused an API.

Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Message-ID: <20260618185746.2023283-1-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-24 11:25:21 +02:00
Linus Torvalds
c98d767b34 Arm:
"This is a bit of an odd merge window on the KVM/arm64 front. There is
 absolutely no new feature in the pull request. It is purely fixes,
 because it is simply becoming too hard to review new stuff when so
 many AI-fuelled fixes hit the list".
 
 - Significant cleanup of the vgic-v5 PPI support which was merged in
   7.1. This makes the code more maintainable, and squashes a couple
   of bugs in the meantime.
 
 - Set of fixes for the handling of the MMU in an NV context,
   particularly VNCR-triggered faults. S1POE support is fixed
   as well.
 
 - Large set of pKVM fixes, mostly addressing recurring issues
   around hypervisor tracking of donated pages in obscure cases
   where the donation could fail and leave things in a bizarre
   state.
 
 - Fixes for the so-called "lazy vgic init", which resulted in
   sleeping operations in non-preemptible sections. This turned
   out to be far more invasive than initially expected...
 
 - Reduce the overhead of L1/L2 context switch by not touching
   the FP registers.
 
 - Fix the way non-implemented page sizes are dealt with when
   a guest insist on using them for S2 translation.
 
 - The usual set of low-impact fixes and cleanups all over the map.
 
 Loongarch:
 
 - On a request for lazy FPU load, load all FPU state that the VM
   supports instead of enabling only the part (FPU, LSX or LASX)
   that caused the FPU load request.
 
 - Some enhancements about interrupt injection.
 
 - Some bug fixes and other small changes.
 
 RISC-V:
 
 - Batch G-stage TLB flushes for GPA range based page table updates
 
 - Convert HGEI line management to fully per-HART
 
 - Fix missing CSR dirty marking when FWFT state updated via ONE_REG
 
 - Fix stale FWFT feature exposure to Guest/VM
 
 - Speed up dirty logging write faults using MMU rwlock and atomic
   PTE updates using cmpxchg() for permission-only changes
 
 - Use flexible array for APLIC IRQ state
 
 - Use kvm_slot_dirty_track_enabled() for logging enable check on
   a memslot
 
 - Avoid skipping valid pages in kvm_riscv_gstage_wp_range()
 
 - Avoid skipping valid pages in kvm_riscv_gstage_unmap_range()
 
 - Use endian-specific __lelong for NACL shared memory
 
 S390:
 
 - KVM_PRE_FAULT_MEMORY support
 
 - Support for 2G hugepages
 
 - Support for the ASTFLEIE 2 facility
 
 - Support for fast inject using kvm_arch_set_irq_inatomic
 
 - Fix potential leak of uninitialized bytes
 
 - A few more misc gmap fixes.
 
 x86:
 
 - Generic support for the more granular permissions allowed by EPT,
   namely "read" (which was previously usurping the U bit) and
   separate execution bits for kernel and userspace.
 
 - Do not assume that all page tables start with U=1/W=1/NX=0 at the
   root, as AMD GMET needs to have U=0 at the root.
 
 - Introduce common assembly macros for use within Intel and AMD
   vendor-specific vmentry code.  This touches the SPEC_CTRL handling,
   which is now entirely done in assembly for Intel (by reusing the
   AMD code that already existed), and register save/restore which
   uses some macro magic to compute the offsets in the struct.  Both
   of these are preparatory changes for upcoming APX support.
 
 - Clean up KVM's register tracking and storage, primarily to prepare for
   APX support, which expands the maximum number of GPRs from 16 to 32.
 
 - Keep a single copy of the PDPTRs rather than two, since architecturally
   there is just one.
 
 - Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor code
   gets a chance to handle things like reaping the PML buffer.
 
 - Update KVM's view of PV async enabling if and only if the MSR write fully
   succeeds.
 
 - Fix a variety of issues where the emulator doesn't honor guest-debug state,
   and clean up related code along the way.
 
 - Synthesize EPT Violation and #NPF "error code" bits when injecting faults
   into L1 that didn't originate in hardware (in which case the VMCS/VMCB
   doesn't hold relevant information).
 
 - Add support for virtualizing (well, emulating) AMD's flavor of CPL>0 CPUID
   faulting.
 
 - Clean up the GPR APIs so that KVM's use of "raw" is consistent, and fix a
   variety of minor bugs along the way.
 
 - Fix an OOB memory access due to not checking the VP ID when handling a
   Hyper-V PV TLB flush for L2.
 
 - Fix a bug in the mediated PMU's handling of fixed counters that allowed the
   guest to bypass the PMU event filter.
 
 - Allow userspace to return EAGAIN when handling SNP and TDX hypercalls, so
   the KVM can forward a "retry" status code to the guest, and reserve all
   unused error codes for future usage.
 
 - Overhaul the TDP MMU => S-EPT code to move as much S-EPT specific logic as
   possible into the TDX code, and to funnel (almost) all S-EPT updates into
   a single chokepoint.  The motivation is largely to prepare for upcoming
   Dynamic PAMT support, but the cleanups are nice to have on their own.
 
 - Plug a hole in shadow page table handling, where KVM fails to recursively
   zap nested EPT/NPT shadow page tables when the nested hypervisor tears
   down its own EPT/NPT page tables from the bottom up
 
 x86 (Intel):
 
 - Support for nested MBEC (Mode-Based Execute Control), see above in the
   generic section; also run with MBEC enabled even for non-nested mode.
 
 - Use the kernel's "enum pg_level" in the TDX APIs instead of the TDX-Module's
   level definitions (which are 0-based).
 
 - Rework the TDX memory APIs to not require/assume that guest memory is
   backed by "struct page" (in prepartion for guest_memfd hugepage support).
 
 - Fix a largely benign bug where KVM TDX would incorrectly state it could
   emulate several x2APIC MSRs.
 
 - Use the "safe" WRMSR API when proxying LBR MSR writes as the to-be-written
   value is guest controlled and completely unvalidated.
 
 x86 (AMD):
 
 - Support for nested GMET (Guest Mode Execution Trap), see above in the
   generic section; also run with GMET enabled even for non-nested mode.
 
 - Fixes and minor cleanups to GHCB handling, on top of the earlier work
   already merged into 7.1-rc.
 
 - Ensure KVM's copy of CR0 and CR3 are up-to-date prior to invoking
   fastpath handlers.
 
 - Add support for virtualizing gPAT (KVM previously just used L1's PAT when
   running L2).
 
 - Fix goofs where KVM mishandles side effects (e.g. single-step and PMC
   updates) when emulating VMRUN.
 
 - Fix a variety of bugs in AVIC's handling of x2APIC MSR interception, most
   notably where KVM didn't disable interception of IRR, ISR, and TMR regs.
 
 - Add support for virtualizing Host-Only/Guest-Only bits in the mediated PMU.
 
 - Don't advertise support for unusable VM types, and account for VM types
   that are disabled by firmware, e.g. to mitigate security vulnerabilities.
 
 - Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with bugs and
   unnecessarily complicated, and add comprehensive tests.
 
 - Clean up and deduplicate the SEV page pinning code.
 
 - Fix minor goofs related to writing back CPUID information after firmware
   rejects a CPUID page for an SNP vCPU.
 
 Generic:
 
 - Rename invalidate_begin() to invalidate_start() throughout KVM to follow
   the kernel's nomenclature, e.g. for mmu_notifiers.
 
 - Use guard() to cleanup up various KVM+VFIO flows.
 
 - Minor cleanups.
 
 guest_memfd:
 
 - Return -EEXIST instead of -EINVAL if userspace attempts to bind a gmem
   range to multiple memslots, and fix the test that was supposed to ensure
   KVM returns -EEXIST.
 
 - Treat memslot binding offsets and sizes as unsigned values to fix a bug
   where KVM interprets a large "offset + size" as a negative value and allows
   a nonsensical offset.
 
 - Use the inode number instead of the page offset for the NUMA interleaving
   index to fix a bug where the effective index would jump by two for
   consecutive pages (the caller also adds in the page offset).
 
 Selftests:
 
 - Randomize the dirty log test's delay when reaping the bitmap on the first
   pass, as always waiting only 1ms hid a KVM RISC-V bug as the test reaped the
   bitmap before KVM could build up enough state to hit the bug.
 
 - A pile of one-off fixes and cleanups.
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCgAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmoypI0UHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroNCSAf8CyuvicojnZGFsNcoY4DZ0whBj5l8
 IwVkHaptuyGLSwPAebC/DwtU2NBY4CQdbtLjGmxenf+fSQOtcz1YWKbdY4Nr0hJG
 euAb/wO47RqKp1C4WPi7lL2P4XHgmUgD5XZ1qDnRV9gnGddZi3QLLmKgAtNq9Z4m
 +pAdGD2/RyZeI0Dx61+5ufE/z8YBlH6GrzgBTxfvvRNKqNDFykBenxYHI7seEfHC
 BPz0LxjHDD8BMDUo86qaylZnaeQSmFTGO/eTgvCvabxG6WbM1s0w3K1owmUmP7o3
 p9o8274Xh53iK2JYD/qGOtAJXejIwbYKT5bnvUQX0O1EH2+ouvSFapLQ9Q==
 =sSqM
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm updates from Paolo Bonzini:
 "arm64:

     This is a bit of an odd merge window on the KVM/arm64 front. There
     is absolutely no new feature in the pull request. It is purely
     fixes, because it is simply becoming too hard to review new stuff
     when so many AI-fuelled fixes hit the list.

   - Significant cleanup of the vgic-v5 PPI support which was merged in
     7.1. This makes the code more maintainable, and squashes a couple
     of bugs in the meantime

   - Set of fixes for the handling of the MMU in an NV context,
     particularly VNCR-triggered faults. S1POE support is fixed as well

   - Large set of pKVM fixes, mostly addressing recurring issues around
     hypervisor tracking of donated pages in obscure cases where the
     donation could fail and leave things in a bizarre state

   - Fixes for the so-called "lazy vgic init", which resulted in
     sleeping operations in non-preemptible sections. This turned out to
     be far more invasive than initially expected..

   - Reduce the overhead of L1/L2 context switch by not touching the FP
     registers

   - Fix the way non-implemented page sizes are dealt with when a guest
     insist on using them for S2 translation

   - The usual set of low-impact fixes and cleanups all over the map

  Loongarch:

   - On a request for lazy FPU load, load all FPU state that the VM
     supports instead of enabling only the part (FPU, LSX or LASX) that
     caused the FPU load request

   - Some enhancements about interrupt injection

   - Some bug fixes and other small changes

  RISC-V:

   - Batch G-stage TLB flushes for GPA range based page table updates

   - Convert HGEI line management to fully per-HART

   - Fix missing CSR dirty marking when FWFT state updated via ONE_REG

   - Fix stale FWFT feature exposure to Guest/VM

   - Speed up dirty logging write faults using MMU rwlock and atomic PTE
     updates using cmpxchg() for permission-only changes

   - Use flexible array for APLIC IRQ state

   - Use kvm_slot_dirty_track_enabled() for logging enable check on a
     memslot

   - Avoid skipping valid pages in kvm_riscv_gstage_wp_range()

   - Avoid skipping valid pages in kvm_riscv_gstage_unmap_range()

   - Use endian-specific __lelong for NACL shared memory

  S390:

   - KVM_PRE_FAULT_MEMORY support

   - Support for 2G hugepages

   - Support for the ASTFLEIE 2 facility

   - Support for fast inject using kvm_arch_set_irq_inatomic

   - Fix potential leak of uninitialized bytes

   - A few more misc gmap fixes

  x86:

   - Generic support for the more granular permissions allowed by EPT,
     namely "read" (which was previously usurping the U bit) and
     separate execution bits for kernel and userspace

   - Do not assume that all page tables start with U=1/W=1/NX=0 at the
     root, as AMD GMET needs to have U=0 at the root

   - Introduce common assembly macros for use within Intel and AMD
     vendor-specific vmentry code. This touches the SPEC_CTRL handling,
     which is now entirely done in assembly for Intel (by reusing the
     AMD code that already existed), and register save/restore which
     uses some macro magic to compute the offsets in the struct. Both of
     these are preparatory changes for upcoming APX support

   - Clean up KVM's register tracking and storage, primarily to prepare
     for APX support, which expands the maximum number of GPRs from 16
     to 32

   - Keep a single copy of the PDPTRs rather than two, since
     architecturally there is just one

   - Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor
     code gets a chance to handle things like reaping the PML buffer

   - Update KVM's view of PV async enabling if and only if the MSR write
     fully succeeds

   - Fix a variety of issues where the emulator doesn't honor
     guest-debug state, and clean up related code along the way

   - Synthesize EPT Violation and #NPF "error code" bits when injecting
     faults into L1 that didn't originate in hardware (in which case the
     VMCS/VMCB doesn't hold relevant information)

   - Add support for virtualizing (well, emulating) AMD's flavor of
     CPL>0 CPUID faulting

   - Clean up the GPR APIs so that KVM's use of "raw" is consistent, and
     fix a variety of minor bugs along the way

   - Fix an OOB memory access due to not checking the VP ID when
     handling a Hyper-V PV TLB flush for L2

   - Fix a bug in the mediated PMU's handling of fixed counters that
     allowed the guest to bypass the PMU event filter

   - Allow userspace to return EAGAIN when handling SNP and TDX
     hypercalls, so the KVM can forward a "retry" status code to the
     guest, and reserve all unused error codes for future usage

   - Overhaul the TDP MMU => S-EPT code to move as much S-EPT specific
     logic as possible into the TDX code, and to funnel (almost) all
     S-EPT updates into a single chokepoint. The motivation is largely
     to prepare for upcoming Dynamic PAMT support, but the cleanups are
     nice to have on their own

   - Plug a hole in shadow page table handling, where KVM fails to
     recursively zap nested EPT/NPT shadow page tables when the nested
     hypervisor tears down its own EPT/NPT page tables from the bottom
     up

  x86 (Intel):

   - Support for nested MBEC (Mode-Based Execute Control), see above in
     the generic section; also run with MBEC enabled even for non-nested
     mode

   - Use the kernel's "enum pg_level" in the TDX APIs instead of the
     TDX-Module's level definitions (which are 0-based)

   - Rework the TDX memory APIs to not require/assume that guest memory
     is backed by "struct page" (in prepartion for guest_memfd hugepage
     support)

   - Fix a largely benign bug where KVM TDX would incorrectly state it
     could emulate several x2APIC MSRs

   - Use the "safe" WRMSR API when proxying LBR MSR writes as the
     to-be-written value is guest controlled and completely unvalidated

  x86 (AMD):

   - Support for nested GMET (Guest Mode Execution Trap), see above in
     the generic section; also run with GMET enabled even for non-nested
     mode

   - Fixes and minor cleanups to GHCB handling, on top of the earlier
     work already merged into 7.1-rc

   - Ensure KVM's copy of CR0 and CR3 are up-to-date prior to invoking
     fastpath handlers

   - Add support for virtualizing gPAT (KVM previously just used L1's
     PAT when running L2)

   - Fix goofs where KVM mishandles side effects (e.g. single-step and
     PMC updates) when emulating VMRUN

   - Fix a variety of bugs in AVIC's handling of x2APIC MSR
     interception, most notably where KVM didn't disable interception of
     IRR, ISR, and TMR regs

   - Add support for virtualizing Host-Only/Guest-Only bits in the
     mediated PMU

   - Don't advertise support for unusable VM types, and account for VM
     types that are disabled by firmware, e.g. to mitigate security
     vulnerabilities

   - Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with
     bugs and unnecessarily complicated, and add comprehensive tests

   - Clean up and deduplicate the SEV page pinning code

   - Fix minor goofs related to writing back CPUID information after
     firmware rejects a CPUID page for an SNP vCPU

  Generic:

   - Rename invalidate_begin() to invalidate_start() throughout KVM to
     follow the kernel's nomenclature, e.g. for mmu_notifiers

   - Use guard() to cleanup up various KVM+VFIO flows

   - Minor cleanups

  guest_memfd:

   - Return -EEXIST instead of -EINVAL if userspace attempts to bind a
     gmem range to multiple memslots, and fix the test that was supposed
     to ensure KVM returns -EEXIST

   - Treat memslot binding offsets and sizes as unsigned values to fix a
     bug where KVM interprets a large "offset + size" as a negative
     value and allows a nonsensical offset

   - Use the inode number instead of the page offset for the NUMA
     interleaving index to fix a bug where the effective index would
     jump by two for consecutive pages (the caller also adds in the page
     offset)

  Selftests:

   - Randomize the dirty log test's delay when reaping the bitmap on the
     first pass, as always waiting only 1ms hid a KVM RISC-V bug as the
     test reaped the bitmap before KVM could build up enough state to
     hit the bug

   - A pile of one-off fixes and cleanups"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (326 commits)
  KVM: x86/mmu: Ensure hugepage is in by slot before checking max mapping level
  KVM: x86: Fix shadow paging use-after-free due to unexpected role
  KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
  KVM: s390: Enable adapter_indicators_set to use mapped pages
  KVM: s390: Add map/unmap ioctl and clean mappings post-guest
  riscv: kvm: Use endian-specific __lelong for NACL shared memory
  KVM: selftests: access_tracking_perf_test: bump number of NUMA nodes to 32
  KVM: s390: vsie: Implement ASTFLEIE facility 2
  KVM: s390: vsie: Refactor handle_stfle
  s390/sclp: Detect ASTFLEIE 2 facility
  KVM: s390: Minor refactor of base/ext facility lists
  KVM: x86/mmu: move pdptrs out of the MMU
  KVM: x86: check that kvm_handle_invpcid is only invoked with shadow paging
  KVM: nSVM: invalidate cached PDPTRs across nested NPT transitions
  KVM: nVMX: remove unnecessary code in prepare_vmcs02_rare
  KVM: x86: remove nested_mmu from mmu_is_nested()
  KVM: arm64: vgic-its: Make ABI commit helpers return void
  KVM: s390: Initialize KVM_S390_GET_CMMA_BITS memory
  LoongArch: KVM: Add missing slots_lock for device register/unregister
  LoongArch: KVM: Validate irqchip index in irqfd routing
  ...
2026-06-19 08:56:49 -07:00
Linus Torvalds
ff5ccdb8d5 x86/cpu updates for v7.2:
- CPUID API updates (Ahmed S. Darwish):
 
    - Introduce a centralized CPUID parser
    - Introduce a centralized CPUID data model
    - Introduce <asm/cpuid/leaf_types.h>
    - Rename cpuid_leaf()/cpuid_subleaf() APIs
    - treewide: Explicitly include the x86 CPUID headers
 
  - Update to x86-cpuid-db v3.1 (Maciej Wieczor-Retman)
 
  - Continued removal of pre-i586 support and related
    simplifications (Ingo Molnar)
 
  - Add Intel CPU model number for rugged Panther Lake
    (Tony Luck)
 
  - Misc fixes, updates and cleanups by Arnd Bergmann, Chao Gao,
    Lukas Bulwahn, Sohil Mehta, Maciej Wieczor-Retman.
 
 Signed-off-by: Ingo Molnar <mingo@kernel.org>
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmou1LcRHG1pbmdvQGtl
 cm5lbC5vcmcACgkQEnMQ0APhK1h16Q/9GvxrQDX13qd7/XrjOKNYtoLIGh1GhvHM
 2ZdbYh80LX36bDOg7Mhoy0bvElwmzaz2x1Zrb5SAIqpQXYqjfuRCe7a1SPQxYtCK
 A6j1YeD/CJMm36jaQITYkuCYxwaw5LQ62u0ShpEvyzZzarEJt8c8COWJjbE57FN2
 qusr+6K1sBwpEUl5mLabbJWXqhpPRYCz+nl2GF9BCHe6z7Nw7Q8VZf4w1weONJOy
 lMpC2X08w5TE2f4OwvnRbZLelyROz6pr1c3osUoQSVtBkprO9TsYHgd9yBva1E2E
 KxdWm+uSSUXM7cFBzY3RLBzKpG6iLtqircoDFBVdxOlT8I8KFggzbYy4JPlDEHQS
 FqUwWL+2duoRciOncnZn9hydxlg5So4w7OscvTVNp94/Pb2pMdpOB+bsN05WTpXc
 VMQVxrfsmwT1rV1oppzRizTMnb6yGg8BQvZPE+wrMDdH+/wwhzok4GdZzKVCzUoD
 nDFhJL7t8ikRwwuq5RAl1EcT7qLFJx3ba77B5REf2Uos2liTExO/XzcmQnzSV1M1
 RXpajQ1ikEBtJPAD3Zy0ASm/QPjB27o+6Kj6hW3NLlPoi5l0Lu6xQzH3wNrpQpst
 02NCWEn2X00YUQt2ukNcDFJiMHJzXAnBQDBSrWwwd97wol3sKnT9RR2NZVFzn7ls
 GOF8yJ0Wuec=
 =tpai
 -----END PGP SIGNATURE-----

Merge tag 'x86-cpu-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip

Pull x86 cpuid updates from Ingo Molnar:

 - CPUID API updates (Ahmed S. Darwish):
    - Introduce a centralized CPUID parser
    - Introduce a centralized CPUID data model
    - Introduce <asm/cpuid/leaf_types.h>
    - Rename cpuid_leaf()/cpuid_subleaf() APIs
    - treewide: Explicitly include the x86 CPUID headers

 - Update to x86-cpuid-db v3.1 (Maciej Wieczor-Retman)

 - Continued removal of pre-i586 support and related simplifications
   (Ingo Molnar)

 - Add Intel CPU model number for rugged Panther Lake (Tony Luck)

 - Misc fixes, updates and cleanups by Arnd Bergmann, Chao Gao, Lukas
   Bulwahn, Sohil Mehta, Maciej Wieczor-Retman.

* tag 'x86-cpu-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (25 commits)
  x86/cpu: Make CONFIG_X86_CX8 unconditional
  x86/cpu: Remove unused !CONFIG_X86_TSC code
  x86/cpuid: Update bitfields to x86-cpuid-db v3.1
  tools/x86/kcpuid: Update bitfields to x86-cpuid-db v3.1
  x86/cpu: Make CONFIG_X86_TSC unconditional
  MAINTAINERS: Drop obsolete FPU EMULATOR section
  x86/cpu: Fix a F00F bug warning and clean up surrounding code
  x86/cpu: Add Intel CPU model number for rugged Panther Lake
  x86/cpuid: Introduce a centralized CPUID parser
  x86/cpu: Introduce a centralized CPUID data model
  x86/cpuid: Introduce <asm/cpuid/leaf_types.h>
  x86/cpuid: Rename cpuid_leaf()/cpuid_subleaf() APIs
  x86/cpu: Do not include the CPUID API header in asm/processor.h
  Documentation: core-api/cpu_hotplug: Remove stale cpu0_hotplug docs
  x86/cpu, cpufreq: Remove AMD ELAN support
  x86/fpu: Remove the math-emu/ FPU emulation library
  x86/fpu: Remove the 'no387' boot option
  x86/fpu: Remove MATH_EMULATION and related glue code
  treewide: Explicitly include the x86 CPUID headers
  x86/cpu: Remove the CONFIG_X86_INVD_BUG quirk
  ...
2026-06-15 15:25:17 +05:30
Paolo Bonzini
4e6df93968 Merge branch 'kvm-single-pdptrs' into HEAD
The non-MMU changes/preliminary cleanups from the "split kvm_mmu in
three" series[1].  The final outcome is to have a single copy of the
PDPTRs (in vcpu->arch) instead of two (in root_mmu and nested_mmu).

[1] https://lore.kernel.org/kvm/20260603105814.10236-1-pbonzini@redhat.com/T/#t
2026-06-12 10:47:24 +02:00
Paolo Bonzini
8b9ef32200 KVM: x86/mmu: move pdptrs out of the MMU
PDPTRs are part of the CPU state.  A bit unconventionally, they are
reached via vcpu->arch.walk_mmu instead of being stored in vcpu->arch
directly.  That is nice in principle---it would allow TDP shadow paging
to have its own PDPTRs---but it is not necessary, because EPT has no
PDPTRs and NPT does not cache them.

Since kvm_pdptr_read does not otherwise need the MMU, drop the pdptrs
from the MMU altogether.  There is however something to be careful
about, in that PDPTRs are now not stored separately in root_mmu and
nested_mmu for L1 and L2 guests.  In practice this was already not
an issue:

- for EPT the VMCS0x has to keep them up to date; and for the purpose
  of emulation they are always loaded from the VMCS on vmentry/vmexit,
  thanks to the clearing of dirty and available register bitmaps in
  vmx_switch_vmcs()

- for NPT, VCPU_EXREG_PDPTR is similarly cleared for nNPT, which does
  not cache the PDPTRs; while for non-nNPT the PDPTRs are loaded
  together with the load of CR3.

Note that page table PDPTRs are not affected, since they are stored
in pae_root.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260530165545.25599-6-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-12 10:43:52 +02:00
Paolo Bonzini
62bad2b2cc KVM: nSVM: invalidate cached PDPTRs across nested NPT transitions
When L2 runs under nested NPT and uses PAE paging, KVM's cached PDPTRs
in mmu->pdptrs[] can hold stale or wrong values after nested
transitions and across migration restore, because both
nested_svm_load_cr3() and svm_get_nested_state_pages() only refresh
PDPTRs on the !nested_npt path.

The user-visible bug is on migration restore of an L2 running with nested
NPT and 32-bit PAE paging, if userspace uses KVM_SET_SREGS rather than
KVM_SET_SREGS2.  In that case, load_pdptrs() leaves VCPU_EXREG_PDPTR
marked as available, and kvm_pdptr_read() will use a stale translation
that used L1 GPAs instead of L2 nGPAs.  svm_get_nested_state_pages()
runs on first KVM_RUN but skips the refresh because nested_npt_enabled()
is true.  The CPU itself reads L2's PDPTRs correctly from memory via
L1's NPT, but KVM-side walking of guest PAE page tables uses the bogus
cached values.

Unlike Intel's GUEST_PDPTR0..3 fields in the VMCS, SVM has no
VMCB-cached PDPTR state: the in-memory PDPTEs at the current CR3 are
the only source of truth, and svm_cache_reg(VCPU_EXREG_PDPTR) simply
reloads them from memory via load_pdptrs().  Clearing the avail
bit (and the dirty bit because !avail/dirty is invalid) to force
a reload when PDPTRs as needed fixes the bug.

Do the same for nested_svm_load_cr3()'s nested_npt branch, so that
the invariant "PDPTRs need reloading" is handled similarly for both
immediate and deferred loading.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260530165545.25599-4-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-12 10:43:52 +02:00
Paolo Bonzini
743204d772 KVM SVN changes for 7.2
- Add support for virtualizing gPAT (KVM previously just used L1's PAT when
    running L2).
 
  - Fix goofs where KVM mishandles side effects (e.g. single-step and PMC
    updates) when emulating VMRUN.
 
  - Fix a variety of bugs in AVIC's handling of x2APIC MSR interception, most
    notably where KVM didn't disable interception of IRR, ISR, and TMR regs.
 
  - Add support for virtualizing Host-Only/Guest-Only bits in the mediated PMU.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmorQ78ACgkQOlYIJqCj
 N/3bCxAAoXU8uJ37PIm58P7ok5hF2OFXeDLCYE1BpOfYOaT4b0vWW+Rb5I3y+2K6
 511IxlBE2RcGBg6LHb1d1a/IEvs0NpzXOZFp7NseJC8sIn01Lndgn2SvoLoMVRy8
 Yda4WyCNV1rSeq+duZKtMRhXiNY5Zdgsz6S5MdnslAK9OCbjS2XsQ6F7afMwM1g0
 D3/kVUdBwUrKrllgmJ+XprLvYGEVji8xVCosdJm1ZAxb0xIr395/Oy7tnx3O2XNz
 NjpEqabFRjcL8JMiOPnYdTfdqsNagr7qSqunJkCMUtuq+ZBaAA332ZKRbR/Rh83c
 XprJMMw3+t77AjMSOHGmQY9ykVugT29bG7mjN/5bqhqqSzS1zjpGKd7ToEqvpKBn
 lCnbUoSUQkvfBMZr6WUQppZvNfZaNi2Nuq4EKkZzqOiKUMO6ggfIh/1+M7s93mmT
 rCa/35JEoBL3uwJcc75c25Y3QZWQyJjuNmFOuDt4s+yzDJ70F6II8FMykC5LGlFn
 m9oZ/KIFEY1H9r8GyjoCIksQI/GBPxAKULA8wfY4WsMr+NzzKIhB39FfR6FnyZTN
 XmPRlAKvWCGAqnrbOwgWvdu8Gsj18SpU38+AhngZRS7cVqB4+tQEZ4tcp9oSz6dy
 UMYsZD0N8TFgGNqRf34DmgB+NSIamttAAYhMPWgAoqdvseHSvp0=
 =4LNF
 -----END PGP SIGNATURE-----

Merge tag 'kvm-x86-svm-7.2' of https://github.com/kvm-x86/linux into HEAD

KVM SVM changes for 7.2

 - Add support for virtualizing gPAT (KVM previously just used L1's PAT when
   running L2).

 - Fix goofs where KVM mishandles side effects (e.g. single-step and PMC
   updates) when emulating VMRUN.

 - Fix a variety of bugs in AVIC's handling of x2APIC MSR interception, most
   notably where KVM didn't disable interception of IRR, ISR, and TMR regs.

 - Add support for virtualizing Host-Only/Guest-Only bits in the mediated PMU.
2026-06-12 10:16:59 +02:00
Paolo Bonzini
b39c6bd6fd KVM SEV changes for 7.2
- Don't advertise support for unusuable VM types, and account for VM types
    that are disabled by firmware, e.g. to mitigate security vulnerabilities.
 
  - Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with bugs and
    unnecessarily complicated, and add comprehensive tests.
 
  - Clean up and deduplicate the SEV page pinning code.
 
  - Fix minor goofs related to writing back CPUID information after firmware
    rejects a CPUID page for an SNP vCPU.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmorQawACgkQOlYIJqCj
 N/1e8RAAhQWGpKAs6kcyd+WhJCPm5bQPtDMRVNM8piCdXdZ1jy/qO2phuxWWXtxu
 7a5Kvmz5WODuUBJxdyzMLJJv+o8+8j/ReNA5XkdDBZrnvxZeQpm6yfjq7ZTfmgBx
 qc6gFETNjhu8gvb2tIqAfOe87TtBb2Ruw3fMiJR8+ATKIN4bgLrqOcnczBxHo80X
 RQZqM3CNjac+6q5Gc4YEK2RbKvdzl/QNEuMrpk25O9fPZ5rnFO6K6ojZQQ2tTdCB
 G6tnwg5zlPGSi41l4fjvbWfaj09UQ1/zupXaXfq6aoj4og773G8GmXGEvh9Kd6Dc
 GurSJMhjnOPY9kJJ1YLHcxX4HtSc0M9WLqV/uZB8PtmbKfotUMJsHz5bv7rJa98N
 XQnt+9U08JhURJaKF12p+s0bmO/K7Dmp7VrUhCP1jeNuHsSdaoXlrfC/qZp1oLip
 t0jSxwZELkCSNsgbNNzpTmED1+xfj9DhqY+FvzZN9HPBedZb2ZHCuWFXJ4DMhn5p
 TiS2cL6eu8gG7vFXCoOYIHJbZoLNm823nqEneFbSJdy9ZSn55/UTYyFuG0I6VY1W
 rfv52Vz7AuhR4WGDAAqz6fPX5otJsHKkS6Kqy9dqTcV2jDsLvu97bef3qgGj/pe+
 5q+J7eeOGTtyJeQZwxljJ7Z+SOSB8vEkKPX6d6bqM7YXWKzn6oo=
 =SlF7
 -----END PGP SIGNATURE-----

Merge tag 'kvm-x86-sev-7.2' of https://github.com/kvm-x86/linux into HEAD

KVM SEV changes for 7.2

 - Don't advertise support for unusuable VM types, and account for VM types
   that are disabled by firmware, e.g. to mitigate security vulnerabilities.

 - Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with bugs and
   unnecessarily complicated, and add comprehensive tests.

 - Clean up and deduplicate the SEV page pinning code.

 - Fix minor goofs related to writing back CPUID information after firmware
   rejects a CPUID page for an SNP vCPU.
2026-06-12 10:13:03 +02:00
Paolo Bonzini
f6d6be78b2 KVM misc x86 changes for 7.2
- Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor code
    gets a chance to handle things like reaping the PML buffer.
 
  - Ensure KVM's copy of CR0 and CR3 are up-to-date on SVM prior to invoking
    fastpath handlers.
 
  - Update KVM's view of PV async enabling if and only if the MSR write fully
    succeeds.
 
  - Fix a variety of issues where the emulator doesn't honor guest-debug state,
    and clean up related code along the way.
 
  - Synthesize EPT Violation and #NPF "error code" bits when injecting faults
    into L1 that didn't originate in hardware (in which case the VMCS/VMCB
    doesn't hold relevant information).
 
  - Add support for virtualizing (well, emulating) AMD's flavor of CPL>0 CPUID
    faulting.
 
  - Clean up the GPR APIs so that KVM's use of "raw" is consistent, and fix a
    variety of minor bugs along the way.
 
  - Fix an OOB memory access due to not checking the VP ID when handling a
    Hyper-V PV TLB flush for L2.
 
  - Fix a bug in the mediated PMU's handling of fixed counters that allowed the
    guest to bypass the PMU event filter.
 
  - Allow userspace to return EAGAIN when handling SNP and TDX hypercalls, so
    the KVM can forward a "retry" status code to the guest, and reserve all
    unused error codes for future usage.
 
  - Misc fixes and cleanups.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmorPacACgkQOlYIJqCj
 N/2f4BAAiWMEfbazgtGLYQyiSd2QtIw41Dpcz9cnSBxvuFztulewTpd5z2B4/8kX
 dSeZ3vyofgOVUVIj4AlD6Ap8ZO7XL3FizLcjWv9c8l4j/sTREL1ix0oejPJpBASg
 5EmSueBjmj+8VbBNxHGCYMuU4okR0UBmU86sC0wBuho+CvqkNegSrmCpntHUh4Kn
 34/HZ6QTik7iZNXV7KVT7/2De/bZV/9RJ7T8EAOkOGRfWft7cVUmkCXBP184pqee
 Z9X1gcymXb17Z73GbOFbM3SEewcYLiy1LdDyk1Tid3Wk6nJlMj0Tc0djsI6Vz7+8
 UvaKNoAXyWydPzJmYuCuTP+OnrZrha3RuEZvpP9UBJqGQgNjzs1jis92jiPL3yf4
 38zcNesukLU8RWKMHw4PYMQtDIVwqPloSPN9tPZ6U+9IWZx88H/IRMd080Jxsbem
 UNCuB88PawtlYV7gNk2XvQ0gcLib3LTjVjrWyTloUDWTjsMEXYxQ7Ehn6Jk+dfya
 BMjivWY246SqsMBmkj4Lj0S6uD25z9Yv3uPcwshZ3nuXCjdmBKlWNSs7cFYopI1t
 h4urP9xX/eURs63f++nB+6ZsqMR2ESQXmbybvsevR/+GFb0Iuy3obJk4ab2kKvP/
 OhFpo8AmJ3jOK2RrIbZ/wUb5rfZbvnCxjAqSQ1YZHIMkrD9vlTY=
 =IHTg
 -----END PGP SIGNATURE-----

Merge tag 'kvm-x86-misc-7.2' of https://github.com/kvm-x86/linux into HEAD

KVM misc x86 changes for 7.2

 - Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor code
   gets a chance to handle things like reaping the PML buffer.

 - Ensure KVM's copy of CR0 and CR3 are up-to-date on SVM prior to invoking
   fastpath handlers.

 - Update KVM's view of PV async enabling if and only if the MSR write fully
   succeeds.

 - Fix a variety of issues where the emulator doesn't honor guest-debug state,
   and clean up related code along the way.

 - Synthesize EPT Violation and #NPF "error code" bits when injecting faults
   into L1 that didn't originate in hardware (in which case the VMCS/VMCB
   doesn't hold relevant information).

 - Add support for virtualizing (well, emulating) AMD's flavor of CPL>0 CPUID
   faulting.

 - Clean up the GPR APIs so that KVM's use of "raw" is consistent, and fix a
   variety of minor bugs along the way.

 - Fix an OOB memory access due to not checking the VP ID when handling a
   Hyper-V PV TLB flush for L2.

 - Fix a bug in the mediated PMU's handling of fixed counters that allowed the
   guest to bypass the PMU event filter.

 - Allow userspace to return EAGAIN when handling SNP and TDX hypercalls, so
   the KVM can forward a "retry" status code to the guest, and reserve all
   unused error codes for future usage.

 - Misc fixes and cleanups.
2026-06-12 10:11:09 +02:00
Paolo Bonzini
9fb628b4cd Merge branch 'kvm-ghcb-for-7.2' into HEAD
Merge the final part of the GHCB 7.2 fixes at
https://lore.kernel.org/kvm/20260529183549.1104619-1-pbonzini@redhat.com/.

Patches 1-17 have already been included in Linux 7.1; these are minor
cleanups, and fixes for behaviors that are suboptimal or contradicting
the specification.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 17:00:06 +02:00
Sean Christopherson
c114187c5d KVM: SEV: Remove sometimes-used function-scoped "ret" from #VMGEXIT handler
Now that only two case-statements actually need a local "ret" variable,
refactor sev_handle_vmgexit() to have all flows return directly when
possible, and bury "ret" as "r" in the two paths that need to propagate a
return value from a helper.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-25-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-25-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:59:20 +02:00
Sean Christopherson
abb97ac118 KVM: SEV: Turn sev_es_validate_vmgexit() into a dedicated predicate
Now that sev_es_validate_vmgexit() is only responsible for checking that
all required GHCB fields are marked valid, turn it into a predicate whose
name reflects exactly that.

No functional change intended.

Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-24-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-24-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:59:19 +02:00
Sean Christopherson
b911484177 KVM: SEV: Handle unknown #VMGEXIT reasons in sev_handle_vmgexit()
Handle unknown #VMGEXIT reasons in sev_handle_vmgexit(), not in
sev_es_validate_vmgexit().  This makes it _much_ more obvious that KVM
simply funnels "legacy" exits to the standard SVM interception handlers,
and is the final preparatory change needed to reduce the scope of
sev_es_validate_vmgexit().

No functional change intended.

Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-23-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-23-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:59:19 +02:00
Sean Christopherson
5eb5d5d123 KVM: SEV: Return INVALID_INPUT, not MISSING_INPUT, for bad GUEST_REQUEST input(s)
Return INVALID_INPUT, not MISSING_INPUT, if the guest provides an unaligned
address for a GUEST_REQUEST, and/or attempts to use the same page for the
source and destination.  The inputs are obviously invalid, not missing.

Opportunistically move the checks out of sev_es_validate_vmgexit(), to
continue the march towards reducing the scope of the helper, and to help
guide future changes into correctly handling bad input.

Fixes: 88caf544c9 ("KVM: SEV: Provide support for SNP_GUEST_REQUEST NAE event")
Fixes: 74458e4859 ("KVM: SEV: Provide support for SNP_EXTENDED_GUEST_REQUEST NAE event")
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-22-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-22-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:59:19 +02:00
Sean Christopherson
95d8c59cdf KVM: SEV: Return INVALID_EVENT for SNP-only #VMGEXIT from non-SNP guest
Signal INVALID_EVENT, not MISSING_INPUT, if a non-SNP guest attempts to
invoke an SNP-only #VMGEXIT.  Opportunistically move the checks out of
sev_es_validate_vmgexit() to continue the march towards making said helper
a predicate whose sole purpose is to verify the guest has marked required
GHCB fields as valid.

Fixes: e366f92ea9 ("KVM: SEV: Support SEV-SNP AP Creation NAE event")
Fixes: 9b54e248d2 ("KVM: SEV: Add support to handle Page State Change VMGEXIT")
Fixes: 88caf544c9 ("KVM: SEV: Provide support for SNP_GUEST_REQUEST NAE event")
Fixes: 74458e4859 ("KVM: SEV: Provide support for SNP_EXTENDED_GUEST_REQUEST NAE event")
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-21-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-21-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:59:19 +02:00
Sean Christopherson
9a6e8d23dd KVM: SEV: Move GHCB "usage" check out of sev_es_validate_vmgexit()
Move the check to verify the guest's requested GHCB out of
sev_es_validate_vmgexit() as the first step towards making said helper a
predicate whose sole purpose is to verify the guest has marked required
GHCB fields as valid.

Using a single "validate" helper sounds good on paper, but in practice it's
difficult to verify that KVM is performing the necessary sanity checks (the
usage of state is far removed from the relevant checks), makes it difficult
to understand that "legacy" exits are simply routed to KVM's existing exit
handlers, and most importantly, has directly contributed to a number of
bugs as adding case-statements to the validation subtly removes them from
the default path that rejects unknown exit codes with INVALID_EVENT.

Deliberately extract the usage code check first so as to preserve the order
of KVM's checks, even though future code extraction will technically fix
bugs.

No functional change intended.

Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-20-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-20-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:59:19 +02:00
Sean Christopherson
1797e00bf8 KVM: SEV: Don't terminate SNP VMs on #VMGEXIT without a registered GHCB
If the guest attempts a non-MSR #VMGEXIT without the registered GHCB,
return a GHCB_HV_RESP_MALFORMED_INPUT+GHCB_ERR_NOT_REGISTERED error to the
guest instead of exiting KVM_RUN with -EINVAL (and in likelihood killing
the VM).  KVM has already mapped the requested GHCB, i.e. can cleanly
report an error, and so exiting with -EINVAL is completely unjustified.

Fixes: 0c76b1d082 ("KVM: SEV: Add support to handle GHCB GPA register VMGEXIT")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-19-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-19-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:59:19 +02:00
Sean Christopherson
db38bcb331 KVM: SEV: Unmap and unpin the GHCB as needed on vCPU free
Unmap and unpin the GHCB as needed when freeing a vCPU.  If the VM is
destroyed after mapping+pinning the GHCB on #VMGEXIT, without re-running
the vCPU, KVM will effectively leak the GHCB and any mappings created for
the GHCB.

Fixes: 291bd20d5d ("KVM: SVM: Add initial support for a VMGEXIT VMEXIT")
Cc: stable@vger.kernel.org
Tested-by: Michael Roth <michael.roth@amd.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-18-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-18-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:43:56 +02:00
Sean Christopherson
f041dc80de KVM: SEV: Decouple the need to sync the GHCB SA from the need to free the SA
Decouple synchronizing the GHCB SA from freeing/unpinning the SA, so that
the free/unpin path can be reused when freeing a vCPU.

Opportunistically add a WARN to harden KVM against stomping over (and thus
leaking) an already-allocated scratch area.

Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-17-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-17-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:43:56 +02:00
Sean Christopherson
08385c5e18 KVM: SEV: Move sev_free_vcpu() down below sev_es_unmap_ghcb()
Relocate sev_free_vcpu() down in sev.c so that it's definition comes after
sev_es_unmap_ghcb().  This will allow sharing unmap functionality between
the two functions without needing a forward declaration (or weird placement
of the common code).

No functional change intended.

Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-16-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-16-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:43:56 +02:00
Sean Christopherson
ce6ea7b33e KVM: SEV: Read start/end indices of PSC requests exactly once per #VMGEXIT
Rework Page State Change (PSC) handling to read the guest-provided start
and end indices exactly once, at the beginning of the request.  Re-reading
the indices is "fine", _if_ the guest is well-behaved.  KVM _should_ be
safe against concurrent guest modification of the indices, but there is
zero reason to introduce unnecessary risk.

Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-14-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-14-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:43:55 +02:00
Sean Christopherson
00b9744a36 KVM: SEV: Add an anonymous "psc" struct to track current PSC metadata
Add a "psc" struct to vcpu_sev_es_state to avoid having to prefix all of
the fields with "psc_".

Take advantage of the code churn to opportunistically rename local
variables to "guest_psc" to make it more obvious that the buffer is guest
data, and more importantly, guest accessible!

Opportunistically rename inflight => batch_size as well, because there can
really only be one operation in-flight (per-vCPU), i.e. "inflight" _looks_
like a boolean, but in actuality is an integer tracking how many pages are
being handled by the current operation.

No functional change intended.

Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-13-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-13-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:43:55 +02:00
Sean Christopherson
17d62e0b15 KVM: SEV: Make it more obvious when KVM is writing back the current PSC index
Increment the guest-visible "cur_entry" index outside of the for-loop
when processing Page State Change entries, and add a comment to make it
more obvious which code is operating on trusted data, and which code is
touching guest-accessible data.

No functional change intended.

Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-12-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260529183549.1104619-12-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03 16:43:55 +02:00
Sean Christopherson
a7e6e09b27 KVM: nSVM: Use kvm_rax_read() now that it's mode-aware
Now that kvm_rax_read() truncates the output value to 32 bits if the
vCPU isn't in 64-bit mode, use it instead of the more verbose (and very
technically slower) kvm_register_read().

Note!  VMLOAD, VMSAVE, and VMRUN emulation are still technically buggy,
as they can use EAX (versus RAX) in 64-bit mode via an operand size
prefix.  Don't bother trying to handle that case, as it would require
decoding the code stream, which would open an entirely different can of
worms, and in practice no sane guest would shove garbage into RAX[63:32]
and then execute VMLOAD/VMSAVE/VMRUN with just EAX.

No functional change intended.

Cc: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260529222223.870923-11-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03 05:35:21 -07:00
Sean Christopherson
fc40b1254c KVM: x86: Add mode-aware versions of kvm_<reg>_{read,write}() helpers
Make kvm_<reg>_{read,write}() mode-aware (where the value is truncated to
32 bits if the vCPU isn't in 64-bit mode), and convert all the intentional
"raw" accesses to kvm_<reg>_{read,write}_raw() versions.  To avoid
confusion and bikeshedding over whether or not explicit 32-bit accesses
should use the "raw" or mode-aware variants, add and use "e" versions, e.g.
for things like RDMSR, WRMSR, and CPUID, where the instruction uses only
bits 31:0, regardless of mode.

No functional change intended (all use of "e" versions is for cases where
the value is already truncated due to bouncing through a u32).

Cc: Binbin Wu <binbin.wu@linux.intel.com>
Cc: Kai Huang <kai.huang@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260529222223.870923-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03 05:35:20 -07:00
Sean Christopherson
ece08316ca KVM: x86: Rename kvm_cache_regs.h => regs.h
Rename kvm_cache_regs.h to simply regs.h, as the "cache" nomenclature is
already a lie (the file deals with state/registers that aren't cached per
se), and so that more code/functionality can be landed in the header
without making it a truly horrible misnomer.

Deliberately drop the kvm_ prefix/namespace to align with other "local"
headers, and to further differentiate regs.h from the public/global
arch/x86/include/asm/kvm_vcpu_regs.h, which sadly needs to stay in asm/
so that the number of registers can be referenced by kvm_vcpu_arch.

No functional change intended.

Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Link: https://patch.msgid.link/20260529222223.870923-7-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03 05:35:18 -07:00
Sean Christopherson
42a842f3f6 KVM: SVM: Truncate INVLPGA address in compatibility mode
Check for full 64-bit mode, not just long mode, when truncating the
virtual address as part of INVLPGA emulation.  Compatibility mode doesn't
support 64-bit addressing.

Note, the FIXME still applies, e.g. if the guest deliberately targeted
EAX while in 64-bit via an address size override.  That flaw isn't worth
fixing as it would require decoding the code stream, which would open an
entirely different can of worms, and in practice no sane guest would shove
garbage into RAX[63:32] and execute INVLPGA.

Note #2, VMSAVE, VMLOAD, and VMRUN all suffer from the same architectural
flaw of not providing the full linear address in a VMCB exit information
field, because, quoting the APM verbatim:

  the linear address is available directly from the guest rAX register

(VMSAVE, VMLOAD, and VMRUN take a physical address, but their behavior
with respect to rAX is otherwise identical).

Fixes: bc9eff67fc ("KVM: SVM: Use default rAX size for INVLPGA emulation")
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Link: https://patch.msgid.link/20260529222223.870923-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03 05:34:45 -07:00
Paolo Bonzini
ffb83b9835 Merge commit 'kvm-vmenter-load-store-regs' into HEAD
Convert the repeated register save/restore sequences into macros,
trading some level of implementation trickiness for conciseness (more
than one register can be saved/restored with one invocation) and a
smaller chance of cut and paste errors between VMX and SVM files.

This becomes more useful with the upcoming support for APX, which
would need to add 32 lines to the VM entry/exit paths.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-30 22:42:15 +02:00
Chang S. Bae
e2e664812f KVM: SEV: Macrofy GPR swapping in __svm_sev_es_vcpu_run()
Convert the SEV-ES entry code to use macros for saving guest GPRs,
following VMX/SVM paths. Drop now-unused register offsets and
__VCPU_REGS_* defines.

No functional change intended.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Link: https://patch.msgid.link/20260512011502.53072-4-chang.seok.bae@intel.com/
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-30 22:41:39 +02:00
Chang S. Bae
8d4fb56794 KVM: SVM: Macrofy GPR swapping in __svm_vcpu_run()
Convert the register save/restore sequences in the SVM entry into macros,
following the VMX code. Drop the now-redundant register offset defines.

No functional change intended.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Link: https://patch.msgid.link/20260512011502.53072-3-chang.seok.bae@intel.com/
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-30 22:41:34 +02:00
Chang S. Bae
c3029c3817 KVM: VMX: Macrofy GPR swapping in __vmx_vcpu_run()
Convert the repeated register save/restore sequences into macros,
trading some level of implementation trickiness for conciseness (more
than one register can be saved/restored with one invocation) and a
smaller chance of cut and paste errors.

This is particularly useful in preparation for extended GPR support;
upcoming support for APX would need to add 32 lines to the VM entry/exit
paths.

No functional change intended.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Link: https://lore.kernel.org/6e67df0e-e5f0-43f5-aa86-22e8b01b75d2@redhat.com
Link: https://patch.msgid.link/20260512011502.53072-2-chang.seok.bae@intel.com/
[Keep kvm_vcpu_regs.h and put the macros in there. -Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-30 22:41:00 +02:00
Paolo Bonzini
713074d53b Merge commit 'kvm-psc-for-7.1' into HEAD 2026-05-29 20:25:59 +02:00
Sean Christopherson
c8cc238093 KVM: SEV: Use READ_ONCE() when reading entries/indices from PSC buffer
Use READ_ONCE() when reading entries/indices from the guest-accessible
Page State Change buffer to defend against TOCTOU bugs.

Don't bother with READ_ONCE()/WRITE_ONCE() for cases where KVM is writing
(and not consuming the result!), as the guest isn't supposed to touch the
buffer while it's being processed.  I.e. using READ_ONCE() is all about
protecting against misbehaving guests.

Fixes: 9b54e248d2 ("KVM: SEV: Add support to handle Page State Change VMGEXIT")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-11-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:29 +02:00
Sean Christopherson
121d88de56 KVM: SEV: Check PSC request indices against the actual size of the buffer
When processing Page State Change (PSC) requests, validate the PSC buffer
against the effective size of the scratch area, which could be less than
the maximum size if the guest provided a pointer that isn't exactly at the
start of the GHCB shared buffer.

Fixes: 9b54e248d2 ("KVM: SEV: Add support to handle Page State Change VMGEXIT")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-10-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:29 +02:00
Sean Christopherson
ebe4b2dc9c KVM: SEV: Don't explicitly pass PSC buffer to snp_begin_psc()
Stop explicitly passing the PSC buffer to snp_begin_psc(): it *must*
be the scratch area.  This will allow fixing a variety of bugs without
further complicating the code.

No functional change intended.

Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-9-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Sean Christopherson
f185e05dce KVM: SEV: WARN if KVM attempts to setup scratch area with min_len==0
Now that all paths in KVM properly validate the length needed for the
scratch area, and are guaranteed to pass in a non-zero length, WARN if KVM
attempts to configured the scratch area with min_len==0 to guard against
future bugs.

Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-8-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Sean Christopherson
5867d7e202 KVM: SEV: Compute the correct max length of the in-GHCB scratch area
When setting the length of the GHCB scratch area, and the area is in the
GHCB shared buffer, set the effective length of the scratch area to the max
possible size given the start of the guest-provided pointer, and the end of
the shared buffer.

The code was "fine" when first introduced, as KVM doesn't consult the
length of the buffer when emulating MMIO, because the passed in @len always
specifies the *max* size required.  But for PSC requests, the incoming @len
is just the minimum length (to process the header), and KVM needs to know
the full size of the scratch area to avoid buffer overflows (spoiler alert).

Opportunistically rename @len => @min_len to better reflect its role.

Fixes: 9b54e248d2 ("KVM: SEV: Add support to handle Page State Change VMGEXIT")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-7-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Sean Christopherson
2be54670bd KVM: SEV: Use the size of the PSC header as the minimum size for PSC requests
When handling a Page State Change (PSC) #VMGEXIT use the size of the PSC
header as the minimum size for the scratch area.  Per the GHCB spec, PSC
requests do NOT provide the length, i.e. using control->exit_info_2 for the
length is completely made up behavior.  The existing code "works", e.g.
even though Linux-as-a-guest always passes '0', because KVM doesn't do
anything with the length when the request is in the GHCB's shared buffer.

Use the header as the min length.  Once the header is retrieved, KVM can
use the specified indices to compute the full size of the request.

Fixes: 9b54e248d2 ("KVM: SEV: Add support to handle Page State Change VMGEXIT")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-6-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Sean Christopherson
3988bd2723 KVM: SEV: Ignore Port I/O requests of length '0'
Explicitly ignore Port I/O requests of length '0' (or count '0'), so that
setting up the software scratch area (and other code) doesn't have to
worry about underflowing the length, and to allow for WARNing on trying
to configure the scratch area with len==0.

Fixes: 291bd20d5d ("KVM: SVM: Add initial support for a VMGEXIT VMEXIT")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Sean Christopherson
dcf1b2d4b0 KVM: SEV: Reject MMIO requests larger than 8 bytes with GHCB v2+
When using GHCB v2+, reject MMIO requests that are larger than 8 bytes.
Per the GHCB spec:

  SW_EXITINFO2 must be less than or equal to 0x7fffffff for version 1 and
  less than or equal to 0x8 for all other versions.

Fixes: 4af663c2f6 ("KVM: SEV: Allow per-guest configuration of GHCB protocol version")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-4-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Sean Christopherson
1aa8a6dc7d KVM: SEV: Ignore MMIO requests of length '0'
Explicitly ignore MMIO requests of length '0', so that setting up the
software scratch area (and other code) doesn't have to worry about
underflowing the length, and to allow for special casing '0' in the
future.

Fixes: 8f423a80d2 ("KVM: SVM: Support MMIO for an SEV-ES guest")
Cc: stable@vger.kernel.org
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Michael Roth
db3f2195d2 KVM: SEV: Require in-GHCB scratch area if GHCB v2+ is in use
As per the GHCB spec, when using GHCB v2+ require the software scratch area
to reside in the GHCB's shared buffer.  Note, things like Page State Change
(PSC) requests _rely_ on this behavior, as the guest can't provide a length
when making the request, i.e. the size of the guest payload is bounded by
the size of the shared buffer.

Failure to force usage of the GHCB, and a slew of other flaws, lets a
malicious SNP guest corrupt host kernel heap memory, and leak host heap
layout information.

setup_vmgexit_scratch() allocates a buffer via kvzalloc(exit_info_2),
where exit_info_2 is guest-controlled. With exit_info_2=24, this yields
a 24-byte allocation in kmalloc-cg-32 (32-byte slab objects). The buffer
holds an 8-byte psc_hdr followed by 8-byte psc_entry structs, so only
entries[0] and entries[1] are in-bounds.

snp_begin_psc() validates end_entry against VMGEXIT_PSC_MAX_COUNT (253)
but NOT against the actual buffer size:

      idx_end = hdr->end_entry;

      if (idx_end >= VMGEXIT_PSC_MAX_COUNT) {   // checks 253, not buffer
          snp_complete_psc(svm, ...);
          return 1;
      }

      for (idx = idx_start; idx <= idx_end; idx++) {
          entry_start = entries[idx];           // OOB when idx >= 2

The guest sets end_entry=10+, causing the host to iterate entries[2+]
which are OOB into adjacent slab objects. For each OOB entry:

  - The host reads 8 bytes (OOB READ / info leak oracle)
  - If the data passes PSC validation, __snp_complete_one_psc() writes
    cur_page = 1 or 512 into the entry (OOB WRITE, sev.c:3806)
  - If validation fails, the error response reveals whether adjacent
    memory is zero vs non-zero (information disclosure to guest)

The guest controls allocation size (exit_info_2), entry range
(cur_entry/end_entry), and can fire unlimited VMGEXITs to repeatedly
hit different slab positions.

By exploiting the variety of bugs, a malicious SEV-SNP guest can:
    - OOB read adjacent kmalloc-cg-32 objects (heap layout disclosure)
    - OOB write cur_page bits into adjacent objects (heap corruption)
    - Trigger use-after-free conditions across VMGEXITs

E.g. with KASAN enabled, a single insmod of the PoC guest module
produces 73 KASAN reports:

    BUG: KASAN: slab-out-of-bounds in snp_begin_psc+0x126/0x890
    Read of size 8 at addr ffff888219ffb5e0 by task qemu-system-x86/2199

    BUG: KASAN: slab-out-of-bounds in snp_begin_psc+0x468/0x890
    Write of size 8 at addr ffff888351566648 by task qemu-system-x86/2199

    The buggy address belongs to the object at ffff888XXXXXXXXX
     which belongs to the cache kmalloc-cg-32 of size 32
    The buggy address is located N bytes to the right of
     allocated 32-byte region [ffff888XXXXXXXXX, ffff888XXXXXXXXX)

  Breakdown:
    62 slab-out-of-bounds (reads + writes past allocation)
     7 slab-use-after-free
     4 use-after-free

All credit to Stan for the wonderful description and reproducer!

Reported-by: Stan Shaw <shawstan96@gmail.com>
Cc: Michael Roth <michael.roth@amd.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Peter Gonda <pgonda@google.com>
Cc: Jacky Li <jackyli@google.com>
Fixes: 4af663c2f6 ("KVM: SEV: Allow per-guest configuration of GHCB protocol version")
Cc: stable@vger.kernel.org
Signed-off-by: Michael Roth <michael.roth@amd.com>
[sean: write changelog]
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260501202250.2115252-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-05-29 20:25:28 +02:00
Paolo Bonzini
b397897016 KVM x86 fixes for 7.1-rcN
- Include the kernel's linux/mman.h in KVM selftests to ensure MADV_COLLAPSE
    is defined, as older libc versions may not provide it.
 
  - Include execinfo.h if and only if KVM selftests are building against glibc,
    and provide a test_dump_stack() for non-glibc builds.
 
  - Fudge around an RCU splat in the emegerncy reboot code that is technically
    a legitimate flaw, but in practice is a non-issue and fixing the flaw, e.g.
    by adding locking, would incur meaningful risk, i.e. do more harm than good.
 
  - Rate-limit global clock updates once again (but without delayed work), as
    KVM was subtly relying on the old rate-limiting for NPT correction to guard
    against "update storms" when running without a master clock on systems with
    overcommitted CPUs.
 
  - Fix a brown paper bag goof where KVM checked if ERAPS is "dirty" instead of
    marking it dirty when emulating INVPCID.
 
  - Flush the TLB when transitioning from xAVIC => x2AVIC to ensure the CPU TLB
    doesn't contain AVIC-tagged entries for the APIC base GPA.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmoZtdgACgkQOlYIJqCj
 N/26sw/+IWOA5AxyoNW/lKAhhkzDTzGWrNCQkpMv+F4tOUbHYniTxI/pv4L3eMvf
 ZLUXijYxhpJtnblLtrnPpSFl5tll4xQdMUv7+fljgpYmy6+erQHodtCgRi5wHDbM
 NlD7DWOgwmpvzYLcybq1RfjZ3n+OBRvq95haQ6Ph4FtoYuIomtJ5tF2mnMlyxlc/
 aIK5wzQ/JeYdQxwwz1ctlHkgE5bPnS+Sxr33+MRFQ5cIpuwdoS9zYRITNBM107kg
 bLeei8Cxh91sgEidgwS8JToLvaEQH8AodkROjcScllwUxYsshPKsHeH7sTMbCOVd
 DiH9VbheZo7d4kb6pvhGsY891ec00dR5E/l2gZYLWHg4v0lINTw6uBdoJuq3t2TO
 Q3KmGVaUWz+c6dY/0qntVpws35zG106S8Pp4mx/1EnUHbJKZYDsUMC1ppwhrr3Pz
 WEyQ9PFXhOyoSbrtOaEfU+wsFPeAfT9eYADu7oV1t7l75TJAKW1EEaSGfzOO/crj
 3GK3vRq2B1cMHX9c4fwhSs4h8k5JvKlI/mtGPxZN3khVorx9dv/rTqOoeQEsFS5+
 8s5XcNPPJlKfNXcu3Jq6rn8U/JA2HnbH298Nk5uXTCfTrZtDgbOnI8YVYWnoadOl
 8xJoie5ccEsysVj1npNNh61LNMF1XBUUC+eNn0I1o0NzeRauxF8=
 =QQUn
 -----END PGP SIGNATURE-----

Merge tag 'kvm-x86-fixes-7.1-rc6' of https://github.com/kvm-x86/linux into HEAD

KVM x86 fixes for 7.1-rcN

 - Include the kernel's linux/mman.h in KVM selftests to ensure MADV_COLLAPSE
   is defined, as older libc versions may not provide it.

 - Include execinfo.h if and only if KVM selftests are building against glibc,
   and provide a test_dump_stack() for non-glibc builds.

 - Fudge around an RCU splat in the emegerncy reboot code that is technically
   a legitimate flaw, but in practice is a non-issue and fixing the flaw, e.g.
   by adding locking, would incur meaningful risk, i.e. do more harm than good.

 - Rate-limit global clock updates once again (but without delayed work), as
   KVM was subtly relying on the old rate-limiting for NPT correction to guard
   against "update storms" when running without a master clock on systems with
   overcommitted CPUs.

 - Fix a brown paper bag goof where KVM checked if ERAPS is "dirty" instead of
   marking it dirty when emulating INVPCID.

 - Flush the TLB when transitioning from xAVIC => x2AVIC to ensure the CPU TLB
   doesn't contain AVIC-tagged entries for the APIC base GPA.
2026-05-29 19:28:16 +02:00
Jim Mattson
f4805dcf2d KVM: x86/pmu: Allow Host-Only/Guest-Only bits with nSVM and mediated PMU
Now that KVM correctly handles Host-Only and Guest-Only bits in the
event selector MSRs, allow the guest to set them if the vCPU advertises
SVM and uses the mediated PMU.

Signed-off-by: Jim Mattson <jmattson@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260527234711.4175166-14-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-05-27 19:02:52 -07:00
Yosry Ahmed
a02a25a652 KVM: x86/pmu: Reprogram Host/Guest-Only counters on nested transitions
Reprogram PMU counters on nested transitions for the mediated PMU, to
re-evaluate Host-Only and Guest-Only bits and enable/disable the PMU
counters accordingly. For example, if Host-Only is set and Guest-Only is
cleared, a counter should be disabled when entering guest mode and
enabled when exiting guest mode.

According to the APM, when EFER.SVME is cleared, setting Host-Only or
Guest-Only disables the counter, so also trigger counter reprogramming
when EFER.SVME is toggled.

Counters setting any of Host-Only and Guest-Only bits are already being
tracked in pmc_has_mode_specific_enables, use the bitmap to reprogram
these counters.

Reprogram the counters synchronously on nested VMRUN/#VMEXIT and
EFER.SVME toggling. This is necessary as these instructions are counted
based on the new CPU state (after the instruction is retired in
hardware).  Hence, the PMU needs to be updated before instruction
emulation is completed and kvm_pmu_instruction_retired() is called.

Defer reprogramming the counters when force leaving guest mode through
svm_leave_nested() to avoid potentially reading stale state (e.g.
incorrect EFER). All flows force leaving nested are non-architectural,
so accuracy is irrelevant.

Refactor a helper out of kvm_pmu_request_reprogram_counters() that
accepts a boolean allowing synchronous vs deferred reprogramming, and
use that from SVM code to support both scenarios.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260527234711.4175166-13-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-05-27 19:02:14 -07:00