Instead of an explicit GUEST_SYNC() after each access+#PF, run another
thread that keeps sending SIGUSR to the vCPU thread, essentially
triggering exits to userspace and save+restore on random points in guest
execution. This makes the test a lot more meaningful as it opens the
door to exercising race conditions between #PF handling in the guest
and save+restore in the host.
The signals are ignored using SIG_IGN outside of __vcpu_run() to avoid
interrupting other ioctls/sysctls performed by the test.
Assisted-by: Gemini:gemini-3.1-pro
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-12-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add a basic stress test for handling #PFs in a guest while the host is
doing save+restore cycles. The guest periodically accesses non-present
memory causing a #PF, and the #PF handler walks the page tables and
updates the PTE to be present, like a proper #PF handler.
After every access (and #PF), the guest triggers a sync and the test
performs save+restore of the VM. This is not very meaningful as
save+restore are performed after the access and #PF handling complete,
but following changes will change that.
Assisted-by: Gemini:gemini-3.1-pro
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-11-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
init_vmcs_control_fields() sets PFEC_MASK and PFEC_MATCH so that they
never match, which reverses the meaning of the PF_VECTOR bit in
EXCEPTION_BITMAP (which is zeroed), effectively enabling #PF
interception by default.
The relevant part of the SDM describes this:
When a page fault occurs, a processor consults (1) bit 14 of the
exception bitmap; (2) the error code produced with the page fault [PFEC];
(3) the page-fault error-code mask field [PFEC_MASK]; and (4) the
page-fault error-code match field [PFEC_MATCH].
It checks if PFEC & PFEC_MASK = PFEC_MATCH. If there is equality, the
specification of bit 14 in the exception bitmap is followed (for example,
a VM exit occurs if that bit is set). If there is inequality, the meaning
of that bit is reversed (for example, a VM exit occurs if that bit is
clear).
Clear PFEC_MATCH such that there is equality, and the #PF bit in the
exception bitmap is followed, so that #PFs are not intercepted by default,
same as every other exception. This also allows tests to set the #PF bit
in the exception bitmap if they want to intercept #PFs, without having to
muck with the PFEC_{MASK,MATCH} fields.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-10-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Expose a guest_mmu to the guest to allow guest code to use the PTE masks
for page table manipulation. Since guest page tables are not mapped in
the guest by default, zero the PGD in guest_mmu in an attempt to make it
more difficult for new tests to shoot themselves in the foot and assume
that page tables can be immediately used by guest code.
Ultimately, guest code can read CR3 any way, so guest_mmu.pgd doesn't
add a lot of value.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-9-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Add a blank line for visual separation, which is especially useful for
tests that use carriage returns for same-line printing, in case a
failure occurs before a blank line is ever logged by the test.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-8-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Now that nVMX test codes preserves GPRs across nested VM-Exits
(specifically RBP, RDX, and RDI among others), drop the ucall-specific
hack to avoid clobbering these registers.
Assisted-by: Gemini:gemini-3.1-pro
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-7-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reuse the GPR switching logic for nVMX by defining VMX_SWITCH_GPRS_ASM,
which is essentially the same as SVM_SWITCH_GPRS_ASM but also switches
RAX and doesn't switch RFLAGS, replacing the push/pop of a subset of the
registers.
The long clobber list of registers is no longer needed as registers are
saved and restored appropriately (and not clobbered by L2).
Define VMX_SWITCH_GPRS_ASM before including evmcs.h, such that it can be
used by evmcs_vmlaunch() and evmcs_vmresume().
This replaces the apparently thread-safe push/pop sequence with the
global GPR switching logic used by SVM, which isn't thread-safe at all.
However this is still an improvement because:
- The VMX logic is half-baked and prompts the UCALL clobber hack as it
doesn't properly save/restore everything. Reusing the GPR switching
logic used by SVM allows for dropping that hack.
- Hitting a problem due to half-baked GPR save/restore logic is arguably
more likely than thread-safety. Evidently, adding more involved stress
tests fails on VMX with the existing push/pop sequence. OTOH, there
are no known failures on SVM due to lack of thread-safety fo
save/restore. Only one test currently uses more than one vCPU with
nested (the memstress test).
The logical next step is to move the guest_regs to be per-vCPU,
making it thread-safe for both VMX and SVM in a proper way.
Assisted-by: Gemini:gemini-3.1-pro
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-6-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Instead of handling rflags separately, add it to guest_regs. No
functional change intended.
Assisted-by: Gemini:Gemini-Next
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-5-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
The assembly code defined by SAVE_GPR_C uses the wrong offsets for some
registers in guest_regs. For example, the offset of RCX should be 0x08
not 0x10. Also, the last offset in the struct (R15) is 0x78, not 0x80,
so the code actually saves and restore beyond the end of gpr64_regs.
Eliminate hardcoded offsets by dynamically generating offsets using
offset_of() and using macros to pass the offsets to assembly as asm
constraints.
To avoid register conflicts in inline assembly (since almost all GPRs are
context-switched), access guest_regs via absolute symbol addressing
(guest_regs + offset) rather than using a base register which could
get overwritten mid-assembly.
While at it, rename SAVE_GPR_C and LOAD_GPR_C to a single macro,
SVM_SWITCH_GPRS_ASM, rename gpr64_regs to guest_regs, and expose it in
processor.h (in preparation for reusing it for VMX).
Assisted-by: Gemini:Gemini-Next
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-4-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
The offsets used (0x170 and 0x1f8) are offsets within vmcb_save_area,
not vmcb. The correct offsets should include the base of vmcb_save_area
within vmcb (which is 0x400 -- so 0x570 and 0x5f8).
Instead of just correcting the offsets, use vmcb->save.rax and
vmcb->save.rflags as parameters to the asm block and avoid hardcoding
offsets completely. While at it, also use guest_regs.rax directly
instead of assuming it's at offset 0 of guest_regs.
Note: "+m" must be used for vmcb_rax and vmcb_rflags, as caching those
fields in registers would be wrong as the underlying KVM will update
them in memory.
The same problem was recently fixed (differently) for kvm-unit-tests
[1].
[1]https://lore.kernel.org/all/20260521092311.86030-1-pbonzini@redhat.com/
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260518202514.2037078-1-yosry%40kernel.org?part=1
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-3-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Drop the custom defined XSTR() macros in KVM selftests and use
__stringify() instead. Include stringify.h in test_util.h to make it
available for all tests instead of including it in all the tests that
need it, as more tests will start using it.
No functional change intended.
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260728174232.2423257-2-yosry@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Pin the steal time test's tasks to a random pCPU in the system instead of
hardcoding the test to always run on pCPU0 as a cheap way of increasing
test coverage, and to do the "right thing" if the parent task of the test
doesn't have pCPU0 in its CPU affinity mask.
Link: https://patch.msgid.link/20260707150706.1198541-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
In the steal time test, don't explicitly set the CPU affinity mask of the
worker child and instead rely on the child inheriting the affinity of the
main thread. Per the pthread_create()[1] and pthread_setaffinity_np()[2]
documentation, new threads inherit the parent's affinity mask.
Linux-specific details
The new thread inherits copies of the calling thread's capability
sets (see capabilities(7)) and CPU affinity mask (see
sched_setaffinity(2)).
Out of an abundance of caution, assert that the child did indeed inherit
the CPU affinity mask, as the test will hang indefinitely if the system is
under light load.
Dropping use of pthread_attr_setaffinity_np() allows building the steal
time test against non-glibc C libraries that don't implement that GNU
extension.
Link: https://man7.org/linux/man-pages/man3/pthread_setaffinity_np.3.html [1]
Link: https://man7.org/linux/man-pages/man3/pthread_create.3.html [1]
Cc: Hisam Mehboob <hisamshar@gmail.com>
Reported-by: Aqib Faruqui <aqibaf@amazon.com>
Closes: https://lore.kernel.org/all/20250829142556.72577-4-aqibaf@amazon.com
Link: https://patch.msgid.link/20260707150706.1198541-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The kvm_mmu is a "god data structure" that includes three different
tasks: describing the guest page table's format, walking the guest
page tables and building the page tables. This means that the
(already poorly named) nested_mmu is only used in part, since it
has no page tables to construct.
Furthermore, some parts are reused across guest and host page
tables (such as the reserved bits detector) but others are not;
for example permission_fault is replaced by simplified code such as
is_executable_pte().
This series cleans this up by splitting kvm_mmu in three parts:
- kvm_pagewalk is the page table walker. There are two of them
per vCPU, gva_walk and ngpa_walk. walk_mmu is *always* replaced
by a single gva_walk no matter if running an L1 or L2 guest,
unlike in the current code that moves it between root_mmu and
nested_mmu.
- kvm_mmu retains the page table building functionality. It uses
a page table walker to build shadow pages; that is always gva_walk
for root_mmu or ngpa_walk for guest_mmu.
- kvm_page_format allows KVM to operate on PTEs that already exist,
and merges the code around permission_mask() with the pre-existing
struct rsvd_bits_validate. Both kvm_pagewalk and kvm_mmu have their
own kvm_page_format, just like struct kvm_mmu had two instances of
struct rsvd_bits_validate for gPTE and SPTE reserved bit checks.
The cleanup alone already does something useful, which is to reduce
the confusion between guest_mmu and nested_mmu. nested_mmu came to
exist long before the introduction of guest_mmu and stole the obvious
name, resulting in comments like "Exempt nested MMUs" where the code
actually exempts guest_mmu. Renaming guest_mmu could be the next
step, though the RFC had multiple opinions about how to do this.
However, the last patch also shows the code reuse benefits can be used
for new features too. By adapting the permission_fault() machinery and
using it to test SPTEs against struct kvm_page_fault, it makes it possible
to support SPTEs that have XS!=XU; these were not supported yet by KVM,
but could now be added via memory attributes.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
is_access_allowed(), and is_executable_pte() within it, are effectively a
special version of permission_fault() that only supports a subset of roles.
In particular it does not allow SMEP, SMAP and PKE; while SMAP and PKE are
not a problem (they are not supported by EPT/NPT and is_access_allowed() is
only used for either EPT/NPT or CR0.PG=0), lack of ring-0 execution checks
support means that KVM can only use MBEC and GMET with shadow paging.
Replace is_access_allowed() with a modified version of permission_fault(),
looking at the same bitmasks but using the struct kvm_mmu's fmt member;
the new version supports MBEC/GMET for free, just by virtue of filling
in the ACC_* masks from the SPTE.
This prepares for a possible future where TDP entries could have XS!=XU,
for example as part of implementing Hyper-V VSM natively inside KVM.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Make it possible to apply the computation loop to both guest
and shadow PTEs formats; the latter do not have an extended role, so
pass the four parameters to the function one by one.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This remove one level of indirection, and adds the data for the permission
bitmask machinery to struct kvm_mmu. This way, it will be possible to
reuse the permission bitmasks for SPTEs as well.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Structs kvm_mmu ("build page tables") and kvm_pagewalk ("walk page
tables") share a piece of common functionality, namely "looking at PTEs".
Both of them have code to check is a specific access (described by
PFERR_* constants) is allowed by a PTE, and both of them also validate
that reserved bits are zero on the respective page tables page tables;
for SPTEs the code is only there to check internal consistency, but in
this case it is indeed shared via struct rsvd_bits_validate.
In preparation for sharing more PTE parsing code between struct
kvm_pagewalk and struct kvm_mmu, create a new struct that contains all
precalculated tables, including the data that is extracted from the
CPU role. For now only struct kvm_pagewalk uses it.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Now that the GVA->GPA page walker is initialized independently,
init_kvm_softmmu() does not do anything more than calling
kvm_init_shadow_mmu() so eliminate it from the call chain.
At the same time, rename kvm_init_shadow_mmu() to
init_kvm_shadow_mmu() for consistency with init_kvm_tdp_mmu().
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
At this point, vcpu->arch.ngva_walk and vcpu->arch.root_gva_walk contain
the same information; compare init_kvm_page_walk() on one side with
init_kvm_softmmu() + shadow_mmu_init_context() on the other. They only
differ in when each is active, and root_gva_walk is also used by shadow
paging, via FNAME(walk_addr) and its callers.
Always use the same instance of kvm_pagewalk to do GVA->GPA translations,
for both guest emulation and shadow paging, instead of flipping the
gva_walk pointer back and forth. After all the page walking does behave
the same no matter if you are in guest mode or not; the difference lies
in the behavior of kvm_translate_gpa and thus in vcpu->arch.mmu, not in
the page walker itself.
This completes the transition from walk_mmu/nested_mmu as the page
walking entry points to gva_walk/ngpa_walk, and removes duplicated
code between the initialization of root_mmu.w and ngva_walk (The
Struct Formerly Known As nested_mmu).
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Replace kvm_mmu's w field with a pointer to an external instance of
struct kvm_pagewalk. This is the first step towards using a single
kvm_pagewalk struct for all GVA walks, whether nested or not.
With this patch, non-MMU code basically does not use kvm_mmu anymore:
it does care about page walks, but it funnels (almost) all interactions
with the TLB to mmu.c.
kvm_mmu_invalidate_addr() still needs to go from kvm_pagewalk to kvm_mmu,
and it cannot anymore use container_of() to do it; the mapping is hardcoded
based on the provided kvm_pagewalk, since there are just three of them.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
nested_mmu is now only used for its w member. While there is still a
single container_of() going from (possibly) gva_walk to its containing
struct kvm_mmu, it is never reached for nested_mmu and therefore it is
safe to strip nested_mmu.w out of its containing struct kvm_mmu.
So do it, and rename it following the model of gva_walk itself.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Now that walk_mmu is only accessed for its "w" member, store
directly the pointer to it. Since it is only used to convert
guest GVAs or nGVAs, call it gva_walk.
Note that there is still one container_of() going from (possibly)
walk_mmu.w to its containing struct kvm_mmu, but for now all instances
of struct kvm_pagewalk do live within a kvm_mmu.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
kvm_mmu_invalidate_addr()'s callers only want to tell it whether to
invalidate a GVA or GPA. This will ultimately be represented by two
different kvm_pagewalk structs, so adjust the type of the parameter.
As of this patch, the GVA case is represented by both root_mmu.w and
nested_mmu.w. Since nested_mmu never has a sync_spte callback, it would
exit at its check, but really nested_mmu should not be a kvm_mmu in the
first place: it is only used as the walk_mmu, and walk_mmu is only used
for its struct kvm_pagewalk member.
Since calling container_of() on the nested_mmu would be bogus after it is
turned into a struct kvm_pagewalk, introduce a separate check to check
if no work is needed beyond kvm_x86_call(flush_tlb_gva). Implement it
so that it is as similar as possible to what was happening until now; that
is, do nothing if the invalidation is happening for a nested GVA.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
As promised, this removes the remaining instances of
container_of(w, struct kvm_mmu, w), meaning that struct
kvm_pagewalk's definition is pretty much complete.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
With this change, walk_addr_generic and its callees do not need to use
container_of() anymore. There are only two remaining occurrences of
container_of, in permission_fault() and kvm_mmu_refresh_passthrough_bits().
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
struct kvm_pagewalk's behavior depends on the CPU state and its
page format. Move related fields so that walk_mmu remains
self contained.
Note that for now, some of the accessors still use kvm_mmu
to split the churn.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Injection of page faults is also part of accesses to guest
page tables; in particular, __kvm_inject_emulated_page_fault()
calls it on walk_mmu. Move it to struct kvm_pagewalk as
part of converting walk_mmu to a struct kvm_pagewalk.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Continue with yet another callback used in FNAME(walk_addr_generic),
as another step towards removing container_of() from there.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
gva_to_gpa is the main entry point into walk_mmu, which
is only used for guest page table walking (as opposed to building
the page tables). Moving gva_to_gpa to struct kvm_pagewalk
is a step towards making walk_mmu a struct kvm_pagewalk, and
removes several uses of struct kvm_mmu in x86.c.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Start moving page walking functionality out of kvm_mmu; the easiest
target is the callbacks.
Change the kvm_mmu_get_guest_pgd() wrapper to take a struct kvm_pagewalk
too, avoiding the MMU indirection (and associated container_of) whenever
the caller already has one. All container_of uses need to go before
nested_mmu can be changed to a struct kvm_pagewalk, signifying that it
is not used to build page tables.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
In preparation for separating walking and building of page tables,
introduce a dummy struct kvm_pagewalk and pass it around instead of
its containing kvm_mmu to functions that do not build the page tables.
Outermost functions retrieve the mmu via container_of, while internal
functions can pass around the struct kvm_pagewalk pointer.
x86.c is still mostly oblivious to the existence of struct kvm_pagewalk,
with are only a couple exceptions for now, but the plan is for it to
use struct kvm_pagewalk whenever dealing with guest page tables and have
only limited knowledge of struct kvm_mmu.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Just always go through kvm_translate_gpa(), which will either invoke
the vendor check or just return hc->ingpa back.
This is a better way to fix the issue of commit 464af6fc2b ("KVM:
x86: check for nEPT/nNPT in slow flush hypercalls", 2026-05-03).
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- Automatically allocate a full page for L2 guest stacks on x86 instead of
requiring test-specific L1 guest code to carve out a portion of the L1
stack for L2 usage, and to ensure the L2 stack also adheres to the x86-64
calling convention ABI.
- Add a selftest to verify {Guest,Host}-Only behavior in x86's mediated PMU.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEKTobbabEP7vbhhN9OlYIJqCjN/0FAmo7/RAACgkQOlYIJqCj
N/36TQ//U6IBajlUQzQ9ChjNq7WBNS0hBzVbDSRYgoqtowgVyVafQ9YBbpg4VMoP
WFfU7jptiPNY1+7Kxz2xylIn33YUjdk7qhDy/G0jsRWyLgkxMLFJwqAGbXbG1nSK
btny8CPwhH5vg3t/a/AEvobtu5m4EWsO8mgGOQphIuh5Dw8GScGxEtcQUuP4Abz9
fl9IkBfUqUNsw22Ddv/F5yLR/6SuyHezRFc/4NYJWckOXYdpDYxgXruh7JS/GE7a
Rh6gpw/TLZXJ0fZUMjqy8i/+VCxG64K+65YqH1vsBiZ5scGJWC7KdUA5kM7coBmA
Ri49LHSP4jROeb/Gu8+v+0JH3Y+Fwl1cRR9qOi8vhc39V8D+vbJyOfO4Hgv2cmOo
xn3qVRA1c2utLZ304oyL2mSJhYKLNp83XFG/PB3BhF9PVgE7X3H8wGlkEVc3tm0q
dhaGPztPi862/C4Wk+XDLecwaPRMR9ypqUD2GvyLG50MeXu8ekG9aJweIqR/orij
qveCAwTiGUtVl4PoUOFcSdRoV5jDVBXGJr8/KRoYNgH+AWNzFslYSZi18F/zlLux
4JZr21w0qR8FWv//6EKfpdkH/g7R+8jhlQ4nuFBid/WYl58r6NFJ++viGCPfeqom
d5f4kZ7buQv38ldR5ORuuSmqDEYMM/8PDMIR6Kmke9WiAJqY6DM=
=AYtV
-----END PGP SIGNATURE-----
Merge tag 'kvm-x86-selftests_l2_stacks-7.3' of https://github.com/kvm-x86/linux into HEAD
KVM selftests for 7.3, early edition
- Automatically allocate a full page for L2 guest stacks on x86 instead of
requiring test-specific L1 guest code to carve out a portion of the L1
stack for L2 usage, and to ensure the L2 stack also adheres to the x86-64
calling convention ABI.
- Add a selftest to verify {Guest,Host}-Only behavior in x86's mediated PMU.
It's still technically spring!
Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding regs.c
(the kvm_cache_regs.h => regs.h is already applied) and msrs.{c,h}, and moving
relevant code out of x86.c.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move the majority of remaining KVM-internal declarations and defines in
kvm_host.h to x86.h, so that kvm_host.h only holds structure and function
definitions that need to be visible to arch-neutral KVM.
Land the emulator interfaces in x86.h, even though kvm_emulate.h *seems*
like a good home, as the interfaces and defines being moved are provided by
x86.c. I.e. keep kvm_emulate.h as an interface to the emulator proper.
Note, any "misses" are likely unintentional.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260613000329.732085-31-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move kvm_mmu_do_page_fault() into mmu.c, as there are no users outside of
mmu.c, and the function typically isn't inlined by the compiler anyways.
This will allow moving the EMULTYPE_xxx definitions into x86.h without
having to include x86.h in mmu_internal.h, i.e. will help preserve the
goal of making x86.h KVM x86's "top-level" include.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-30-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move the implementation of kvm_arch_async_page_ready() "down" in mmu.c so
that it lives below kvm_tdp_page_fault(). This will allow moving
kvm_mmu_do_page_fault() into mmu.c without needing a forward declaration.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-29-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Rename kvm_arch_interrupt_allowed() to kvm_is_interrupt_allowed() and
change its return type to a boolean, as the purpose of the helper is purely
to check if interrupts are architecturally allowed. I.e. whether or not
interrupts are temporarily disallowed due a pending nested VM-Enter is
irrelevant (and callers are most definitely not supposed to care).
Opportunistically bury the helper in x86.c, as it hasn't been referenced by
arch-neutral code since commit a1b37100d9 ("KVM: Reduce runnability
interface with arch support code"), and has long since gained _very_
x86-specific semantics (see above).
Opportunistically add a comment to call out that treating -EBUSY as
"allowed" is intentional.
For all intents and purposes, no functional change intended (KVM treats
-EBUSY as "allowed" before and after).
Cc: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260613000329.732085-28-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
When querying whether or not interrupts (IRQs) are allowed, check for a
pending nested run _after_ checking whether or not interrupts are blocked.
If L1 is running L2 _without_ nested_exit_on_intr(), i.e. if L1 IRQs can
be blocked while running L2, and interrupts will indeed be blocked once the
nested VM-Enter to L2 is completed, then KVM should treat interrupts as not
being allowed.
For injection, this avoids an unnecessary (forced) VM-Exit, as KVM can
immediately request an IRQ window, instead of forcing an exit and _then_
requesting an IRQ window (because after the forced exit, KVM will see that
interrupts are blocked).
For non-injection usage, only kvm_vcpu_ready_for_interrupt_injection() is
affected in practice. Barring KVM bugs or misbehaving userspace (at which
point all architectural guarantees are off), kvm_vcpu_has_events() is
unreachable when a nested run is pending. To reach kvm_vcpu_has_events(),
kvm_vcpu_running() needs to return false, i.e. vcpu->arch.mp_state needs
to be something other than RUNNABLE. If nested_run_pending is true, then
mp_state *must* be RUNNABLE (again barring bugs or stupid userspace),
because KVM shouldn't emulate VMRUN/VMLAUNCH/VMRESUME while the vCPU is
!RUNNABLE.
The one "near miss" is VMX's GUEST_ACTIVITY_STATE field, which allows L1 to
put the vCPU into HLT or WFS as part of nested VMLAUNCH/VMRESUME. However,
KVM clears nested_run_pending prior to calling kvm_emulate_halt_noskip()
when putting L2 into HLT via GUEST_ACTIVITY_HLT, and also clears the flag
before setting mp_state to INIT_RECEIVED. SVM has no equivalent to
GUEST_ACTIVITY_STATE.
I.e. the vCPU will always be runnable if a nested run is pending, and thus
kvm_arch_vcpu_runnable() => kvm_vcpu_has_events() is effectively dead code,
as is __kvm_emulate_halt() => kvm_vcpu_has_events(). Oh, and TDX doesn't
support nested VMX. Similarly, kvm_can_do_async_pf() is unreachable as
KVM shouldn't be faulting in memory with a pending nested VM-Enter.
As for kvm_vcpu_ready_for_interrupt_injection(), KVM's current behavior of
incorrectly treating interrupts as being allowed could result in KVM
prematurely exiting to userspace to accept an ExtINT. But, KVM will still
hold/block the ExtINT and request its own IRQ window. I.e. the net effect
is more or less the same as the for-injection case, the unnecessary exit
just happens at a different boundary.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260613000329.732085-27-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move the declaration of kvm_pv_send_ipi() into lapic.h, as its
implementation is provided by lapic.c (sending PV IPIs relies on the
optimized APIC map provided by the in-kernel local APIC), and it's only
used by KVM x86 code.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-26-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move the function declaration for APIs to get/query pending IRQs from
kvm_host.h to irq.h, as the APIs are only used by KVM x86 code.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-25-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Bury __kvm_irq_line_state() in CONFIG_KVM_IOAPIC=y code, as it's only used
by PIC and I/O APIC code.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-24-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move a variety of "VALID MASK" defines, e.g. that capture which flags in
a given ioctl are supported by KVM, from kvm_host.h to x86.c. The set of
valid flags/bits is very much a KVM-internal detail, as the values from the
hardcoded #defines are often captured and massaged by KVM's setup code, i.e.
*directly* using the macros outside of KVM x86 would be actively dangerous.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-23-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move kvm_{load,read}_ldt() into vmx.c, as vmx_{load,store}_ldt(), as they
are exclusively used by VMX to save/restore host state, and have no
business being globally visible.
Ideally, KVM-specific helpers wouldn't exist at all, as they are nothing
more than assembly wrappers for SLDT and LLDT, i.e. should be provided by
the kernel, not by KVM. Punt that cleanup to the future, as
arch/x86/include/asm/desc.h _does_ provide helpers, but load_ldt() is only
available for CONFIG_PARAVIRT_XXL=n builds, and both {load,store}_ldt()
unnecessarily constrain the operands to memory.
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-22-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move a pile of MMU helper declarations and macros into mmu.h, as they are
very much KVM x86 internal APIs and details, and not intended to be exposed
to arch-neutral KVM, and certainly not to the broader kernel.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-21-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Move the definition of "struct kvm_x86_pmu_event_filter" to pmu.c, as the
as the details of the filters are very much implementation details that can
and should be buried in pmu.c. While the _existence_ of filters is public
knowledge, almost by definition, the contents don't need to be exposed
outside of the PMU code as the filter data is provided by userspace, i.e.
it pretty much has to be dynamically allocated, and thus never should be
fully embedded in a globally visible structure.
No functional change intended
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Message-ID: <20260613000329.732085-20-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
