Add xe_svm_devmem_lru_bump() and set it as drm_gpusvm_ctx.devmem_fn in
the prefetch worker so each backing BO is moved to the LRU tail during
get_pages.
v2:
- Guard xe_svm_devmem_lru_bump() against foreign device-memory
allocations before casting to xe_bo: return early when
devmem_allocation->ops != &dpagemap_devmem_ops. A shared
device_private_page_owner can hand us another driver's allocation.
(Matthew Brost / Sashiko AI review)
- Add stub for function (CI)
v3:
- Guard xe_svm_devmem_lru_bump() body with CONFIG_DRM_XE_PAGEMAP; it
references dpagemap_devmem_ops and to_xe_bo(), which only exist when
PAGEMAP is enabled. Fixes the build with GPUSVM=y, PAGEMAP=n.
(Sashiko AI review)
v4:
- Fix xe_svm_devmem_lru_bump() kernel-doc to explain the locking (Matt)
Suggested-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20261001033602.3166274-9-himal.prasad.ghimiray@intel.com
Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Add an optional drm_gpusvm_ctx.devmem_fn, invoked for each device-memory
allocation backing the range's faulted pages. It runs under the notifier
lock and independently of the DMA map, so it also fires with no_dma_map.
Adjacent pages of one allocation are coalesced, so the callback must be
idempotent.
v2
Reset last to NULL for non-devmem pages(Sashiko/Matt)
v3
Run drm_gpusvm_pages_valid_unlocked() on the map_dma path even when
devmem_fn is set, so an invalidated range's stale inline mapping is
reset.
Suggested-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20261001033602.3166274-8-himal.prasad.ghimiray@intel.com
Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
drm_pagemap_zdd is private, so add drm_pagemap_page_to_devmem() to let
callers get the drm_pagemap_devmem backing a device-private/coherent
page. Add a NULL stub for !CONFIG_ZONE_DEVICE.
Cc: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20261001033602.3166274-7-himal.prasad.ghimiray@intel.com
Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
On GT reset replay, guc_exec_queue_start() rewinds the software ring
tail to the first unsignalled job's head and sets the LRC tail to the
LRC head. This assumes the saved head has not passed an unstarted job,
which holds for ordinary submission as the CS cannot run past a job it
has not executed.
A ULLS job's slot is covered by a tail its predecessor published from
within the ring, so the CS can consume the slot and idle without the
job having run, leaving the saved head past it. On replay the job is
re-emitted behind the head, the context restarts with head == tail and
the job never starts.
Rewind the LRC head to the job's head as well when the first pending
job is a ULLS job, so the re-emitted job is ahead of it. Non-ULLS
replay is unchanged.
Fixes: 40c756bdbb ("drm/xe: Add ULLS migration job support to GuC submission")
Cc: Matthew Brost <matthew.brost@intel.com>
Assisted-by: LLM
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260930094031.3365707-8-varun.gupta@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
A chained ULLS job skips the ring tail write in submit_exec_queue()
since its predecessor's postamble has already published it from within
the ring. After a GT reset the context is re-registered with scheduling
disabled and that postamble is gone, so nothing publishes the replayed
job's tail: the context is enabled with head == tail and the job never
starts. On the kernel migration queue the second timeout wedges the
device.
Publish the tail for a chained job when the queue is not enabled. This
is the state in which submit_exec_queue() already sends the scheduling
enable H2G instead of relying on a resident context, and the tail write
precedes that H2G. guc_exec_queue_stop() clears ENABLED on reset and
register_exec_queue() does not set it, so the first replayed ULLS job
always takes this path; its re-emitted postamble then publishes the
next tail as usual. With the queue enabled nothing changes.
job->restore_replay does not identify this case, it is only set on the
SR-IOV VF pause/unpause path.
Fixes: 40c756bdbb ("drm/xe: Add ULLS migration job support to GuC submission")
Cc: Matthew Brost <matthew.brost@intel.com>
Assisted-by: LLM
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260930094031.3365707-7-varun.gupta@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
The ULLS postamble publishes the next job's ring tail and only then
parks on that job's semaphore. The CS fetches ring contents up to the
tail while parked, so it can fetch the next job's slot while it still
holds padding NOOPs. When the CPU later writes the job and signals the
semaphore, the CS executes the stale fetch, drains to the tail and
idles. The job never runs and its fence never signals.
This is hit whenever the CS parks before the next chained job is
written, e.g. when there is a pause between entering ULLS and the next
migration job. A lost ULLS_EXIT is silent as the next job's seqno write
covers its fence, but a lost ULLS_ACTIVE hangs the migration queue and
the subsequent kernel job timeout wedges the device.
Park first and publish the tail after the wait. The CS never fetches
beyond RING_TAIL, so the next slot cannot be fetched until the job is
in place. This puts the non-posted tail write on the wake-up path,
which the original order was chosen to avoid, but that order is not
safe.
Fixes: 6ec0b87160 ("drm/xe: Add ULLS migration job support to ring ops")
Cc: Matthew Brost <matthew.brost@intel.com>
Assisted-by: LLM
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260930094031.3365707-6-varun.gupta@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Trace the start of VM teardown and ASID removal so late page faults can be
correlated with the VM lifecycle. Emit one failure event for all page fault
errors. Include the existing page fault reason alongside the errno.
Keep the trace set focused by avoiding separate events for userptr pin
attempts, individual close phases, ASID lookup rejection, and faults during
shutdown.
v2: Fix checkpatch warnings.
v3: Pass xe to the page fault tracepoint and print VM mode as
dma-fence/preempt-fence/fault (Matt)
Assisted-by: GitHub Copilot using Claude Sonnet 5
Signed-off-by: Daniel Charles <daniel.charles@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260930164645.1243801-1-daniel.charles@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
It is not necessary to iterate over and dma_fence_put current_fence
fences in the err_out path of ops_execute because current_fences cannot
be non-zero when jumping to that section of code.
Caught by static analysis.
Signed-off-by: Jonathan Cavitt <jonathan.cavitt@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Cc: Shuicheng Lin <shuicheng.lin@intel.com>
Cc: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260929205606.1076177-1-jonathan.cavitt@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Add a print vfunc to xe_pagefault_ops so different producer backends can
log specific relevant debug info while keeping the consumer layer free
of potential backend-specific formatting logic.
Consumer-specific error prints are handled by passing as input into
the vfunc.
v2:
- Decode producer specific fields for print message. (Matthew)
- Combine consumer and producer prints to one print call. (Sashiko)
v3:
- Change assert to WARN_ON_ONCE. (Matthew)
Suggested-by: Stuart Summers <stuart.summers@intel.com>
Signed-off-by: Brian Nguyen <brian3.nguyen@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260928222241.3815153-2-brian3.nguyen@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Commit d4a7e16117 ("drm/xe: batch CT pagefault acks with periodic
flush") added a defer_flush argument to xe_guc_ct_send_locked(). At the
same time it dropped the g2h_len and num_g2h arguments, which had no
in-tree users, and hardcoded a zero G2H reservation.
That leaves no way to send an H2G that expects a G2H reply while
already holding ct->lock. The G2H receive path (parse_g2h_event() and
g2h_fast_path()) releases credit unconditionally for replies such as
DEREGISTER_CONTEXT_DONE, SCHED_CONTEXT_MODE_DONE and
TLB_INVALIDATION_DONE. If the credit was never reserved,
__g2h_release_space() reports an "Invalid G2H release" and marks the CT
dead with CT_DEAD_G2H_RELEASE.
Replacing two u32 arguments with a bool also means a caller written
against the old signature can silently change meaning when the two
reservation arguments get merged into one: the call still compiles,
still sends the H2G, and loses only the reservation.
Add g2h_len and num_g2h back, ahead of defer_flush, so that the
function matches xe_guc_ct_send() and forwards both values to
guc_ct_send_locked(). The kernel-doc now says when a reservation is
required and that a non-zero reservation must not be made from a G2H
handler, because waiting for credit can dequeue G2H inline.
The only existing caller, the pagefault ack path, does not expect a
reply. It passes 0, 0 and behaves exactly as before.
Cc: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Sujaritha Sundaresan <sujaritha.sundaresan@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260929050342.221789-1-sujaritha.sundaresan@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Add a tuning entry to set bit 14 (NONCOMPMEMRD256BOVRFETCHEN) of
L3SQCREG2 for Xe3p and later platforms.
BSpec: 72161, 59927
Signed-off-by: Daniel Charles <daniel.charles@intel.com>
Reviewed-by: Matt Roper <matthew.d.roper@intel.com>
Link: https://patch.msgid.link/20260929200839.843116-1-daniel.charles@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Now that the GSC FW is defined, we can enable PXP for NVL-S. The feature
will only be turned on if the binary is found on disk.
Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
Reviewed-by: Julia Filipchuk <julia.filipchuk@intel.com>
Link: https://patch.msgid.link/20260923183600.1186943-6-alan.previn.teres.alexis@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
NVL-S is identified by GSC major version 108.
The compatibility version is 0.0.
Signed-off-by: Alan Previn <alan.previn.teres.alexis@intel.com>
Reviewed-by: Julia Filipchuk <julia.filipchuk@intel.com>
Link: https://patch.msgid.link/20260923183600.1186943-5-alan.previn.teres.alexis@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Opt in to the GuC workaround, to ignore context completion status
with preemption or preemption-to-idle in the CSB when the multi-queue
feature is enabled, which is supported on Graphics Version 35.10.
This WA is supported starting from GuC 70.73.
Apply Wa_16030224309 to Graphics Version 35.10.
v2: Place this code near to KLV WAs. -Lin Shuicheng
Signed-off-by: Tangudu Tilak Tirumalesh <tilak.tirumalesh.tangudu@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Link: https://patch.msgid.link/20260925172722.238924-1-tilak.tirumalesh.tangudu@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Original commit 0f485f8865 ("drm/xe/i2c: use device_create_managed_software_node()")
got silently removed by commit cc46796964 ("Merge drm/drm-next into drm-xe-next").
Re-apply it.
Signed-off-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
Link: https://patch.msgid.link/20260922-xe-i2c-reapply-swnode-unreg-v1-1-1bdda5d44701@oss.qualcomm.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
[Rodrigo: Modified commit message to reflect the right history]
Add a kernel-doc DOC section at the top of xe_migrate.c describing the
Ultra Low Latency Submission (ULLS) scheme used for migration jobs.
Cover the motivation (removing the H2G / GuC / context switch latency
from the page fault and SVM prefetch critical paths), the platform
requirements, the LRC PPHWSP semaphore layout and its relationship to
the migration queue job count, the fixed ULLS job size and why it is
needed, the ring preamble / postamble emitted by the ring ops including
the in-ring tail update, the semaphore-only submission fast path in the
GuC backend, and the enter / delayed exit flow along with the ULLS job
flags.
Hook the new section into Documentation/gpu/xe/xe_migrate.rst.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Assisted-by: Github-Copilot:Claude-opus-5
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-25-matthew.brost@intel.com
Add a configfs attribute to control the migrate ULLS period.
The migrate ULLS period specifies how long, in milliseconds, migrate
ULLS remains active before exiting. Migrate ULLS is currently used by
page-fault handling and SVM prefetch operations to process migration
work through a continuously running batch.
The attribute defaults to 5 ms. Setting the value to 0 disables ULLS.
This knob is primarily intended for performance tuning and evaluation,
allowing experimentation with the trade-off between reducing ULLS
startup/teardown overhead and limiting the time spent in a continuously
running migration workload.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-24-matthew.brost@intel.com
Call xe_migrate_ulls_enter upon page fault or SVM prefetch in an
effort speed up these critical paths.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-23-matthew.brost@intel.com
Add ULLS migration job support to GuC submission backend.
Changes required:
- On migration queue, reduce max jobs to the number of ULLS semaphores
minus one
- Skip writing the saved ring tail for ULLS jobs except for the first
ULLS job - the ring updates both the saved tail and the tail register
itself
- Set ULLS sempahore for current job releasing last job except for first
ULLS job
- Suppress submit H2G for ULLS except for first ULLS job
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-22-matthew.brost@intel.com
Add preamble and postamble for ULLS migrations jobs. Preamble clears
current semaphore for reuse. Postamble waits on next semaphore which is
set upon next job submission, then advances the ring tail over that job
with an LRI to RING_TAIL, so submitting it costs the CPU nothing beyond
signalling the semaphore.
A job updates the tail on behalf of a successor which has not been
emitted yet, so it cannot know how much ring that successor will occupy.
Pad every ULLS job out to a fixed ULLS_JOB_SIZE_BYTES, which makes the
next tail derivable from where the current job starts. The pad also
supplies the NOPs which must follow an in-ring tail update.
The last ULLS migration job skips BB submission, the postamble and the
tail update (clear current semaphore, write seqno, exit ULLS), padding
the difference so that it still fills a job slot.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-21-matthew.brost@intel.com
Add function to enter ULLS mode for migration job and delayed worker to
exit (power saving). ULLS mode expected to entered upon page fault or
SVM prefetch. ULLS mode exit delay is currently set to 5ms.
ULLS mode only support on DGFX and USM platforms where a hardware engine
is reserved for migrations jobs. When in ULLS mode, set several flags on
migration jobs so submission backend / ring ops can properly submit in
ULLS mode.
Upon ULLS mode enter, send a job trigger waiting a semphore pipling
initial GuC / HW conetxt switch.
Upon ULLS mode exit, send a job to trigger that current ULLS
semaphore so the ring can be taken off the hardware.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-20-matthew.brost@intel.com
Define memory layout for ULLS semaphores stored in LRC memory. Add
support functions to return GGTT address and set semaphore based on a
job's seqno.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-19-matthew.brost@intel.com
Some multi-tile devices may want to mirror page tables across tiles for
memory-bandwidth reasons, while others may not. Add a device flag that
allows enabling or disabling page-table mirroring across tiles.
Setting the flag to true (the existing behavior) on PVC, but both modes
have been tested and are working on PVC.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-18-matthew.brost@intel.com
With CPU binds, it no longer makes sense to implement CPU bind handling
in the migrate layer, as these operations are entirely decoupled from
hardware. Introduce a dedicated CPU bind layer stored at the device
level.
Since CPU binds are tile-independent, update the PT layer to generate a
single bind job even when pages are mirrored across tiles.
This patch is large because the refactor touches multiple file / layers
and ensures functional equivalence before and after the change.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-17-matthew.brost@intel.com
Since bind jobs execute on the CPU rather than the GPU, maintaining a
per-tile bind queue no longer provides value. Convert the driver to use
a single bind queue shared across tiles. The primary change is routing
all GT TLB invalidations through this unified bind queue.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-16-matthew.brost@intel.com
Both populate and clear have unused void* ptr arguments, remove these.
Also consolidate ofs, num_qwords in update argument.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-15-matthew.brost@intel.com
No reason to use the GPU for binds.
Benefits of CPU-based binds:
- Lower latency once dependencies are resolved, as there is no
interaction with the GuC or a hardware context switch both of which
are relatively slow.
- Large arrays of binds do not risk running out of migration PTEs,
avoiding -ENOBUFS being returned to userspace.
- Kernel binds are decoupled from the migration exec queue (which issues
copies and clears), so they cannot get stuck behind unrelated
jobs—this can be a problem with parallel GPU faults.
- Paves the for path decouping binds from tiles and individual engines
- Enables ULLS on the migration exec queue, as this queue has exclusive
access to the paging copy engine.
Update migration layer to formulate a PT job which will issue CPU bind
in the submission backend.
All code related to GPU-based binding has been removed.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-14-matthew.brost@intel.com
Now that the CPU is always used for binds even in jobs, CPU bind jobs
can pass GPU jobs in the same exec queue resulting dma-fences signaling
out-of-order. Use a dedicated exec queue for binds issued from page
faults to avoid ordering issues and avoid blocking kernel binds on
unrelated copies / clears.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-13-matthew.brost@intel.com
The level was previously extracted from struct xe_pt inside
xe_vm_pgtable_update during CPU binds, which always occurred during the
bind IOCTL. With CPU binds now supported in bind jobs, struct xe_pt may
no longer be valid in memory at that point. To address this, store the
level directly in struct xe_vm_pgtable_update.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-12-matthew.brost@intel.com
PT jobs bypass GPU execution for the final step of a bind job, using the
CPU to program the required page tables. Teach the GuC submission backend
how to execute these jobs.
PT job submission is implemented in the GuC backend for simplicity. A
follow-up patch could introduce a dedicated backend for PT jobs.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-11-matthew.brost@intel.com
Add struct xe_pt_job_ops, a dynamically refcounted object that contains
the information required to issue a CPU bind via a job after the initial
bind IOCTL returns.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-10-matthew.brost@intel.com
Add helpers to access PT ops, making it easier to shuffle the location of
the ops structures without requiring widespread code changes.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-9-matthew.brost@intel.com
Update the scheduler job layer to support PT jobs. PT jobs are executed
entirely on the CPU and do not require LRC fences or a batch address.
Repurpose the LRC fence storage to hold PT‑job arguments and update the
scheduler job layer to distinguish between PT jobs and jobs that require
an LRC.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-8-matthew.brost@intel.com
Update the xe_bo_put_deferred arguments to include a writeback flag,
which indicates whether the BO was added to the deferred list. This is
useful when the caller needs to take additional actions after the BO has
been queued for deferred release.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-7-matthew.brost@intel.com
Add the job count to the GuC exec queue snapshot, as this is useful
debug information.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-6-matthew.brost@intel.com
We already maintain a job count for each exec queue, so simplify the idle
check to rely on the job count rather than the LRC state. This decouples
exec queues from LRC-based backends and avoids unnecessarily coupling idle
detection to backend-specific implementation details.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-5-matthew.brost@intel.com
Add the xe_migrate_update_pgtables_cpu_execute helper, which performs
the CPU-side page-table update. This will support implementing CPU
binds, as the submission backend can call this helper once a bind job’s
dependencies are resolved. While here, add assertions to provide basic
sanity checks on tht function arguments.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-4-matthew.brost@intel.com
Remove the xe_migrate_pt_update argument from the populate and clear
vfuns. This structure will not be available in run_job, where CPU binds
will be implemented. The populate path no longer needs it, and the clear
path already uses the VM field instead.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-3-matthew.brost@intel.com
PT BOs share the VM's dma-resv and therefore must keep the VM alive
until all PT BOs are destroyed.
Take a VM reference from PT BOs to ensure the shared dma-resv remains
valid for the lifetime of the BOs. Scope the change to PT BOs rather
than all kernel BOs to minimize risk and avoid extending VM lifetimes
unnecessarily.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-2-matthew.brost@intel.com
'|' binds tighter than the conditional operator, so the trailing
"?: 0" tests the whole OR expression instead of just the TLB flag.
MI_FLUSH_DW is non-zero, so the condition is always true and the
expression evaluates to itself.
Drop the ternary. The mask already yields 0 when MI_INVALIDATE_TLB
is not set.
No functional change.
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Assisted-by: LLM
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260923171527.663537-1-shuicheng.lin@intel.com
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
The NULL check on madvise_funcs[attr_type] in xe_vm_madvise_ioctl()
is unreachable. madvise_args_are_sane() already validates args->type
via a switch statement whose default case rejects any type not present
in madvise_funcs[]. Every type that passes the sanity check has a
corresponding entry in the table.
Beyond being dead code, the check's error path (goto err_fini) would
also skip xe_svm_notifier_unlock() if it were somehow reached with
has_svm_userptr_vmas set, since err_fini only tears down the drm_exec.
Removing the check eliminates both the redundancy and the latent
unwind hazard.
No functional change.
Assisted-by: LLM
Reviewed-by: Xin Wang <x.wang@intel.com>
Link: https://patch.msgid.link/20260915011229.203424-1-shuicheng.lin@intel.com
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
This reverts commit 23ea2c7572.
A recent bspec update dropped the details about CCS_MODE being a masked
register (i.e., bits [15:0] are only updated by register writes when the
corresponding bit in [31:16] mask is also set by the write) on Xe2 and
later. Apparently the documented mask bits were never actually
implemented in hardware, so all register writes will update all bits of
[15:0], just as the register behaved on Xe1 platforms.
Setting the non-existent mask bits in Xe didn't have any negative impact
since none of our existing code was trying to use the mask to limit
which bits were updated. However this could have caused problems in the
future if we ever rework this code in a way that tried to rely on the
masking behavior. Update the register definition and drop the use of
the mask to align with the new bspec definition and avoid potential
mistakes in the future.
Bspec: 55887
Cc: Balasubramani Vivekanandan <balasubramani.vivekanandan@intel.com>
Reviewed-by: Daniel Charles <daniel.charles@intel.com>
Link: https://patch.msgid.link/20260924-ccs_mode_drop_mask-v1-1-5b2b35708b83@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
On platforms with a standalone media GT and media version 13.00
(MTL/ARL), memory shared between the CPU and the media GT's GuC must
not be mapped cached on the CPU side: the CPU can otherwise read stale
cache lines for data the GuC has already written.
i915 implements this as Wa_22016122933 (see
intel_gt_needs_wa_22016122933(), used by intel_guc_allocate_vma() and
intel_gt_coherent_map_type()); xe never inherited it.
The visible symptom on ARL is TLB invalidation acks stalling for a
near-constant ~2.3s: the GuC writes the G2H ack in time, but the CPU
keeps reading a stale (empty) view of the G2H CTB until the line is
naturally evicted, so the fence timeout at 2.25s fires first. GuC log
decode confirmed all invalidations were handled promptly by the
firmware, and only the media GT was affected. See Link for the full
investigation (three machines affected: 7d51, 7dd1, Arc Pro 130T).
Making the mapping coherent instead of uncached does not help: with a
GGTT PAT entry repurposed to WB|COH_2WAY the driver comes up and the
media GT GuC runs, but the stalls remain (request-to-ack 2290ms).
Uncached really is required here. Note that 2-way coherency is not
normally reachable from a GGTT PTE (only 2 PAT bits), so that
experiment needed a modified PAT table and may not reflect a supported
configuration.
Add the OOB workaround scoped like i915 (media version 13.00, media GT
only - MEDIA_VERSION() OOB rules only match the media GT on standalone
media platforms) and apply XE_BO_FLAG_NEEDS_UC to the GuC-shared
allocations the CPU reads from: the CTBs, the GuC log, ADS, the SLPC
shared data and the engine activity buffers. hwconfig and the G2G
buffer are allocated on the primary GT only, where the workaround does
not apply.
Note that XE_BO_FLAG_NEEDS_UC drives both the CPU mapping (uncached)
and the GGTT cache mode (XE_CACHE_NONE instead of XE_CACHE_WB), which is
stricter than i915: i915 documents the workaround as WC on the CPU side
and UC on the GPU side. A CPU-WC variant with the GGTT side kept at
XE_CACHE_NONE was measured to be equally effective, but expressing it
would require either a new BO flag or decoupling the GGTT cache-mode
selection from XE_BO_FLAG_NEEDS_UC - simply swapping in
XE_BO_FLAG_FORCE_WC silently relaxes the GPU side back to WB and the
stalls return. The stricter mapping is kept here since it is the tested
configuration and no throughput difference between the two was
measurable; the extra plumbing can be added later if parity with i915 is
preferred.
Scope is limited to the GuC-shared allocations, which is where the
failures were observed. i915 additionally covers media-GT LRC/ring
state; extending xe to match can be done as a follow-up if wanted.
Validation on two ARL machines (7d51 and 7dd1): before, 20-60 TLB
invalidation ack stalls per day, every day, for weeks, on every kernel
and on two GuC firmware versions (70.53.0 and 70.72.1). After: zero
stalls in six weeks of combined runtime, over 10M TLB invalidations
processed under the same workloads, across kernels 7.1.6, 7.1.8 and
7.2. The 7dd1 machine, which could not survive a day of media
workloads on xe without a platform freeze, has been running xe full
time since 11 August, including days with heavy video transcoding,
with zero stalls and zero freezes.
The coherency experiment, the CPU-WC measurements and the engine
activity coverage gap were found by Navon John Lukose while A/B
testing v2 on an ARL 7d51.
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Tested-by: Navon John Lukose <navonjohnlukose@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-4-talesam@gmail.com
When a TLB invalidation fence times out we log the timeout, but if the
ack for that seqno later shows up there is no record of it, making it
impossible to tell from logs whether the ack was lost forever or merely
(very) late.
Track the most recent timed out seqno and log how late its ack arrives,
relative to both the original request and the moment the fence was
signaled with -ETIME.
On ARL with GuC 70.53.0 this shows the acks are never lost: they
consistently arrive ~2.3s after the request, tens of milliseconds after
the TDR has already signaled the fence:
TLB invalidation fence timeout, seqno=10992 recv=10991
TLB invalidation late ack: seqno=10992 recv=10992,
request-to-ack=2314ms, timeout-to-ack=45ms
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-3-talesam@gmail.com
TLB invalidation timeouts currently leave no record of the firmware
state behind: there is no exec queue or job to blame, so nothing calls
xe_devcoredump() and the GuC log content at the time of the hang is
lost.
Add xe_devcoredump_gt(), a variant of xe_devcoredump() for hangs that
are not tied to an exec queue or job. It captures the GuC log and CT
state of the affected GT, reusing the existing snapshot machinery and
the "only first snapshot" policy, and hook it up to the TLB invalidation
timeout path.
This was instrumental in diagnosing GuC TLB invalidation ack stalls on
ARL (see Link), where the invalidation request is consumed from the H2G
CTB immediately but the ack G2H only arrives ~2.3s later, after the
timeout has already fired.
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-2-talesam@gmail.com
XE_GT0 and XE_GT1 are old macros that were used for hardcoding specific
GTs during early development of the driver; neither should be anymore
since the driver shouldn't be making assumptions about what ID numbers
correspond to specific types of IDs.
* XE_GT0 was only being used in the GuC init code to skip reading the
hwconfig table on more than one GT. Comparing the GT against
xe_root_mmio_gt(xe) is a more direct way to ensure this happens on
just a single GT per device (and should work even if the ID numbers
wind up changing in the future).
* XE_GT1 was only being used to set XE_MAX_TILES_PER_DEVICE which makes
no sense at all; even though the highest tile count we have on any
device today (2 on PVC) happens to match the number of GTs that we
can potentially have inside a tile (2: primary+media on non-PVC
platforms), these counts are unrelated and should not be tied
together like this.
Cleanup the places where these were used and drop the two defines so
that they don't get mis-used anywhere else going forward.
Reviewed-by: Dnyaneshwar Bhadane <dnyaneshwar.bhadane@intel.com>
Link: https://patch.msgid.link/20260923-drop_old_gt_defines-v1-1-8f0811238582@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
xe_sysctrl_send_command() currently uses a fixed mailbox response
timeout of %XE_SYSCTRL_MB_DEFAULT_TIMEOUT_MS for all commands.
Add a timeout_ms field to struct xe_sysctrl_mailbox_command to allow
callers to override the default timeout when needed. A value of 0
preserves the existing default behavior.
Signed-off-by: Anoop Vijay <anoop.c.vijay@intel.com>
Reviewed-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Signed-off-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Link: https://patch.msgid.link/20260922111416.94416-2-anoop.c.vijay@intel.com
Reference-drop trace in drm_pagemap_dev_unhold_work() was changed
from drm_dbg() to drm_dbg_ratelimited() to reduce serial console
bottlenecks.
However, when the ratelimit kicks in, the kernel's ratelimiting subsystem
logs a "callbacks suppressed" notification at the KERN_WARNING level.
Revert the trace macro back to drm_dbg()
Fixes: b66b036c95 ("drm/pagemap: Prevent CPU stalls during unbounded pagemap teardown")
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Maxime Ripard <mripard@kernel.org> # For merging through drm-xe only. (IRC)
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patch.msgid.link/20260921062648.1166903-2-varun.gupta@intel.com
