Original commit 0f485f8865 ("drm/xe/i2c: use device_create_managed_software_node()")
got silently removed by commit cc46796964 ("Merge drm/drm-next into drm-xe-next").
Re-apply it.
Signed-off-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
Link: https://patch.msgid.link/20260922-xe-i2c-reapply-swnode-unreg-v1-1-1bdda5d44701@oss.qualcomm.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
[Rodrigo: Modified commit message to reflect the right history]
Add a kernel-doc DOC section at the top of xe_migrate.c describing the
Ultra Low Latency Submission (ULLS) scheme used for migration jobs.
Cover the motivation (removing the H2G / GuC / context switch latency
from the page fault and SVM prefetch critical paths), the platform
requirements, the LRC PPHWSP semaphore layout and its relationship to
the migration queue job count, the fixed ULLS job size and why it is
needed, the ring preamble / postamble emitted by the ring ops including
the in-ring tail update, the semaphore-only submission fast path in the
GuC backend, and the enter / delayed exit flow along with the ULLS job
flags.
Hook the new section into Documentation/gpu/xe/xe_migrate.rst.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Assisted-by: Github-Copilot:Claude-opus-5
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-25-matthew.brost@intel.com
Add a configfs attribute to control the migrate ULLS period.
The migrate ULLS period specifies how long, in milliseconds, migrate
ULLS remains active before exiting. Migrate ULLS is currently used by
page-fault handling and SVM prefetch operations to process migration
work through a continuously running batch.
The attribute defaults to 5 ms. Setting the value to 0 disables ULLS.
This knob is primarily intended for performance tuning and evaluation,
allowing experimentation with the trade-off between reducing ULLS
startup/teardown overhead and limiting the time spent in a continuously
running migration workload.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-24-matthew.brost@intel.com
Call xe_migrate_ulls_enter upon page fault or SVM prefetch in an
effort speed up these critical paths.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-23-matthew.brost@intel.com
Add ULLS migration job support to GuC submission backend.
Changes required:
- On migration queue, reduce max jobs to the number of ULLS semaphores
minus one
- Skip writing the saved ring tail for ULLS jobs except for the first
ULLS job - the ring updates both the saved tail and the tail register
itself
- Set ULLS sempahore for current job releasing last job except for first
ULLS job
- Suppress submit H2G for ULLS except for first ULLS job
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-22-matthew.brost@intel.com
Add preamble and postamble for ULLS migrations jobs. Preamble clears
current semaphore for reuse. Postamble waits on next semaphore which is
set upon next job submission, then advances the ring tail over that job
with an LRI to RING_TAIL, so submitting it costs the CPU nothing beyond
signalling the semaphore.
A job updates the tail on behalf of a successor which has not been
emitted yet, so it cannot know how much ring that successor will occupy.
Pad every ULLS job out to a fixed ULLS_JOB_SIZE_BYTES, which makes the
next tail derivable from where the current job starts. The pad also
supplies the NOPs which must follow an in-ring tail update.
The last ULLS migration job skips BB submission, the postamble and the
tail update (clear current semaphore, write seqno, exit ULLS), padding
the difference so that it still fills a job slot.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-21-matthew.brost@intel.com
Add function to enter ULLS mode for migration job and delayed worker to
exit (power saving). ULLS mode expected to entered upon page fault or
SVM prefetch. ULLS mode exit delay is currently set to 5ms.
ULLS mode only support on DGFX and USM platforms where a hardware engine
is reserved for migrations jobs. When in ULLS mode, set several flags on
migration jobs so submission backend / ring ops can properly submit in
ULLS mode.
Upon ULLS mode enter, send a job trigger waiting a semphore pipling
initial GuC / HW conetxt switch.
Upon ULLS mode exit, send a job to trigger that current ULLS
semaphore so the ring can be taken off the hardware.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-20-matthew.brost@intel.com
Define memory layout for ULLS semaphores stored in LRC memory. Add
support functions to return GGTT address and set semaphore based on a
job's seqno.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-19-matthew.brost@intel.com
Some multi-tile devices may want to mirror page tables across tiles for
memory-bandwidth reasons, while others may not. Add a device flag that
allows enabling or disabling page-table mirroring across tiles.
Setting the flag to true (the existing behavior) on PVC, but both modes
have been tested and are working on PVC.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-18-matthew.brost@intel.com
With CPU binds, it no longer makes sense to implement CPU bind handling
in the migrate layer, as these operations are entirely decoupled from
hardware. Introduce a dedicated CPU bind layer stored at the device
level.
Since CPU binds are tile-independent, update the PT layer to generate a
single bind job even when pages are mirrored across tiles.
This patch is large because the refactor touches multiple file / layers
and ensures functional equivalence before and after the change.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-17-matthew.brost@intel.com
Since bind jobs execute on the CPU rather than the GPU, maintaining a
per-tile bind queue no longer provides value. Convert the driver to use
a single bind queue shared across tiles. The primary change is routing
all GT TLB invalidations through this unified bind queue.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-16-matthew.brost@intel.com
Both populate and clear have unused void* ptr arguments, remove these.
Also consolidate ofs, num_qwords in update argument.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-15-matthew.brost@intel.com
No reason to use the GPU for binds.
Benefits of CPU-based binds:
- Lower latency once dependencies are resolved, as there is no
interaction with the GuC or a hardware context switch both of which
are relatively slow.
- Large arrays of binds do not risk running out of migration PTEs,
avoiding -ENOBUFS being returned to userspace.
- Kernel binds are decoupled from the migration exec queue (which issues
copies and clears), so they cannot get stuck behind unrelated
jobs—this can be a problem with parallel GPU faults.
- Paves the for path decouping binds from tiles and individual engines
- Enables ULLS on the migration exec queue, as this queue has exclusive
access to the paging copy engine.
Update migration layer to formulate a PT job which will issue CPU bind
in the submission backend.
All code related to GPU-based binding has been removed.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-14-matthew.brost@intel.com
Now that the CPU is always used for binds even in jobs, CPU bind jobs
can pass GPU jobs in the same exec queue resulting dma-fences signaling
out-of-order. Use a dedicated exec queue for binds issued from page
faults to avoid ordering issues and avoid blocking kernel binds on
unrelated copies / clears.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-13-matthew.brost@intel.com
The level was previously extracted from struct xe_pt inside
xe_vm_pgtable_update during CPU binds, which always occurred during the
bind IOCTL. With CPU binds now supported in bind jobs, struct xe_pt may
no longer be valid in memory at that point. To address this, store the
level directly in struct xe_vm_pgtable_update.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-12-matthew.brost@intel.com
PT jobs bypass GPU execution for the final step of a bind job, using the
CPU to program the required page tables. Teach the GuC submission backend
how to execute these jobs.
PT job submission is implemented in the GuC backend for simplicity. A
follow-up patch could introduce a dedicated backend for PT jobs.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-11-matthew.brost@intel.com
Add struct xe_pt_job_ops, a dynamically refcounted object that contains
the information required to issue a CPU bind via a job after the initial
bind IOCTL returns.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-10-matthew.brost@intel.com
Add helpers to access PT ops, making it easier to shuffle the location of
the ops structures without requiring widespread code changes.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-9-matthew.brost@intel.com
Update the scheduler job layer to support PT jobs. PT jobs are executed
entirely on the CPU and do not require LRC fences or a batch address.
Repurpose the LRC fence storage to hold PT‑job arguments and update the
scheduler job layer to distinguish between PT jobs and jobs that require
an LRC.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-8-matthew.brost@intel.com
Update the xe_bo_put_deferred arguments to include a writeback flag,
which indicates whether the BO was added to the deferred list. This is
useful when the caller needs to take additional actions after the BO has
been queued for deferred release.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-7-matthew.brost@intel.com
Add the job count to the GuC exec queue snapshot, as this is useful
debug information.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-6-matthew.brost@intel.com
We already maintain a job count for each exec queue, so simplify the idle
check to rely on the job count rather than the LRC state. This decouples
exec queues from LRC-based backends and avoids unnecessarily coupling idle
detection to backend-specific implementation details.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-5-matthew.brost@intel.com
Add the xe_migrate_update_pgtables_cpu_execute helper, which performs
the CPU-side page-table update. This will support implementing CPU
binds, as the submission backend can call this helper once a bind job’s
dependencies are resolved. While here, add assertions to provide basic
sanity checks on tht function arguments.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-4-matthew.brost@intel.com
Remove the xe_migrate_pt_update argument from the populate and clear
vfuns. This structure will not be available in run_job, where CPU binds
will be implemented. The populate path no longer needs it, and the clear
path already uses the VM field instead.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Link: https://patch.msgid.link/20260925201526.1355279-3-matthew.brost@intel.com
PT BOs share the VM's dma-resv and therefore must keep the VM alive
until all PT BOs are destroyed.
Take a VM reference from PT BOs to ensure the shared dma-resv remains
valid for the lifetime of the BOs. Scope the change to PT BOs rather
than all kernel BOs to minimize risk and avoid extending VM lifetimes
unnecessarily.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260925201526.1355279-2-matthew.brost@intel.com
'|' binds tighter than the conditional operator, so the trailing
"?: 0" tests the whole OR expression instead of just the TLB flag.
MI_FLUSH_DW is non-zero, so the condition is always true and the
expression evaluates to itself.
Drop the ternary. The mask already yields 0 when MI_INVALIDATE_TLB
is not set.
No functional change.
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Assisted-by: LLM
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260923171527.663537-1-shuicheng.lin@intel.com
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
The NULL check on madvise_funcs[attr_type] in xe_vm_madvise_ioctl()
is unreachable. madvise_args_are_sane() already validates args->type
via a switch statement whose default case rejects any type not present
in madvise_funcs[]. Every type that passes the sanity check has a
corresponding entry in the table.
Beyond being dead code, the check's error path (goto err_fini) would
also skip xe_svm_notifier_unlock() if it were somehow reached with
has_svm_userptr_vmas set, since err_fini only tears down the drm_exec.
Removing the check eliminates both the redundancy and the latent
unwind hazard.
No functional change.
Assisted-by: LLM
Reviewed-by: Xin Wang <x.wang@intel.com>
Link: https://patch.msgid.link/20260915011229.203424-1-shuicheng.lin@intel.com
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
This reverts commit 23ea2c7572.
A recent bspec update dropped the details about CCS_MODE being a masked
register (i.e., bits [15:0] are only updated by register writes when the
corresponding bit in [31:16] mask is also set by the write) on Xe2 and
later. Apparently the documented mask bits were never actually
implemented in hardware, so all register writes will update all bits of
[15:0], just as the register behaved on Xe1 platforms.
Setting the non-existent mask bits in Xe didn't have any negative impact
since none of our existing code was trying to use the mask to limit
which bits were updated. However this could have caused problems in the
future if we ever rework this code in a way that tried to rely on the
masking behavior. Update the register definition and drop the use of
the mask to align with the new bspec definition and avoid potential
mistakes in the future.
Bspec: 55887
Cc: Balasubramani Vivekanandan <balasubramani.vivekanandan@intel.com>
Reviewed-by: Daniel Charles <daniel.charles@intel.com>
Link: https://patch.msgid.link/20260924-ccs_mode_drop_mask-v1-1-5b2b35708b83@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
On platforms with a standalone media GT and media version 13.00
(MTL/ARL), memory shared between the CPU and the media GT's GuC must
not be mapped cached on the CPU side: the CPU can otherwise read stale
cache lines for data the GuC has already written.
i915 implements this as Wa_22016122933 (see
intel_gt_needs_wa_22016122933(), used by intel_guc_allocate_vma() and
intel_gt_coherent_map_type()); xe never inherited it.
The visible symptom on ARL is TLB invalidation acks stalling for a
near-constant ~2.3s: the GuC writes the G2H ack in time, but the CPU
keeps reading a stale (empty) view of the G2H CTB until the line is
naturally evicted, so the fence timeout at 2.25s fires first. GuC log
decode confirmed all invalidations were handled promptly by the
firmware, and only the media GT was affected. See Link for the full
investigation (three machines affected: 7d51, 7dd1, Arc Pro 130T).
Making the mapping coherent instead of uncached does not help: with a
GGTT PAT entry repurposed to WB|COH_2WAY the driver comes up and the
media GT GuC runs, but the stalls remain (request-to-ack 2290ms).
Uncached really is required here. Note that 2-way coherency is not
normally reachable from a GGTT PTE (only 2 PAT bits), so that
experiment needed a modified PAT table and may not reflect a supported
configuration.
Add the OOB workaround scoped like i915 (media version 13.00, media GT
only - MEDIA_VERSION() OOB rules only match the media GT on standalone
media platforms) and apply XE_BO_FLAG_NEEDS_UC to the GuC-shared
allocations the CPU reads from: the CTBs, the GuC log, ADS, the SLPC
shared data and the engine activity buffers. hwconfig and the G2G
buffer are allocated on the primary GT only, where the workaround does
not apply.
Note that XE_BO_FLAG_NEEDS_UC drives both the CPU mapping (uncached)
and the GGTT cache mode (XE_CACHE_NONE instead of XE_CACHE_WB), which is
stricter than i915: i915 documents the workaround as WC on the CPU side
and UC on the GPU side. A CPU-WC variant with the GGTT side kept at
XE_CACHE_NONE was measured to be equally effective, but expressing it
would require either a new BO flag or decoupling the GGTT cache-mode
selection from XE_BO_FLAG_NEEDS_UC - simply swapping in
XE_BO_FLAG_FORCE_WC silently relaxes the GPU side back to WB and the
stalls return. The stricter mapping is kept here since it is the tested
configuration and no throughput difference between the two was
measurable; the extra plumbing can be added later if parity with i915 is
preferred.
Scope is limited to the GuC-shared allocations, which is where the
failures were observed. i915 additionally covers media-GT LRC/ring
state; extending xe to match can be done as a follow-up if wanted.
Validation on two ARL machines (7d51 and 7dd1): before, 20-60 TLB
invalidation ack stalls per day, every day, for weeks, on every kernel
and on two GuC firmware versions (70.53.0 and 70.72.1). After: zero
stalls in six weeks of combined runtime, over 10M TLB invalidations
processed under the same workloads, across kernels 7.1.6, 7.1.8 and
7.2. The 7dd1 machine, which could not survive a day of media
workloads on xe without a platform freeze, has been running xe full
time since 11 August, including days with heavy video transcoding,
with zero stalls and zero freezes.
The coherency experiment, the CPU-WC measurements and the engine
activity coverage gap were found by Navon John Lukose while A/B
testing v2 on an ARL 7d51.
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Tested-by: Navon John Lukose <navonjohnlukose@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-4-talesam@gmail.com
When a TLB invalidation fence times out we log the timeout, but if the
ack for that seqno later shows up there is no record of it, making it
impossible to tell from logs whether the ack was lost forever or merely
(very) late.
Track the most recent timed out seqno and log how late its ack arrives,
relative to both the original request and the moment the fence was
signaled with -ETIME.
On ARL with GuC 70.53.0 this shows the acks are never lost: they
consistently arrive ~2.3s after the request, tens of milliseconds after
the TDR has already signaled the fence:
TLB invalidation fence timeout, seqno=10992 recv=10991
TLB invalidation late ack: seqno=10992 recv=10992,
request-to-ack=2314ms, timeout-to-ack=45ms
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-3-talesam@gmail.com
TLB invalidation timeouts currently leave no record of the firmware
state behind: there is no exec queue or job to blame, so nothing calls
xe_devcoredump() and the GuC log content at the time of the hang is
lost.
Add xe_devcoredump_gt(), a variant of xe_devcoredump() for hangs that
are not tied to an exec queue or job. It captures the GuC log and CT
state of the affected GT, reusing the existing snapshot machinery and
the "only first snapshot" policy, and hook it up to the TLB invalidation
timeout path.
This was instrumental in diagnosing GuC TLB invalidation ack stalls on
ARL (see Link), where the invalidation request is consumed from the H2G
CTB immediately but the ack G2H only arrives ~2.3s later, after the
timeout has already fired.
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-2-talesam@gmail.com
XE_GT0 and XE_GT1 are old macros that were used for hardcoding specific
GTs during early development of the driver; neither should be anymore
since the driver shouldn't be making assumptions about what ID numbers
correspond to specific types of IDs.
* XE_GT0 was only being used in the GuC init code to skip reading the
hwconfig table on more than one GT. Comparing the GT against
xe_root_mmio_gt(xe) is a more direct way to ensure this happens on
just a single GT per device (and should work even if the ID numbers
wind up changing in the future).
* XE_GT1 was only being used to set XE_MAX_TILES_PER_DEVICE which makes
no sense at all; even though the highest tile count we have on any
device today (2 on PVC) happens to match the number of GTs that we
can potentially have inside a tile (2: primary+media on non-PVC
platforms), these counts are unrelated and should not be tied
together like this.
Cleanup the places where these were used and drop the two defines so
that they don't get mis-used anywhere else going forward.
Reviewed-by: Dnyaneshwar Bhadane <dnyaneshwar.bhadane@intel.com>
Link: https://patch.msgid.link/20260923-drop_old_gt_defines-v1-1-8f0811238582@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
xe_sysctrl_send_command() currently uses a fixed mailbox response
timeout of %XE_SYSCTRL_MB_DEFAULT_TIMEOUT_MS for all commands.
Add a timeout_ms field to struct xe_sysctrl_mailbox_command to allow
callers to override the default timeout when needed. A value of 0
preserves the existing default behavior.
Signed-off-by: Anoop Vijay <anoop.c.vijay@intel.com>
Reviewed-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Signed-off-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Link: https://patch.msgid.link/20260922111416.94416-2-anoop.c.vijay@intel.com
Reference-drop trace in drm_pagemap_dev_unhold_work() was changed
from drm_dbg() to drm_dbg_ratelimited() to reduce serial console
bottlenecks.
However, when the ratelimit kicks in, the kernel's ratelimiting subsystem
logs a "callbacks suppressed" notification at the KERN_WARNING level.
Revert the trace macro back to drm_dbg()
Fixes: b66b036c95 ("drm/pagemap: Prevent CPU stalls during unbounded pagemap teardown")
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Maxime Ripard <mripard@kernel.org> # For merging through drm-xe only. (IRC)
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patch.msgid.link/20260921062648.1166903-2-varun.gupta@intel.com
When VFs are enabled on dGFX the driver resizes the PF VF_LMEM_BAR to
fit the requested layout. After VFs are disabled the PF VF BAR
size is left as-is. On platforms with tight MMIO apertures a
subsequent unplug/rescan followed by another enable may fail with:
"VF BAR …: can't assign; no space"
because the PCI core reserves address space based on the (now large) VF
template, often multiplied by totalvfs.
Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5937
Fixes: 94eae6ee4c ("drm/xe/pf: Set VF LMEM BAR size")
Signed-off-by: Marcin Bernatowicz <marcin.bernatowicz@linux.intel.com>
Cc: Michał Wajdeczko <michal.wajdeczko@intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260918110130.700332-1-marcin.bernatowicz@linux.intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Avoid programming the IDLEDLY timer to less than 5 microseconds.
Apply wa_14025941587 to Graphics Versions 20.01 to 35.11
and Media Versions 13.01 to 35.03
v2: Use xe_rtp_match_not_sriov_vf, move to local variable
Remove warn and other knits - Matt R
v3: Add verbose comment - Tejas
v4: Restore IDLE_DLY register on engine reset.
Add it to GUC save-restore list. -Vivek
v5: Extend WA to Media Versions 13.01 to 35.03 - Vinay
v6: Avoid clearing inhibit switch - Bala
Refactor code accordingly by adding idle_reg_val.
v7: Rebased with the divide-by-zero/overflow guards living in
a separate hardening patch.
v8: Preserve the Wa_16023105232 floor (DIV_ROUND_DOWN_ULL) and the
maxcnt == 0 guard from the hardening patch. Round up
(DIV_ROUND_UP_ULL) the Wa_14025941587 minimum conversion instead,
so the tick-quantized delay cannot round back below 5 us.
v9: Evaluate the Wa_16023105232 xe_gt_WARN_ON() against the value
read from hardware instead of the Wa_14025941587-bumped value,
so it no longer fires on the driver's own floor. Re-check the
rounded-up tick value against maxcnt and floor it if tick
quantization pushed it back to/above maxcnt, logging via
xe_gt_dbg since this is the driver's own value, not a hardware
anomaly.
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Tangudu Tilak Tirumalesh <tilak.tirumalesh.tangudu@intel.com>
Reviewed-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Link: https://patch.msgid.link/20260916100545.779894-3-tilak.tirumalesh.tangudu@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
adjust_idledly() has several corner-case issues flagged during review:
1. If xe_gt_clock_init() failed to recognise the crystal clock,
gt->info.timestamp_base is 0, which makes idledly_units_ps also 0.
The subsequent DIV_ROUND_CLOSEST(..., idledly_units_ps) is then a
divide-by-zero and panics the kernel.
2. The tick-to-ns conversions are done in u32:
idledly * idledly_units_ps, (maxcnt - 1) * 1000
Both overflow u32 before DIV_ROUND_CLOSEST() sees them.
3. If IDLE_WAIT_TIME reads back as 0, maxcnt evaluates to 0 and
the maxcnt - 1 clamp wraps to 0xFFFFFFFF in u32.
4. The register only stores whole ticks, so the clamped ns value has
to be converted to ticks and back. DIV_ROUND_CLOSEST() can round
that conversion up past maxcnt:
maxcnt = 640 ns, one tick = 666664 ps
clamp: maxcnt - 1 = 639 ns
ns -> ticks: 639000 / 666664 = 0.958 -> rounds to 1 tick
tick -> ns: 1 * 666664 / 1000 = 667 ns
667 ns is programmed into RING_IDLEDLY, but 667 >= maxcnt (640),
so xe_gt_WARN_ON() fires again on every subsequent init.
Return early if timestamp_base is 0 (the unknown-crystal path).
Do the conversions in u64 via the *_ULL() helpers so they cannot wrap.
Clamp with a floor (DIV_ROUND_DOWN_ULL) so the programmed delay stays
strictly below maxcnt, and guard the maxcnt == 0 case with a zero delay
while still writing RING_IDLEDLY so INHIBIT_SWITCH_UNTIL_PREEMPTED is
cleared.
v2: Drop the redundant warn on the timestamp_base == 0 path;
xe_gt_clock_init() already warns on an unrecognised crystal clock.
Keep the early return to avoid the divide-by-zero. - Vinay
v3: Field-mask the RING_IDLEDLY write with REG_FIELD_PREP(IDLE_DELAY, ...)
instead of writing the raw tick count, which could clobber
INHIBIT_SWITCH_UNTIL_PREEMPTED and reserved bits. Split the
inhibit-switch clear from the maxcnt clamp so a set inhibit bit no
longer forces a needless delay overwrite when the delay itself is
already valid. Use gt_to_xe(gt) instead of gt_to_xe(hwe->gt).
Fixes: d2de4410a8 ("drm/xe: Apply Wa_16023105232")
Cc: stable@vger.kernel.org
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Tangudu Tilak Tirumalesh <tilak.tirumalesh.tangudu@intel.com>
Reviewed-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Link: https://patch.msgid.link/20260916100545.779894-2-tilak.tirumalesh.tangudu@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
We should avoid taking the mutex in the helpers. Move guard from
the helper to the main entry function.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Link: https://patch.msgid.link/20260916212949.13120-1-michal.wajdeczko@intel.com
In xe_vm_close_and_put(), external-BO VMAs are queued on the contested
list for deferred destruction via xe_vma_destroy_unlocked(). However,
xe_vm_pt_destroy() was previously invoked before processing contested
VMAs, destroying vm->pt_root while those VMAs were still linked to their
respective buffer objects (vm_bo->list.gpuva).
If a concurrent thread evicts one of those shared buffer objects,
xe_bo_trigger_rebind() holding only bo->resv walks the BO's VMAs and, in
fault mode, calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(). Because
vm->pt_root[tile->id] is already NULL, dereferencing pt->level causes a
NULL ptr deref.
Fix this by deferring xe_vm_free_scratch() and xe_vm_pt_destroy() until
after all contested VMAs have been unlinked and destroyed.
User is reporting hitting a NULL ptr deref in xe_pt_zap_ptes(), which
could be explained by this race.
Assisted-by: LLM
Fixes: b06d47be7c ("drm/xe: Port Xe to GPUVA")
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/9290
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: <stable@vger.kernel.org> # v6.12+
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260918131034.598078-2-matthew.auld@intel.com
Ensure GT C6 state before D3 transition in runtime suspend flow.
This avoids the scenario where GT is being accessed after D3 state
is programmed. If the check for C6 fails, resume the device so it is
not stuck in an intermediate state.
v2: Force resume when we cancel suspend due to C6 check (Sashiko)
v3: Abort the suspend properly when idle check fails (Sashiko)
v4: Only check for C6 during suspend (Rodrigo). Have check in
xe_pci_runtime_suspend() instead of xe_pci_suspend().
v5: Don't checkfor PVC (Sashiko), rename function from_device_c6
to _device_idle
v6: Rename to _wait_all_c6 and move PVC check to gt_idle (Rodrigo)
Cc: Badal Nilawar <badal.nilawar@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260916190614.3047807-1-vinay.belgaumkar@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Drop the definition of MI_SEMAPHORE_WAIT_TOKEN and replace its uses with
MI_SEMAPHORE_WAIT. Both #define's were representing the same opcode
0x1C and this GPU instruction has always been named "MI_SEMAPHORE_WAIT"
in the official specs.
The "*_TOKEN" definition appears to have been copied from i915's code.
i915 needed two separate definitions to reflect the slightly different
instruction lengths gen8-gen11 platforms (4 dwords) vs gen12 and later
platforms (5 dwords, with the extra dword representing a token), but Xe
only supports newer platforms with the 5-dword form, so only a single
definition is needed and should match the official name given in the
specs.
Bspec: 14054, 45749, 60244
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260917-semaphore_inst-v1-1-61e49405f71d@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
pci_p2pdma_distance() can optionally print details about how it
computed the ACS/PCIe topology distance between the two devices, but
xe_dma_buf_attach() always called it with verbose set to false,
silently hiding useful diagnostics whenever a dma-buf attach unset
attach->peer2peer.
Tie the verbose argument to a dedicated dynamic-debug callsite tagged
"xe dma-buf p2p debug" (in addition to keeping the existing
DRM_UT_DRIVER category as a default), so the extra p2pdma distance
logging can be switched on without a recompile in any of the following
ways:
- Enable the whole DRIVER category for xe via drm.debug, which is
already used elsewhere in the driver:
# echo 0x10 > /sys/module/drm/parameters/debug
- Enable dynamic debug for the whole file:
# echo 'file xe_dma_buf.c +p' > \
/sys/kernel/debug/dynamic_debug/control
- Enable dynamic debug for just this callsite by function name:
# echo 'func xe_dma_buf_attach +p' > \
/sys/kernel/debug/dynamic_debug/control
- Enable dynamic debug by matching the format string used to
register the callsite:
# echo 'format "xe dma-buf p2p debug" +p' > \
/sys/kernel/debug/dynamic_debug/control
- Enable it at boot via the dyndbg= kernel command line parameter:
dyndbg="func xe_dma_buf_attach +p"
Any of the above flips verbose_p2p_dist to true for that attach call,
causing pci_p2pdma_distance() to log its distance calculation without
requiring the whole DRM_UT_DRIVER category to be enabled.
v2:
- Guard the _dynamic_func_call() invocation and
xe_ddebug_flip_bool() with a CONFIG_DYNAMIC_DEBUG check so the
build does not break when dynamic debug is disabled. (Sashiko-bot)
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Assisted-by: LLM
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Link: https://patch.msgid.link/20260914114617.112012-1-thomas.hellstrom@linux.intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
drm_pagemap_dev_unhold_work() processes an unbatched llist of pagemaps
on the system workqueue. This creates potential latency traps during
heavy teardown cycles due to two compounding factors:
1. The loop contains a per-item drm_dbg() trace. If dynamic debug is
enabled on a slow serial console, synchronous print latency scales
linearly with the batch size and can block worker execution.
2. The llist can accumulate large batches during intensive unmap
operations, monopolizing worker resources without yielding.
This causes below log warning:
"workqueue: drm_pagemap_dev_unhold_work [drm_gpusvm_helper] hogged CPU
for >10000us 19 times, consider switching to WQ_UNBOUND"
Fix this by implementing a latency mitigation strategy:
- Move the teardown work to system_unbound_wq to avoid tying up
per-CPU bound worker pools.
- Add cond_resched() inside the teardown loop to yield the CPU during
large batch teardowns, ensuring system responsiveness.
- Convert the drm_dbg() trace to drm_dbg_ratelimited() to mitigate
serial console bottlenecks while preserving debuggability.
v2:
- Changed workqueue from deprecated system_unbound_wq to system_dfl_wq.
(Arvind Yadav)
Fixes: a26084328a ("drm/pagemap, drm/xe: Manage drm_pagemap provider lifetimes")
Reviewed-by: Arvind Yadav <arvind.yadav@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Link: https://patch.msgid.link/20260918081320.30389-2-varun.gupta@intel.com
Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
The desired behavior for SVM eviction failures, which can occur due to
various uncontrollable races, is for TTM to continue walking the LRU
list and look for another eviction candidate. This is expressed by
returning -ENOSPC from the ->move() callback.
Adjust the SVM eviction failure path because of races in ->move() to
return -ENOSPC so that TTM continues searching for another buffer to
evict.
Fixes: 3ca608dc75 ("drm/xe: Basic SVM BO eviction")
Cc: stable@vger.kernel.org
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patch.msgid.link/20260917203158.292823-1-matthew.brost@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
xe_pagefault_print() was firing without indicating why servicing of a
fault failed, making it hard to distinguish benign races (e.g. a VM's
file descriptor closing mid-fault) from real bugs.
Add enum xe_pagefault_error, uniquely identifying the high-level point
at which xe_pagefault_service(), xe_pagefault_handle_vma(), and
xe_svm_handle_pagefault() fail. Since consumer.page_addr is always
page (4K) aligned, steal the reserved low byte to carry this code
without growing struct xe_pagefault. Add xe_pagefault_set_error() /
xe_pagefault_get_error() to encode/decode it, and xe_pagefault_addr()
as the single accessor for the real (masked) faulted address; update
all real address consumers (xe_pagefault_match(), cache alignment,
xe_vm_add_fault_entry_pf()) to go through it instead of reading
consumer.page_addr directly. xe_pagefault_set_start_addr() now
preserves any already-recorded error bits rather than clobbering them.
xe_pagefault_print() decodes the error via a new
xe_pagefault_error_to_str() and reports it in an added "Error:" line.
xe_pagefault_asid_to_vm() now distinguishes an ASID with no VM at all
(XE_PAGEFAULT_ERROR_VM_NOT_FOUND, e.g. the owning file descriptor was
already closed) from a VM found but not in fault mode
(XE_PAGEFAULT_ERROR_VM_NOT_IN_FAULT_MODE).
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Assisted-by: GitHub_Copilot:claude-sonnet-5
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patch.msgid.link/20260917203850.321385-1-matthew.brost@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
A TLB invalidation issued on a wedged device fails with
-ENOTRECOVERABLE. xe_tlb_inval_issue() squashes only -ECANCELED, so the
error reaches callers that treat it as unexpected and WARN, tainting the
kernel on a wedge that was deliberately caused:
ggtt_invalidate_gt_tlb() drivers/gpu/drm/xe/xe_ggtt.c
xe_svm_invalidate() drivers/gpu/drm/xe/xe_svm.c
xe_bo_trigger_rebind() drivers/gpu/drm/xe/xe_bo.c
xe_vma_userptr_do_inval() drivers/gpu/drm/xe/xe_userptr.c
igt@xe_exec_reset@gt-reset-fault-injection hits the GGTT one, turning an
otherwise passing run into an abort:
*ERROR* SIGID=102 FATAL (-EIO) WEDGED: Device declared wedged!
Tile0: GT1: Failed to invalidate GGTT (-ENOTRECOVERABLE)
WARNING: drivers/gpu/drm/xe/xe_ggtt.c:588 at ggtt_invalidate_gt_tlb
Workqueue: xe-guc-destroy-wq __guc_exec_queue_destroy_async [xe]
ggtt_node_remove+0xe3/0x100 [xe]
xe_ggtt_remove_bo+0x89/0x2c0 [xe]
xe_ttm_bo_destroy+0xcb/0x330 [xe]
...
xe_lrc_destroy+0x74/0x90 [xe]
xe_exec_queue_fini+0x2d/0x60 [xe]
-ECANCELED and -ENOTRECOVERABLE mean the same thing at this layer: the
message was dropped rather than delivered, and the fence has already
been signalled before the error is returned, so there is nothing left to
wait for. Squash both.
A wedged device is only recovered by a fresh initialisation, so the
error return in xe_bo_trigger_rebind() becomes unreachable by design.
v2: fix all invalidation paths. (Sashiko)
Fixes: 50fa9acac2 ("drm/xe/guc: distinguish wedged from recoverable cancellation")
Assisted-by: Claude:claude-opus-5
Cc: Sk Anirban <sk.anirban@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260914215318.200603-1-shuicheng.lin@intel.com
CRI defines bit 5 of the perf limit reasons register as a power brake
(PWRBRK) indicator. Add PWRBRK_MASK and a reason_pwrbrk sysfs attribute
for CRI in place of reason_ratl.
Signed-off-by: Sk Anirban <sk.anirban@intel.com>
Fixes: 8578e6d054 ("drm/xe/gt_throttle: Drop individual show functions")
Reviewed-by: Raag Jadav <raag.jadav@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260909114931.1039331-2-sk.anirban@intel.com
Extend the existing "hw_engines" debugfs to iterate over all hardware
engines and dump their idle state: whether the engine pipe is idle or
busy, whether C6 entry is allowed, and (where available) the forcewake
request status.
v2: Use existing debugfs (Matt Brost)
v3: Rename macro (Vinay)
Signed-off-by: Sk Anirban <sk.anirban@intel.com>
Reviewed-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260910165855.1124574-2-sk.anirban@intel.com
Start using recently added KLV helpers for parsing migration
packet to properly detect any malformed KLV formats.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Link: https://patch.msgid.link/20260916215827.13488-4-michal.wajdeczko@intel.com
