This reverts commit 23ea2c7572.
A recent bspec update dropped the details about CCS_MODE being a masked
register (i.e., bits [15:0] are only updated by register writes when the
corresponding bit in [31:16] mask is also set by the write) on Xe2 and
later. Apparently the documented mask bits were never actually
implemented in hardware, so all register writes will update all bits of
[15:0], just as the register behaved on Xe1 platforms.
Setting the non-existent mask bits in Xe didn't have any negative impact
since none of our existing code was trying to use the mask to limit
which bits were updated. However this could have caused problems in the
future if we ever rework this code in a way that tried to rely on the
masking behavior. Update the register definition and drop the use of
the mask to align with the new bspec definition and avoid potential
mistakes in the future.
Bspec: 55887
Cc: Balasubramani Vivekanandan <balasubramani.vivekanandan@intel.com>
Reviewed-by: Daniel Charles <daniel.charles@intel.com>
Link: https://patch.msgid.link/20260924-ccs_mode_drop_mask-v1-1-5b2b35708b83@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
On platforms with a standalone media GT and media version 13.00
(MTL/ARL), memory shared between the CPU and the media GT's GuC must
not be mapped cached on the CPU side: the CPU can otherwise read stale
cache lines for data the GuC has already written.
i915 implements this as Wa_22016122933 (see
intel_gt_needs_wa_22016122933(), used by intel_guc_allocate_vma() and
intel_gt_coherent_map_type()); xe never inherited it.
The visible symptom on ARL is TLB invalidation acks stalling for a
near-constant ~2.3s: the GuC writes the G2H ack in time, but the CPU
keeps reading a stale (empty) view of the G2H CTB until the line is
naturally evicted, so the fence timeout at 2.25s fires first. GuC log
decode confirmed all invalidations were handled promptly by the
firmware, and only the media GT was affected. See Link for the full
investigation (three machines affected: 7d51, 7dd1, Arc Pro 130T).
Making the mapping coherent instead of uncached does not help: with a
GGTT PAT entry repurposed to WB|COH_2WAY the driver comes up and the
media GT GuC runs, but the stalls remain (request-to-ack 2290ms).
Uncached really is required here. Note that 2-way coherency is not
normally reachable from a GGTT PTE (only 2 PAT bits), so that
experiment needed a modified PAT table and may not reflect a supported
configuration.
Add the OOB workaround scoped like i915 (media version 13.00, media GT
only - MEDIA_VERSION() OOB rules only match the media GT on standalone
media platforms) and apply XE_BO_FLAG_NEEDS_UC to the GuC-shared
allocations the CPU reads from: the CTBs, the GuC log, ADS, the SLPC
shared data and the engine activity buffers. hwconfig and the G2G
buffer are allocated on the primary GT only, where the workaround does
not apply.
Note that XE_BO_FLAG_NEEDS_UC drives both the CPU mapping (uncached)
and the GGTT cache mode (XE_CACHE_NONE instead of XE_CACHE_WB), which is
stricter than i915: i915 documents the workaround as WC on the CPU side
and UC on the GPU side. A CPU-WC variant with the GGTT side kept at
XE_CACHE_NONE was measured to be equally effective, but expressing it
would require either a new BO flag or decoupling the GGTT cache-mode
selection from XE_BO_FLAG_NEEDS_UC - simply swapping in
XE_BO_FLAG_FORCE_WC silently relaxes the GPU side back to WB and the
stalls return. The stricter mapping is kept here since it is the tested
configuration and no throughput difference between the two was
measurable; the extra plumbing can be added later if parity with i915 is
preferred.
Scope is limited to the GuC-shared allocations, which is where the
failures were observed. i915 additionally covers media-GT LRC/ring
state; extending xe to match can be done as a follow-up if wanted.
Validation on two ARL machines (7d51 and 7dd1): before, 20-60 TLB
invalidation ack stalls per day, every day, for weeks, on every kernel
and on two GuC firmware versions (70.53.0 and 70.72.1). After: zero
stalls in six weeks of combined runtime, over 10M TLB invalidations
processed under the same workloads, across kernels 7.1.6, 7.1.8 and
7.2. The 7dd1 machine, which could not survive a day of media
workloads on xe without a platform freeze, has been running xe full
time since 11 August, including days with heavy video transcoding,
with zero stalls and zero freezes.
The coherency experiment, the CPU-WC measurements and the engine
activity coverage gap were found by Navon John Lukose while A/B
testing v2 on an ARL 7d51.
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Tested-by: Navon John Lukose <navonjohnlukose@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-4-talesam@gmail.com
When a TLB invalidation fence times out we log the timeout, but if the
ack for that seqno later shows up there is no record of it, making it
impossible to tell from logs whether the ack was lost forever or merely
(very) late.
Track the most recent timed out seqno and log how late its ack arrives,
relative to both the original request and the moment the fence was
signaled with -ETIME.
On ARL with GuC 70.53.0 this shows the acks are never lost: they
consistently arrive ~2.3s after the request, tens of milliseconds after
the TDR has already signaled the fence:
TLB invalidation fence timeout, seqno=10992 recv=10991
TLB invalidation late ack: seqno=10992 recv=10992,
request-to-ack=2314ms, timeout-to-ack=45ms
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-3-talesam@gmail.com
TLB invalidation timeouts currently leave no record of the firmware
state behind: there is no exec queue or job to blame, so nothing calls
xe_devcoredump() and the GuC log content at the time of the hang is
lost.
Add xe_devcoredump_gt(), a variant of xe_devcoredump() for hangs that
are not tied to an exec queue or job. It captures the GuC log and CT
state of the affected GT, reusing the existing snapshot machinery and
the "only first snapshot" policy, and hook it up to the TLB invalidation
timeout path.
This was instrumental in diagnosing GuC TLB invalidation ack stalls on
ARL (see Link), where the invalidation request is consumed from the H2G
CTB immediately but the ack G2H only arrives ~2.3s later, after the
timeout has already fired.
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Signed-off-by: Tales A. Mendonça <talesam@gmail.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260922144634.55130-2-talesam@gmail.com
XE_GT0 and XE_GT1 are old macros that were used for hardcoding specific
GTs during early development of the driver; neither should be anymore
since the driver shouldn't be making assumptions about what ID numbers
correspond to specific types of IDs.
* XE_GT0 was only being used in the GuC init code to skip reading the
hwconfig table on more than one GT. Comparing the GT against
xe_root_mmio_gt(xe) is a more direct way to ensure this happens on
just a single GT per device (and should work even if the ID numbers
wind up changing in the future).
* XE_GT1 was only being used to set XE_MAX_TILES_PER_DEVICE which makes
no sense at all; even though the highest tile count we have on any
device today (2 on PVC) happens to match the number of GTs that we
can potentially have inside a tile (2: primary+media on non-PVC
platforms), these counts are unrelated and should not be tied
together like this.
Cleanup the places where these were used and drop the two defines so
that they don't get mis-used anywhere else going forward.
Reviewed-by: Dnyaneshwar Bhadane <dnyaneshwar.bhadane@intel.com>
Link: https://patch.msgid.link/20260923-drop_old_gt_defines-v1-1-8f0811238582@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
xe_sysctrl_send_command() currently uses a fixed mailbox response
timeout of %XE_SYSCTRL_MB_DEFAULT_TIMEOUT_MS for all commands.
Add a timeout_ms field to struct xe_sysctrl_mailbox_command to allow
callers to override the default timeout when needed. A value of 0
preserves the existing default behavior.
Signed-off-by: Anoop Vijay <anoop.c.vijay@intel.com>
Reviewed-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Signed-off-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Link: https://patch.msgid.link/20260922111416.94416-2-anoop.c.vijay@intel.com
Reference-drop trace in drm_pagemap_dev_unhold_work() was changed
from drm_dbg() to drm_dbg_ratelimited() to reduce serial console
bottlenecks.
However, when the ratelimit kicks in, the kernel's ratelimiting subsystem
logs a "callbacks suppressed" notification at the KERN_WARNING level.
Revert the trace macro back to drm_dbg()
Fixes: b66b036c95 ("drm/pagemap: Prevent CPU stalls during unbounded pagemap teardown")
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Maxime Ripard <mripard@kernel.org> # For merging through drm-xe only. (IRC)
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patch.msgid.link/20260921062648.1166903-2-varun.gupta@intel.com
When VFs are enabled on dGFX the driver resizes the PF VF_LMEM_BAR to
fit the requested layout. After VFs are disabled the PF VF BAR
size is left as-is. On platforms with tight MMIO apertures a
subsequent unplug/rescan followed by another enable may fail with:
"VF BAR …: can't assign; no space"
because the PCI core reserves address space based on the (now large) VF
template, often multiplied by totalvfs.
Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5937
Fixes: 94eae6ee4c ("drm/xe/pf: Set VF LMEM BAR size")
Signed-off-by: Marcin Bernatowicz <marcin.bernatowicz@linux.intel.com>
Cc: Michał Wajdeczko <michal.wajdeczko@intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260918110130.700332-1-marcin.bernatowicz@linux.intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Avoid programming the IDLEDLY timer to less than 5 microseconds.
Apply wa_14025941587 to Graphics Versions 20.01 to 35.11
and Media Versions 13.01 to 35.03
v2: Use xe_rtp_match_not_sriov_vf, move to local variable
Remove warn and other knits - Matt R
v3: Add verbose comment - Tejas
v4: Restore IDLE_DLY register on engine reset.
Add it to GUC save-restore list. -Vivek
v5: Extend WA to Media Versions 13.01 to 35.03 - Vinay
v6: Avoid clearing inhibit switch - Bala
Refactor code accordingly by adding idle_reg_val.
v7: Rebased with the divide-by-zero/overflow guards living in
a separate hardening patch.
v8: Preserve the Wa_16023105232 floor (DIV_ROUND_DOWN_ULL) and the
maxcnt == 0 guard from the hardening patch. Round up
(DIV_ROUND_UP_ULL) the Wa_14025941587 minimum conversion instead,
so the tick-quantized delay cannot round back below 5 us.
v9: Evaluate the Wa_16023105232 xe_gt_WARN_ON() against the value
read from hardware instead of the Wa_14025941587-bumped value,
so it no longer fires on the driver's own floor. Re-check the
rounded-up tick value against maxcnt and floor it if tick
quantization pushed it back to/above maxcnt, logging via
xe_gt_dbg since this is the driver's own value, not a hardware
anomaly.
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Tangudu Tilak Tirumalesh <tilak.tirumalesh.tangudu@intel.com>
Reviewed-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Link: https://patch.msgid.link/20260916100545.779894-3-tilak.tirumalesh.tangudu@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
adjust_idledly() has several corner-case issues flagged during review:
1. If xe_gt_clock_init() failed to recognise the crystal clock,
gt->info.timestamp_base is 0, which makes idledly_units_ps also 0.
The subsequent DIV_ROUND_CLOSEST(..., idledly_units_ps) is then a
divide-by-zero and panics the kernel.
2. The tick-to-ns conversions are done in u32:
idledly * idledly_units_ps, (maxcnt - 1) * 1000
Both overflow u32 before DIV_ROUND_CLOSEST() sees them.
3. If IDLE_WAIT_TIME reads back as 0, maxcnt evaluates to 0 and
the maxcnt - 1 clamp wraps to 0xFFFFFFFF in u32.
4. The register only stores whole ticks, so the clamped ns value has
to be converted to ticks and back. DIV_ROUND_CLOSEST() can round
that conversion up past maxcnt:
maxcnt = 640 ns, one tick = 666664 ps
clamp: maxcnt - 1 = 639 ns
ns -> ticks: 639000 / 666664 = 0.958 -> rounds to 1 tick
tick -> ns: 1 * 666664 / 1000 = 667 ns
667 ns is programmed into RING_IDLEDLY, but 667 >= maxcnt (640),
so xe_gt_WARN_ON() fires again on every subsequent init.
Return early if timestamp_base is 0 (the unknown-crystal path).
Do the conversions in u64 via the *_ULL() helpers so they cannot wrap.
Clamp with a floor (DIV_ROUND_DOWN_ULL) so the programmed delay stays
strictly below maxcnt, and guard the maxcnt == 0 case with a zero delay
while still writing RING_IDLEDLY so INHIBIT_SWITCH_UNTIL_PREEMPTED is
cleared.
v2: Drop the redundant warn on the timestamp_base == 0 path;
xe_gt_clock_init() already warns on an unrecognised crystal clock.
Keep the early return to avoid the divide-by-zero. - Vinay
v3: Field-mask the RING_IDLEDLY write with REG_FIELD_PREP(IDLE_DELAY, ...)
instead of writing the raw tick count, which could clobber
INHIBIT_SWITCH_UNTIL_PREEMPTED and reserved bits. Split the
inhibit-switch clear from the maxcnt clamp so a set inhibit bit no
longer forces a needless delay overwrite when the delay itself is
already valid. Use gt_to_xe(gt) instead of gt_to_xe(hwe->gt).
Fixes: d2de4410a8 ("drm/xe: Apply Wa_16023105232")
Cc: stable@vger.kernel.org
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Tangudu Tilak Tirumalesh <tilak.tirumalesh.tangudu@intel.com>
Reviewed-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Link: https://patch.msgid.link/20260916100545.779894-2-tilak.tirumalesh.tangudu@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
We should avoid taking the mutex in the helpers. Move guard from
the helper to the main entry function.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Link: https://patch.msgid.link/20260916212949.13120-1-michal.wajdeczko@intel.com
In xe_vm_close_and_put(), external-BO VMAs are queued on the contested
list for deferred destruction via xe_vma_destroy_unlocked(). However,
xe_vm_pt_destroy() was previously invoked before processing contested
VMAs, destroying vm->pt_root while those VMAs were still linked to their
respective buffer objects (vm_bo->list.gpuva).
If a concurrent thread evicts one of those shared buffer objects,
xe_bo_trigger_rebind() holding only bo->resv walks the BO's VMAs and, in
fault mode, calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(). Because
vm->pt_root[tile->id] is already NULL, dereferencing pt->level causes a
NULL ptr deref.
Fix this by deferring xe_vm_free_scratch() and xe_vm_pt_destroy() until
after all contested VMAs have been unlinked and destroyed.
User is reporting hitting a NULL ptr deref in xe_pt_zap_ptes(), which
could be explained by this race.
Assisted-by: LLM
Fixes: b06d47be7c ("drm/xe: Port Xe to GPUVA")
Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/9290
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: <stable@vger.kernel.org> # v6.12+
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260918131034.598078-2-matthew.auld@intel.com
Ensure GT C6 state before D3 transition in runtime suspend flow.
This avoids the scenario where GT is being accessed after D3 state
is programmed. If the check for C6 fails, resume the device so it is
not stuck in an intermediate state.
v2: Force resume when we cancel suspend due to C6 check (Sashiko)
v3: Abort the suspend properly when idle check fails (Sashiko)
v4: Only check for C6 during suspend (Rodrigo). Have check in
xe_pci_runtime_suspend() instead of xe_pci_suspend().
v5: Don't checkfor PVC (Sashiko), rename function from_device_c6
to _device_idle
v6: Rename to _wait_all_c6 and move PVC check to gt_idle (Rodrigo)
Cc: Badal Nilawar <badal.nilawar@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260916190614.3047807-1-vinay.belgaumkar@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Drop the definition of MI_SEMAPHORE_WAIT_TOKEN and replace its uses with
MI_SEMAPHORE_WAIT. Both #define's were representing the same opcode
0x1C and this GPU instruction has always been named "MI_SEMAPHORE_WAIT"
in the official specs.
The "*_TOKEN" definition appears to have been copied from i915's code.
i915 needed two separate definitions to reflect the slightly different
instruction lengths gen8-gen11 platforms (4 dwords) vs gen12 and later
platforms (5 dwords, with the extra dword representing a token), but Xe
only supports newer platforms with the 5-dword form, so only a single
definition is needed and should match the official name given in the
specs.
Bspec: 14054, 45749, 60244
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260917-semaphore_inst-v1-1-61e49405f71d@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
pci_p2pdma_distance() can optionally print details about how it
computed the ACS/PCIe topology distance between the two devices, but
xe_dma_buf_attach() always called it with verbose set to false,
silently hiding useful diagnostics whenever a dma-buf attach unset
attach->peer2peer.
Tie the verbose argument to a dedicated dynamic-debug callsite tagged
"xe dma-buf p2p debug" (in addition to keeping the existing
DRM_UT_DRIVER category as a default), so the extra p2pdma distance
logging can be switched on without a recompile in any of the following
ways:
- Enable the whole DRIVER category for xe via drm.debug, which is
already used elsewhere in the driver:
# echo 0x10 > /sys/module/drm/parameters/debug
- Enable dynamic debug for the whole file:
# echo 'file xe_dma_buf.c +p' > \
/sys/kernel/debug/dynamic_debug/control
- Enable dynamic debug for just this callsite by function name:
# echo 'func xe_dma_buf_attach +p' > \
/sys/kernel/debug/dynamic_debug/control
- Enable dynamic debug by matching the format string used to
register the callsite:
# echo 'format "xe dma-buf p2p debug" +p' > \
/sys/kernel/debug/dynamic_debug/control
- Enable it at boot via the dyndbg= kernel command line parameter:
dyndbg="func xe_dma_buf_attach +p"
Any of the above flips verbose_p2p_dist to true for that attach call,
causing pci_p2pdma_distance() to log its distance calculation without
requiring the whole DRM_UT_DRIVER category to be enabled.
v2:
- Guard the _dynamic_func_call() invocation and
xe_ddebug_flip_bool() with a CONFIG_DYNAMIC_DEBUG check so the
build does not break when dynamic debug is disabled. (Sashiko-bot)
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Assisted-by: LLM
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Link: https://patch.msgid.link/20260914114617.112012-1-thomas.hellstrom@linux.intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
drm_pagemap_dev_unhold_work() processes an unbatched llist of pagemaps
on the system workqueue. This creates potential latency traps during
heavy teardown cycles due to two compounding factors:
1. The loop contains a per-item drm_dbg() trace. If dynamic debug is
enabled on a slow serial console, synchronous print latency scales
linearly with the batch size and can block worker execution.
2. The llist can accumulate large batches during intensive unmap
operations, monopolizing worker resources without yielding.
This causes below log warning:
"workqueue: drm_pagemap_dev_unhold_work [drm_gpusvm_helper] hogged CPU
for >10000us 19 times, consider switching to WQ_UNBOUND"
Fix this by implementing a latency mitigation strategy:
- Move the teardown work to system_unbound_wq to avoid tying up
per-CPU bound worker pools.
- Add cond_resched() inside the teardown loop to yield the CPU during
large batch teardowns, ensuring system responsiveness.
- Convert the drm_dbg() trace to drm_dbg_ratelimited() to mitigate
serial console bottlenecks while preserving debuggability.
v2:
- Changed workqueue from deprecated system_unbound_wq to system_dfl_wq.
(Arvind Yadav)
Fixes: a26084328a ("drm/pagemap, drm/xe: Manage drm_pagemap provider lifetimes")
Reviewed-by: Arvind Yadav <arvind.yadav@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Signed-off-by: Varun Gupta <varun.gupta@intel.com>
Link: https://patch.msgid.link/20260918081320.30389-2-varun.gupta@intel.com
Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
The desired behavior for SVM eviction failures, which can occur due to
various uncontrollable races, is for TTM to continue walking the LRU
list and look for another eviction candidate. This is expressed by
returning -ENOSPC from the ->move() callback.
Adjust the SVM eviction failure path because of races in ->move() to
return -ENOSPC so that TTM continues searching for another buffer to
evict.
Fixes: 3ca608dc75 ("drm/xe: Basic SVM BO eviction")
Cc: stable@vger.kernel.org
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patch.msgid.link/20260917203158.292823-1-matthew.brost@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
xe_pagefault_print() was firing without indicating why servicing of a
fault failed, making it hard to distinguish benign races (e.g. a VM's
file descriptor closing mid-fault) from real bugs.
Add enum xe_pagefault_error, uniquely identifying the high-level point
at which xe_pagefault_service(), xe_pagefault_handle_vma(), and
xe_svm_handle_pagefault() fail. Since consumer.page_addr is always
page (4K) aligned, steal the reserved low byte to carry this code
without growing struct xe_pagefault. Add xe_pagefault_set_error() /
xe_pagefault_get_error() to encode/decode it, and xe_pagefault_addr()
as the single accessor for the real (masked) faulted address; update
all real address consumers (xe_pagefault_match(), cache alignment,
xe_vm_add_fault_entry_pf()) to go through it instead of reading
consumer.page_addr directly. xe_pagefault_set_start_addr() now
preserves any already-recorded error bits rather than clobbering them.
xe_pagefault_print() decodes the error via a new
xe_pagefault_error_to_str() and reports it in an added "Error:" line.
xe_pagefault_asid_to_vm() now distinguishes an ASID with no VM at all
(XE_PAGEFAULT_ERROR_VM_NOT_FOUND, e.g. the owning file descriptor was
already closed) from a VM found but not in fault mode
(XE_PAGEFAULT_ERROR_VM_NOT_IN_FAULT_MODE).
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Assisted-by: GitHub_Copilot:claude-sonnet-5
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patch.msgid.link/20260917203850.321385-1-matthew.brost@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
A TLB invalidation issued on a wedged device fails with
-ENOTRECOVERABLE. xe_tlb_inval_issue() squashes only -ECANCELED, so the
error reaches callers that treat it as unexpected and WARN, tainting the
kernel on a wedge that was deliberately caused:
ggtt_invalidate_gt_tlb() drivers/gpu/drm/xe/xe_ggtt.c
xe_svm_invalidate() drivers/gpu/drm/xe/xe_svm.c
xe_bo_trigger_rebind() drivers/gpu/drm/xe/xe_bo.c
xe_vma_userptr_do_inval() drivers/gpu/drm/xe/xe_userptr.c
igt@xe_exec_reset@gt-reset-fault-injection hits the GGTT one, turning an
otherwise passing run into an abort:
*ERROR* SIGID=102 FATAL (-EIO) WEDGED: Device declared wedged!
Tile0: GT1: Failed to invalidate GGTT (-ENOTRECOVERABLE)
WARNING: drivers/gpu/drm/xe/xe_ggtt.c:588 at ggtt_invalidate_gt_tlb
Workqueue: xe-guc-destroy-wq __guc_exec_queue_destroy_async [xe]
ggtt_node_remove+0xe3/0x100 [xe]
xe_ggtt_remove_bo+0x89/0x2c0 [xe]
xe_ttm_bo_destroy+0xcb/0x330 [xe]
...
xe_lrc_destroy+0x74/0x90 [xe]
xe_exec_queue_fini+0x2d/0x60 [xe]
-ECANCELED and -ENOTRECOVERABLE mean the same thing at this layer: the
message was dropped rather than delivered, and the fence has already
been signalled before the error is returned, so there is nothing left to
wait for. Squash both.
A wedged device is only recovered by a fresh initialisation, so the
error return in xe_bo_trigger_rebind() becomes unreachable by design.
v2: fix all invalidation paths. (Sashiko)
Fixes: 50fa9acac2 ("drm/xe/guc: distinguish wedged from recoverable cancellation")
Assisted-by: Claude:claude-opus-5
Cc: Sk Anirban <sk.anirban@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260914215318.200603-1-shuicheng.lin@intel.com
CRI defines bit 5 of the perf limit reasons register as a power brake
(PWRBRK) indicator. Add PWRBRK_MASK and a reason_pwrbrk sysfs attribute
for CRI in place of reason_ratl.
Signed-off-by: Sk Anirban <sk.anirban@intel.com>
Fixes: 8578e6d054 ("drm/xe/gt_throttle: Drop individual show functions")
Reviewed-by: Raag Jadav <raag.jadav@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260909114931.1039331-2-sk.anirban@intel.com
Extend the existing "hw_engines" debugfs to iterate over all hardware
engines and dump their idle state: whether the engine pipe is idle or
busy, whether C6 entry is allowed, and (where available) the forcewake
request status.
v2: Use existing debugfs (Matt Brost)
v3: Rename macro (Vinay)
Signed-off-by: Sk Anirban <sk.anirban@intel.com>
Reviewed-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260910165855.1124574-2-sk.anirban@intel.com
Start using recently added KLV helpers for parsing migration
packet to properly detect any malformed KLV formats.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Link: https://patch.msgid.link/20260916215827.13488-4-michal.wajdeczko@intel.com
Instead of retrieving data from the KLV dwords manually, add
simple helpers that will do that together with sanity checks.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Link: https://patch.msgid.link/20260916215827.13488-2-michal.wajdeczko@intel.com
Add region 5 (System Controller) to the list of regions
available on device.
Signed-off-by: Alexander Usyskin <alexander.usyskin@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260914-nvm_add_sysctl_region-v1-1-75d710daf5d0@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
We will soon want to auto provision all VFs in a single step, while
already holding the master mutex. Split existing function into two.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Link: https://patch.msgid.link/20260911183257.14999-4-michal.wajdeczko@intel.com
We will soon want to unprovision all VFs in a single step, while
already holding the master mutex. Split existing function into two.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Link: https://patch.msgid.link/20260911183257.14999-3-michal.wajdeczko@intel.com
We will soon want to provision VFs in a single step, while already
holding the master mutex. Split existing function into two.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Link: https://patch.msgid.link/20260911183257.14999-2-michal.wajdeczko@intel.com
Currently, struct xe_i2c is freed before SGUnit IRQ is disabled in unbind
path, leaving a potential UAF in case I2C IRQ is hit during this small
window. Explicitly disable I2C IRQ in xe_i2c_remove() and fix this.
Fixes: 0bb78ce099 ("drm/xe/i2c: Wire up reset/postinstall for I2C IRQ")
Signed-off-by: Raag Jadav <raag.jadav@intel.com>
Reviewed-by: Heikki Krogerus <heikki.krogerus@linux.intel.com>
Link: https://patch.msgid.link/20260911121547.2407261-1-raag.jadav@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
It turned out that during early probe phase, VFs use detached from
the xe_device, temporary xe_gt objects, which when used as location
in xe_log() macros, will be treated by the dmesg decoration code as
bogus, possibly triggering a WARN, and the output will look like:
[drm] *ERROR* SIGID=104 (-ETIMEDOUT) LOC3.0? GUC: MMIO request ...
instead of expected:
[drm] *ERROR* SIGID=104 (-ETIMEDOUT) Tile0: GT0: GUC: MMIO request ...
Relax the tile/GT id validation and instead of looking for the real
objects, only check if encoded id is within the range of possible
tiles or GTs on the current xe device, using data from the device
descriptor rather then the object list.
Fixes: 1151b9f6f4 ("drm/xe/log: Add component/location decorations to dmesg")
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260904170531.516-7-michal.wajdeczko@intel.com
We can now obtain the const pointer to struct xe_device_desc directly
from the xe, no need to pass it as explicit parameter.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Link: https://patch.msgid.link/20260904170531.516-6-michal.wajdeczko@intel.com
We can now obtain the const pointer to struct xe_device_desc directly
from the xe, no need to pass it as explicit parameter.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Link: https://patch.msgid.link/20260904170531.516-5-michal.wajdeczko@intel.com
We can now obtain the const pointers to struct xe_device_desc and
xe_subplatform_desc directly from the xe, no need to pass them as
explicit parameters.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Link: https://patch.msgid.link/20260904170531.516-4-michal.wajdeczko@intel.com
While we already use struct intel_device_info, which has some fields
copied or derived from the original struct xe_device_desc, we also
want to keep the reference to the whole device descriptor, in case
we would like to access more non-mutable device parameters.
Similarly keep the reference to matched subplatform descriptor.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Link: https://patch.msgid.link/20260904170531.516-3-michal.wajdeczko@intel.com
We don't need anything from struct xe_device_desc in xe_info_init()
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Link: https://patch.msgid.link/20260904170531.516-2-michal.wajdeczko@intel.com
If a test does not specify subplatform details about the fake device
it wants to create, our helper may select a sentinel of the subplatform
descriptors list instead of the first available subplatform.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Link: https://patch.msgid.link/20260905200157.3164-3-michal.wajdeczko@intel.com
If a test does not specify details about the fake device it wants
to create, our helper selects the device descriptor from the first
entry from the pciidlist and uses NULL for the subplatform descriptor.
While this works fine today, as the first device is a TGL which does
not have any subplatforms, it will be broken if we reorder pciidlist.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Gustavo Sousa <gustavo.sousa@intel.com>
Link: https://patch.msgid.link/20260905200157.3164-2-michal.wajdeczko@intel.com
VRAM page offlining can clear a page-table BO's vmap before VM teardown
calls xe_pt_clear(). Check for a purged BO under its dma-resv lock before
writing to the mapping.
v4(Sashiko):
- dont leave svm notifier lock early
v3(Sashiko):
- Validate NULL vmap
v2(Sashiko):
- Fix lock ordering
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260909111620.1276402-2-tejas.upadhyay@intel.com
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
Drop the local PL_WRITE_MBX_TIMEOUT_MS macro and use the common
PCODE_DEFAULT_TIMEOUT_MS for the power limit write mailbox command.
This removes a duplicate timeout definition and keeps hwmon in sync with
the pcode default (now 10ms).
Signed-off-by: Karthik Poosa <karthik.poosa@intel.com>
Link: https://patch.msgid.link/20260907083520.192418-3-karthik.poosa@intel.com
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
The previous 1ms timeout may be insufficient for some mailbox commands,
leading to spurious -ETIMEDOUT failures under load. Increase the default
timeout to 10ms and introduce PCODE_DEFAULT_TIMEOUT_MS for a shared
timeout definition, in line with pcode mailbox specification guidance.
Signed-off-by: Karthik Poosa <karthik.poosa@intel.com>
Link: https://patch.msgid.link/20260907083520.192418-2-karthik.poosa@intel.com
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
__xe_shrinker_walk() walks the SYSTEM and TT LRUs without a runtime PM
reference. Shrinking a bo outside system memory invalidates its GPU
mappings, which needs the device resumed, so while it is runtime
suspended the page table zap trips an assert and the TLB invalidation
returns -ENODEV:
WARNING: drivers/gpu/drm/xe/xe_bo.c:770 at xe_bo_move_notify+0x1fc/0x450 [xe]
xe_bo_shrink+0x20f/0x2b0 [xe]
__xe_shrinker_walk+0x174/0x410 [xe]
xe_shrinker_scan+0x10c/0x1e0 [xe]
do_shrink_slab+0x176/0x7e0
drop_caches_sysctl_handler+0x9c/0xf0
Take a reference before walking a memory type other than XE_PL_SYSTEM
and stop there if it cannot be acquired. Reuse the shrinker's existing
acquire path, which resumes the device directly where reclaim allows
that and otherwise queues the PM worker for a later scan. Stop the walk
once the scan target is met, so a satisfied scan does not wake the
device. System memory is still reclaimed while the device is suspended.
Gate this on xe_device_is_l2_flush_optimized(), the same condition under
which xe_bo_trigger_rebind() issues the invalidation for a non-fault-mode
vm, so reclaim is unaffected elsewhere. The System CCS copy already has
its own reference in xe_bo_shrink().
Only a non-fault-mode vm can reach this, since a fault-mode vm requires
LR mode and that holds a runtime PM reference for the vm's lifetime.
Reproduced with igt@xe_madvise@dontneed-before-exec while the GPU is
runtime suspended.
v2: simplify needs_rpm check. (Matt)
retarget Fixes tag since the issue occurs with the non-fault-mode
path added by 4e7ebff69a.
v3: handle this in xe_shrinker.c instead of xe_bo.c (Thomas)
v4: stop the walk once the scan target is met. (Sashiko)
v5: rebase on the freed page accounting fix. (Sashiko)
v6: reuse the shrinker acquire path so runtime pm can be resumed
directly instead of always queueing a worker. (Thomas)
v7: replace xe_pm_runtime_put() with xe_shrinker_runtime_pm_put(). (Thomas)
Fixes: 4e7ebff69a ("drm/xe/xe3p_lpg: flush shrinker bo cachelines manually")
Assisted-by: Claude:claude-opus-5
Cc: Tejas Upadhyay <tejas.upadhyay@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patch.msgid.link/20260909162102.1097006-3-shuicheng.lin@intel.com
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
__xe_shrinker_walk() and xe_shrinker_walk() return either the number of
pages freed or a negative error, so the two cannot be reported at once.
On error the pages already freed are dropped, and since xe_shrinker_scan()
only accumulates non-negative returns while *scanned is updated by
pointer, the shrinker tells mm that it scanned without freeing.
Accumulate the count into a caller-provided counter and return only the
status, so an error no longer discards what the walk had freed.
Fixes: 00c8efc318 ("drm/xe: Add a shrinker for xe bos")
Assisted-by: Claude:claude-opus-5
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260909162102.1097006-2-shuicheng.lin@intel.com
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Add xe_sysctrl_check_app_status() to query the state of a System
Controller application using get_app_status_by_id mailbox command.
The helper maps xe_sysctrl_app_id values to firmware application
IDs and returns the reported application state. Add a convenience
wrapper to check diag firmware application readiness.
Signed-off-by: Anoop Vijay <anoop.c.vijay@intel.com>
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Reviewed-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Signed-off-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Link: https://patch.msgid.link/20260904134935.674507-5-anoop.c.vijay@intel.com
Convert the PCI barrier mmap over to use xe_mmio_gem, which is a good
match for this functionality. This has the following advantages:
1. Removes a bunch of code.
2. Replaces the fragile hard coded fake offset design.
3. Adds the first user for xe_mmio_gem, which is preferred over nuking
it. There are also potentially other upcoming usecases wanting this
type of functionality, so having standard component to do this would
be good.
There shouldn't be any big functional change here. From userspace pov,
they still query the fake offset like before, just that now it is no
longer hard coded in the KMD.
v2 (Thomas):
- Prefer scoped_guard(). Also, just annotate ALL locations, even if
not strictly needed. Reflect that in the kernel-doc. This will also
shut up static analysis tools.
Assisted-by: LLM
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Tejas Upadhyay <tejas.upadhyay@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Ilia Levi <ilia.levi@intel.com>
Reviewed-by: Ilia Levi <ilia.levi@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patch.msgid.link/20260908165046.1393557-18-matthew.auld@intel.com
Hardware MMIO registers should never be executable; reject VM_EXEC
at mmap time and clear VM_MAYEXEC to prevent later mprotect attempts.
Also drop VM_DONTCOPY so that child processes across fork() can
inherit the mapping and lazily fault in the PFNs, matching standard
DRM GEM semantics and making the existing drm_gem_vm_open() callback
functional.
This aligns with existing PCI_BARRIER, which will use this in the next
patch. We don't want any noticeable behaviour change there, since this
will be user visible. This will also be the first user.
Assisted-by: LLM
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Tejas Upadhyay <tejas.upadhyay@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Ilia Levi <ilia.levi@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patch.msgid.link/20260908165046.1393557-17-matthew.auld@intel.com
xe_mmio_gem_destroy() currently frees the GEM object directly, bypassing
reference counting. Since existing VMAs hold a reference and the fault
handler accesses the object through vma->vm_private_data, this is
use-after-free. Additionally, nothing prevents the fault handler from
installing PTEs to the real MMIO after destroy.
Fix this with proper synchronization and refcounting. Also, do not set
vm_pgoff to zero. Many DRM drivers do this because helpers like
dma_mmap_pages() interpret vm_pgoff as an intra-buffer page offset;
leaving the DRM fake offset there would break these helpers.
Those drivers can get away with zeroing it because they map eagerly -
all PTEs are established before mmap returns, so vm_pgoff is never
consulted again. Our driver does not use such helpers and the newly
introduced call to drm_vma_node_unmap() relies on vm_pgoff being untouched.
v2: (Matt Auld)
- use dma_resv lock to serialize fault handler with destroy
- SIGBUS on access after destroy
Fixes: 1ffcf8b8ae ("drm/xe: Support for mmap-ing mmio regions")
Assisted-by: GitHub-Copilot:claude-opus-4.6
Signed-off-by: Ilia Levi <ilia.levi@intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260908165046.1393557-16-matthew.auld@intel.com
Currently, when the fault handler provides a dummy page, it
allocates a new one on every invocation and ties its lifetime to
the drm_device via drmm_add_action_or_reset(). Concurrent faults
after hot-unplug therefore accumulate pages that persist until
device teardown.
Cache a single dummy page in the xe_mmio_gem object and use dma_resv
lock to protect its allocation. Free it with the object.
v2: use dma_resv lock to protect the allocation (Matt Auld)
Assisted-by: GitHub-Copilot:claude-opus-4.6
Signed-off-by: Ilia Levi <ilia.levi@intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260908165046.1393557-15-matthew.auld@intel.com
