All dmesg log lines generated by the xe_log() functions or helper
macros are expected to have a stable format. Add tests to verify
that generated output is not accidentally changed. Add demo pseudo
tests to show how to use all variants of the xe_log() helpers.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-17-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
While we have helpers pdev_to_xe_device() and kdev_to_xe_device()
to obtain the xe_device pointer from the struct device pointer or
from the struct pci_dev pointer, we must also setup the drm_device
pointer as driver data in the device to make them work.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-16-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
While not strictly required, we want to make sure that components
definitions used by the RAS firmware are in the same fixed relation
with HW components definitions used by the LOG infrastructure to
allow code simplification. Add static asserts to enforce that.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Aravind Iddamsetty <aravind.iddamsetty@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-15-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
When CONFIG_PRINTK_INDEX is enabled, it is expected that all device
level printk messages are indexed for audit. While usually this is
done automatically behind the scenes when code is using regular
dev_printk macros, since we are generating different dmesg messages
inside xe_log_emit() based on the severity, component and location,
we only get those entries in /sys/kernel/debug/printk/index/xe:
<3> drivers/gpu/drm/xe/xe_log.c:142 log_dmesg_vprintk "%s %s: [drm] *ERROR* %pV"
<6> drivers/gpu/drm/xe/xe_log.c:140 log_dmesg_vprintk "%s %s: [drm] %pV"
Explicitly generate printk index using dev_printk_index_emit() with
some generic prefix that includes the SIGID tag.
Suggested-by: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-12-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
For the software based SIGID errors, we will usually want to pass
only the errno value as the data to be logged in the dmesg line or
the CPER record. Add simple wrappers for that.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com> #v3
Link: https://patch.msgid.link/20260814171801.13347-11-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
There are only few CPER severity levels defined, add some helpers
to avoid forcing developers to write full CPER severity enum name
in every macro call.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-10-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Our component list already defines relation between the component
name, identified by the TAG, and its numerical identifier and the
assigned SIGID. We can also already prepare location identifier
based on the pointer type used to emit the log entry. Add helper
that will prepare SIGID, COMPONENT and LOCATION parameters for
xe_log_emit() using provided components TAG and the pointer.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-9-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
There are only few CPER severity levels defined, add more helpers
to avoid forcing developers to write full CPER severity name in
every function call.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-8-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
We can determine the location of the SIGID error source based on the
type of the xe pointer being currently available. Add helper macros
that will accept any of xe_device, xe_tile or xe_gt pointers instead
of plain pci_dev to generate valid XE_LOG_LOCATION identifier and then
call low-level xe_log_emit(pdev) function.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-7-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
There are only few CPER severity levels defined, add simple helper
for each of them, to avoid forcing developers to write full severity
name in every function call.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-6-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
While we can't directly use our xe_tile|gt_err|info helpers to get
nice Tile/GT decorations, we can still add them manually based on
the structured location parameter. Similarly, we can add component
name prefix based on the component identifier.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-5-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Introduce structured identifiers for each component type that
could emit a SIGID log entry and for their locations. We plan
to store those IDs in the CPER records for better filtering.
Define also structured identifiers for the supported locations.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com> #v3
Link: https://patch.msgid.link/20260814171801.13347-4-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Today the driver reports faults with ad-hoc drm_err()/xe_gt_err()
strings that have no stable shape. That is readable for a human, but it
gives fleet tooling nothing durable to match on: the wording changes
between releases, lines can be rate-limited or dropped under an error
storm, and there is no consistent way to ask "which recognised fault
just happened?".
Introduce a signature identifier (SIGID): a small, stable integer that
names one recognised Xe fault site and serves as the primary handle for
triage. A SIGID maps, through published end-user documentation, to a
description and a recommended action; the driver only has to emit the
right SIGID next to the usual human-readable text.
Signed-off-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Assisted-by: Copilot:Opus-4.8
Co-developed-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Riana Tauro <riana.tauro@intel.com>
Cc: Stuart Summers <stuart.summers@intel.com>
cc: Badal Nilawar <badal.nilawar@intel.com>
Reviewed-by: Badal Nilawar <badal.nilawar@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-3-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
In upcoming patches we want to define macros that will work with
either xe_device or xe_tile or xe_gt pointers. To make them work
and to allow compiler optimizations, introduce set of helpers
that will return either expected pointer type or NULL or make
necessary conversions to/from the struct xe/device/pci_dev.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Cc: Jani Nikula <jani.nikula@intel.com>
Link: https://patch.msgid.link/20260814171801.13347-2-michal.wajdeczko@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Enable package and vram temperatures support from hwmon on CRI.
The required MMIO offsets of these are same as on BMG.
Signed-off-by: Karthik Poosa <karthik.poosa@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260813193546.511093-1-karthik.poosa@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
When PUNIT (power management unit) errors are detected that persist across
warm resets, mark the device as wedged with DRM_WEDGE_RECOVERY_COLD_RESET
and notify userspace that a complete device power cycle is required to
restore normal operation.
Signed-off-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Reviewed-by: Raag Jadav <raag.jadav@intel.com>
Link: https://patch.msgid.link/20260805071152.1225416-9-mallesh.koujalagi@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
When ``WEDGED=cold-reset`` is sent, it indicates that the device has
encountered an error condition that cannot be resolved through other
recovery methods such as driver rebind or bus reset, and requires a
complete device power cycle to restore functionality.
Signed-off-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Reviewed-by: Raag Jadav <raag.jadav@intel.com>
Link: https://patch.msgid.link/20260805071152.1225416-8-mallesh.koujalagi@intel.com
Acked-by: Maxime Ripard <mripard@kernel.org>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Introduce DRM_WEDGE_RECOVERY_COLD_RESET (BIT(4)) recovery method to handle
scenarios requiring device power cycle.
This method addresses cases where other recovery mechanisms
(driver reload, PCIe reset, etc.) are insufficient to restore device
functionality. When set, it indicates to userspace that only device power
cycle can recover the device from its current error state.
Signed-off-by: Mallesh Koujalagi <mallesh.koujalagi@intel.com>
Reviewed-by: Raag Jadav <raag.jadav@intel.com>
Link:
https://patch.msgid.link/20260805071152.1225416-7-mallesh.koujalagi@intel.com
Acked-by: Maxime Ripard <mripard@kernel.org>
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
This is supposed to print the error code but there is a copy and
paste bug so it prints "bo" instead of "err".
Fixes: dd08ebf6c3 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Signed-off-by: Dan Carpenter <error27@gmail.com>
Link: https://patch.msgid.link/an1tu0z3T-qX1ogn@stanley.mountain
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
When an interrupt is received for correctable errors indicating that error
counter has crossed its threshold, read the current counter value and
deliver a drm-ras error-event to userspace for each affected component.
Also send drm-ras error-event to userspace for uncorrectable errors on
receiving an AER.
To avoid sending duplicate events when the same component appears multiple
times in the response, send the error-event once per component.
Signed-off-by: Riana Tauro <riana.tauro@intel.com>
Reviewed-by: Raag Jadav <raag.jadav@intel.com>
Link: https://patch.msgid.link/20260810112008.1858731-6-riana.tauro@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Define a new netlink event 'error-event' and a new multicast group
'error-report' in drm_ras. Each event contains device name, node and
error information to identify the error triggering the event.
Add drm_ras_nl_error_event() to trigger an event from the driver.
Userspace must subscribe to 'error-report' to receive 'error-event'
notifications.
Usage:
$ sudo ynl --family drm_ras --subscribe error-report
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: Zack McKevitt <zachary.mckevitt@oss.qualcomm.com>
Cc: Lijo Lazar <lijo.lazar@amd.com>
Cc: Hawking Zhang <Hawking.Zhang@amd.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Paolo Abeni <pabeni@redhat.com>
Cc: Eric Dumazet <edumazet@google.com>
Signed-off-by: Riana Tauro <riana.tauro@intel.com>
Reviewed-by: Raag Jadav <raag.jadav@intel.com>
Link: https://patch.msgid.link/20260810112008.1858731-5-riana.tauro@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
DRM RAS interface provides a standardized way for GPU/accelerator
drivers to expose error counters and other reliability nodes to
user space via Generic Netlink.
Since this is cross-driver infrastructure carrying its own uAPI, add a
dedicated MAINTAINERS entry covering the implementation, headers,
netlink spec and documentation, maintained through drm-misc.
Cc: Simona Vetter <simona@ffwll.ch>
Cc: David Airlie <airlied@redhat.com>
Cc: Donald Hunter <donald.hunter@gmail.com>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>
Cc: Simon Horman <horms@kernel.org>
Cc: Raag Jadav <raag.jadav@intel.com>
Cc: Riana Tauro <riana.tauro@intel.com>
Cc: Aravind Iddamsetty <aravind.iddamsetty@linux.intel.com>
Cc: Zack McKevitt <zachary.mckevitt@oss.qualcomm.com>
Acked-by: Jakub Kicinski <kuba@kernel.org>
Acked-by: David Airlie <airlied@redhat.com>
Link: https://patch.msgid.link/20260805094949.536651-2-rodrigo.vivi@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
In cases where min frequency was actually greater than BMG_MIN_FREQ,
we were not using the updated min frequency as there was a missing
call to pc_action_query_task_state() between the two settings of
min frequency. Since we know what min_freq was last set, use that
cached value while comparing to BMG_MIN_FREQ to fix this issue.
v2: pc->freq_ready is not set until after pc_adjust_freq_bounds(). Stay
with pc_action_query_task_state() instead.
v3: Update commit message (Stuart)
Fixes: bdde16c9ac ("drm/xe/bmg: Update Wa_14022085890")
Signed-off-by: Balasubramani Vivekanandan <balasubramani.vivekanandan@intel.com>
Signed-off-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Link: https://patch.msgid.link/20260805234649.2076384-1-vinay.belgaumkar@intel.com
There is no need to check the queue inside the pagefault handler
as it is immediately checked later by the xe_pagefault_queue_add()
function and we can just look for the NULL lpf entry. There will
be also already a full WARN triggered, so no need keep drm_warn().
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260811200906.10400-3-michal.wajdeczko@intel.com
It is safer and easier to use guard() instead of explicit lock
unlock calls.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patch.msgid.link/20260811200906.10400-2-michal.wajdeczko@intel.com
drmm_mutex_init() can fail while registering its managed cleanup action.
On failure, the reset path destroys the mutex, so continuing OA setup
leaves an unusable lock that later paths may acquire.
Return the error from per-GT OA initialization and abort device-wide OA
initialization if the metrics lock cannot be initialized.
Fixes: a9f905ae7b ("drm/xe/oa/uapi: Initialize OA units")
Fixes: cdf02fe1a9 ("drm/xe/oa/uapi: Add/remove OA config perf ops")
Signed-off-by: Linmao Li <lilinmao@kylinos.cn>
Reviewed-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Signed-off-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Link: https://patch.msgid.link/20260713083053.321091-1-lilinmao@kylinos.cn
xe_oa_emit_oa_config() releases the sync entries and the syncs array
only on its success path. When it fails before the point of no return
(fence allocation, config buffer allocation or batch submission), it
returns without touching stream->syncs.
The stream open path handles such failures in the caller, but
xe_oa_config_locked() propagates the error without any cleanup, so the
syncs array and the fence references held by the parsed entries are
leaked. The next config ioctl overwrites stream->syncs, making the
memory unreachable for good.
Clean up the parsed syncs when xe_oa_emit_oa_config() fails, matching
the cleanup done by the stream open error path.
Fixes: 9920c8b88c ("drm/xe/oa: Add syncs support to OA config ioctl")
Signed-off-by: Linmao Li <lilinmao@kylinos.cn>
Reviewed-by: Ashutosh Dixit <ashutosh.dixit@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.dixit@intel.com>
Link: https://patch.msgid.link/20260731011932.3426219-1-lilinmao@kylinos.cn
Problem: CTX_TIMESTAMP MMIO reads could be stale if a context
switched out between check and read; LRC stores a sentinel while
a context starts that must not be treated as a real timestamp.
Fix: Check the LRC-stored sentinel before and after the MMIO read;
return the LRC value if the context switched out to avoid TOCTOU.
Note: Keep XE_LRC_CTX_TIMESTAMP_ACTIVE in xe_lrc.h as the
canonical sentinel.
Fixes: d243ef6a39 ("drm/xe/lrc: Refactor xe_lrc_timestamp to simplify logic")
Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/7956
Assisted-by: GitHub-Copilot:claude-sonnet-5
Signed-off-by: Gajendra Uttamchand <gajendra.uttamchand@intel.com>
Reviewed-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Acked-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
Link: https://patch.msgid.link/20260810071812.213358-4-gajendra.uttamchand@intel.com
has_drm_ras flag is meant to facilitate drm_ras feature. Move it to the
correct layer where it belongs.
Fixes: 63dfab5786 ("drm/xe/xe_ras: Add drm_ras feature flag")
Signed-off-by: Raag Jadav <raag.jadav@intel.com>
Reviewed-by: Riana Tauro <riana.tauro@intel.com>
Link: https://patch.msgid.link/20260810124101.105832-1-raag.jadav@intel.com
Signed-off-by: Matt Roper <matthew.d.roper@intel.com>
Suspend/resume causes the PXP keys to become invalid, but doesn't
actually kill the session. The driver also doesn't explicitly kill and
re-start the session until a new PXP request comes in, which means that
the "zombie" session can potentially stick around if there are no new
requests from userspace. While this is not an issue for PXP, HDCP has a
new behavior starting on PTL where a communication is sent to GSC if a
session is active at suspend time (even if it doesn't have a valid key),
which can lead to delays in the suspend flow if we suspend while the
zombie session is still active.
To avoid this, we can trigger a termination on resume and kill the
zombie session immediately, instead of delaying the termination to the
next PXP request. Due to restrictions in the rpm suspend/resume flow, we
can't call the termination flow from within the resume call itself, so
the pxp irq worker is expanded to cover this scenario.
The existing logic in the worker doesn't work as-is for the new flow,
because the pm_get_if_active will fail if the worker runs before the
pci_resume call has completed (which is possible, since we queue it
from within that call) or after we're started to suspend again.
Given that we always want to run the worker after a resume (differently
from the irq case, where we want to skip if we're suspended), we can
solve this by just taking the PM reference before queueing the worker.
As part of this rework, the pxp->events variable has been moved to atomic,
to avoid having to take xe->irq.lock from non-irq related paths.
Fixes: b1dcec9bd8 ("drm/xe/ptl: Enable PXP for PTL")
Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Julia Filipchuk <julia.filipchuk@intel.com>
Cc: Alan Previn <alan.previn.teres.alexis@intel.com>
Reviewed-by: Alan Previn <alan.previn.teres.alexis@intel.com>
Link: https://patch.msgid.link/20260720222757.3876338-2-daniele.ceraolospurio@intel.com
We will soon want to provision all VFs, including their GuC doorbell
IDs config, in one step. Split existing doorbells bulk configuration
function into two functions, one of which will expect the master lock
to be already taken. Use this new function in the fair GuC doorbells
provisioning.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com>
Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com>
Link: https://patch.msgid.link/20260807164826.5650-1-michal.wajdeczko@intel.com
We will soon want to provision all VFs, including their GuC context
IDs config, in one step. Split existing GuC IDs bulk configuration
function into two functions, one of which will expect the master
lock to be already taken. Use this new function in the fair GuC IDs
provisioning.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Link: https://patch.msgid.link/20260805154733.604-3-michal.wajdeczko@intel.com
We will soon want to provision all VFs, including their GGTT config,
in one step. Split existing GGTT bulk configuration function into
two functions, one of which will expect the master lock to be already
taken. Use this new function in the fair GGTT provisioning.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Reviewed-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Link: https://patch.msgid.link/20260805154733.604-2-michal.wajdeczko@intel.com
The ring buffer and the LRC context image are both mapped WC, so the ring
tail update can become visible to the device before the ring contents it
is meant to publish.
The GuC CT send does contain an xe_device_wmb(), so sending the H2G would
flush the ring contents. The problem is that it comes too late:
xe_lrc_set_ring_tail() publishes the tail before the H2G is sent, and the
device samples the tail from the context image independently of it, either
at context switch-in or while the context is already resident. A submitter
which is interrupted between updating the tail and sending its H2G
therefore leaves the device free to observe the new tail while the ring
contents behind it are not yet visible:
1. Thread A emits a job into the ring, sets the tail to T_A and sends
the H2G, which flushes A's ring contents. The GuC starts scheduling
the context in, but it is not executing yet.
2. Thread B emits a job into ring[T_A..T_B]. Those writes are not yet
visible to the device.
3. Thread B updates the ring tail to T_B. That write targets a
different page and becomes visible first.
4. Thread B is interrupted before it sends its H2G, so the flush which
would have published ring[T_A..T_B] has not happened yet.
5. The context is switched in and samples the ring tail from the
context image, picking up T_B rather than T_A.
6. The GPU executes A's job, advances HEAD to T_A, and continues on to
ring[T_A..T_B], which still holds the previous wrap's contents, so
the CS parses stale commands.
The result is command stream corruption, which typically manifests as a
hang or a spurious pagefault rather than anything that points back at the
submission path.
Kernel jobs are by far the most likely to hit this. Kernel queues such as
the migration queue are shared and can be driven by many threads
concurrently, producing back-to-back submissions on an LRC which is
already executing. User queues are typically tied to a single submitting
thread, so the same interleaving is much harder to produce.
Add an xe_device_wmb() at the end of xe_lrc_write_ring() so that it covers
every ring tail publication site, and so the invariant is local: once
xe_lrc_write_ring() returns, the ring contents are visible to the device.
Fixes: dd08ebf6c3 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8651
Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/7810
Cc: stable@vger.kernel.org
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Assisted-by: GitHub_Copilot:claude-opus-5
Reviewed-by: Stuart Summers <stuart.summers@intel.com>
Link: https://patch.msgid.link/20260807171716.140475-1-matthew.brost@intel.com
Add a new GT statistic, PARALLEL_PAGEFAULT_COUNT, to record when
multiple page fault workers are active concurrently.
When a worker dequeues a fault, scan peer workers for an active
cache entry and increment the counter if another fault is already
in flight. This provides basic visibility into parallel fault
handling behavior for performance analysis and tuning.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maciej Patelczyk <maciej.patelczyk@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-13-matthew.brost@intel.com
Pagefault storms can generate long chains of acknowledgments back to the
GuC. Sending each ack as a full CT submission forces a barrier,
descriptor update and doorbell per fault.
Extend xe_guc_ct_send_locked() with a “defer-flush” mode that copies the
message into the H2G ring but defers publishing the descriptor and
ringing the doorbell. Add xe_guc_ct_send_flush() to publish pending
writes and notify GuC once per batch. Wire this into the pagefault
producer via new ack_fault_begin/ack_fault_end callbacks and CT lock
wrappers.
To avoid excessive flush latency while still amortizing MMIO costs, use
a simple periodic flush heuristic for GuC pagefault acks: batch most
acks as defer-flush and force a publish at a fixed interval (e.g., every
16th ack), with a final flush at end-of-batch.
Also increase the H2G CTB size to 16K to better absorb bursts.
Assisted-by: ChatGPT:gpt-5 # Documentation
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-12-matthew.brost@intel.com
Add a debugfs entry to dump Xe page fault queue state. The output
includes queue geometry (entry size, total size, head/tail), per-entry
allocation state counts, and whether each page fault worker cache is
currently valid.
This is intended to help debug page fault storms, chaining, and retry
behaviour without needing tracing.
Assisted-by: ChatGPT:gpt-5 # Documentation
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maciej Patelczyk <maciej.patelczyk@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-11-matthew.brost@intel.com
Add GT stats to quantify pagefault chaining behavior during fault storms.
Track total chained faults, faults chained directly from the IRQ
handler, cases where IRQ chaining also drained the queue, chained faults
that had to be requeued due to range mismatch, and cases where the last
serviced range allowed immediate ack.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maciej Patelczyk <maciej.patelczyk@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-10-matthew.brost@intel.com
Some Xe platforms can generate pagefault storms where many faults target
the same address range in a short time window (e.g. many EU threads
faulting the same page). The current worker/locking model effectively
serializes faults for a given range and repeatedly performs VMA/range
lookups for each fault, which creates head-of-queue blocking and wastes
CPU in the hot path.
Introduce a page fault chaining cache that coalesces faults targeting
the same ASID and address range.
Each worker tracks the active fault range it is servicing. Fault entries
reside in stable queue storage, allowing the IRQ handler to match new
faults against the worker cache and directly chain cache hits onto the
active entry without allocation or waiting for dequeue. Once the leading
fault completes, the worker acknowledges the entire chain.
A small allocation state is added to each entry so queue, worker, and
IRQ paths can safely reference the same fault object. This prevents
reuse while the fault is active and guarantees that chained faults
remain valid until acknowledged.
Fault handlers also record the serviced range so subsequent faults can
be acknowledged without re-running the full resolution path.
This removes repeated fault resolution during fault storms and
significantly improves forward progress in SVM workloads.
Since threaded prefetches now use a dedicated prefetch workqueue
(usm.prefetch_wq) rather than sharing the page fault workqueue, page
fault servicing can no longer deadlock on vm->lock, so this cache does
not need an -EAGAIN retry path for a failed vm->lock acquisition.
Assisted-by: ChatGPT:gpt-5 # Documentation
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-9-matthew.brost@intel.com
Add a GT stat measuring total time spent servicing pagefault workqueue
iterations. The counter accumulates the runtime of the pagefault worker
in microseconds, allowing correlation of fault storms and chaining
behavior with CPU time spent in the fault handler.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maciej Patelczyk <maciej.patelczyk@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-8-matthew.brost@intel.com
Pack the engine class and instance fields into a single u8 to save space
in struct xe_pagefault. This also makes future extensions easier.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maciej Patelczyk <maciej.patelczyk@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-7-matthew.brost@intel.com
Add a module parameter to control the number of page-fault work threads,
making it easy to experiment with how different numbers of work threads
impact performance.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maciej Patelczyk <maciej.patelczyk@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-6-matthew.brost@intel.com
With fine-grained page-fault locking, it no longer makes sense to
maintain multiple page-fault queues, as we no longer hash queues based
on the VM’s ASID. Multiple workers can pull page faults from a single
queue, eliminating any head-of-queue blocking. Refactor the structures
and code to use a single shared queue.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Maciej Patelczyk <maciej.patelczyk@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-5-matthew.brost@intel.com
The migrate_vma_* functions are very CPU-intensive; as a result,
prefetching SVM ranges is limited by CPU performance rather than paging
copy engine bandwidth. To accelerate SVM range prefetching, the step
that calls migrate_vma_* is now threaded. A dedicated prefetch
workqueue is used for threading so prefetch work is never mixed with
page fault or garbage collector work.
Running xe_exec_system_allocator --r prefetch-benchmark, which tests
64MB prefetches, shows an increase from ~4.35 GB/s to 12.25 GB/s with
this patch on drm-tip. Enabling high SLPC further increases throughput
to ~15.25 GB/s, and combining SLPC with ULLS raises it to ~16 GB/s. Both
of these optimizations are upcoming.
Since the dedicated prefetch workqueue is not shared with page fault
or SVM garbage collector work, page fault servicing and garbage
collection can keep using a plain down_read() on vm->lock: there is no
risk of a blocked reader starving a worker that a pending writer is
waiting to flush, because that flushing is now confined to the
separate prefetch workqueue.
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-4-matthew.brost@intel.com
Prefetch-only VM bind IOCTLs do not modify VMAs or use userptr pages.
Downgrade vm->lock to read mode once setup is complete.
Lays the groundwork for prefetch IOCTLs to use threaded migration.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-3-matthew.brost@intel.com
Enable page faults to be serviced while holding vm->lock in read mode.
Introduce additional locks to:
- Ensure only one page fault thread services a given range or VMA
- Serialize SVM garbage collection
- Protect SVM range insertion and removal
While these locks may contend during page faults, expensive operations
like migration can now run in parallel within a single VM.
In addition to new locking, ranges must be reference-counted after
lookup, as another thread could immediately remove them from the GPU SVM
tree, potentially dropping the last reference.
Lastly, decouple the VM’s ASID from the page fault queue selection to
allow parallel page fault handling within the same VM.
Lays the groundwork for prefetch IOCTLs to use threaded migration too.
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Gwan-gyeong Mun <gwan-gyeong.mun@intel.com>
Link: https://patch.msgid.link/20260806185202.3922432-2-matthew.brost@intel.com
