NPA DMA-Bufs can only be imported on the same device as the
original BO. In source-identification mode NPA addresses are
importer-specific. Each GPU needs to import the UALink handle
by themselves. Sharing the NPA addresses between devices is
meaningless and potentially dangerous. Put controls to ensure
this is not violated.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Create a drm client for storing the GEM handles corresponding
to NPA BOs. We always want to keep one GEM handle alive for
the NPA BOs during the lifetime of the BO.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a use_lsdma flag to amdgpu_ualink_remote and set it during SDMA
entity initialization. When use_lsdma is set, use amdgpu_lsdma_copy_mem()
instead of SDMA IBs to write the remote command ring data, wptr and
doorbell to their respective NPA GART addresses.
This provides an alternative low-latency path for sending UALink remote
interrupt commands via LSDMA, avoiding SDMA ring/IB overhead for these
small NPA writes.
v2: fix 32 bit build (Alex)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Register the UALink IRQ source with the IH subsystem. Hardware IH
entries are re-queued onto a dedicated soft ring and drained by a work
queue handler. Add the soft ring, worker, and delegate helper to the
IRQ layer.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Sequence the setup steps into a single init path with rollback on
failure, and a teardown that halts firmware first. Hook both paths into
the virtual pod commit so resources rebuild on reconfiguration.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Expose the two public send operations for remote TLB invalidation and
remote interrupt delivery. Set up per-peer GART entries covering ring
buffers, write pointer pages, and doorbells for every active remote GPU.
v2: fix 32 bit build (Ivan)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Compute the writeback buffer addresses for a ring, send a configuration
command to firmware, and mark the ring ready on acknowledgement.
Configuration is deferred to first use.
v2: fix 32 bit build (Ivan)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add 16-dword packet layouts for TLB shootdown, remote interrupt, and
writeback address update. The dispatch path issues three sequential SDMA
copies (ring entry, write pointer, doorbell) and polls the firmware
writeback buffer for the acknowledged sequence number.
v2: fix 32 bit build (Alex)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Allocate pinned memory for firmware metadata, ring buffers, and write
and read pointer pages. Populate per-GPU metadata entries and send them
to MPNHT firmware via SMN mailbox, then poll for a ready acknowledgement.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add helpers to insert and remove NPA addresses in the GART page table
without a TTM buffer object, allowing the local SDMA engine to reach
remote GPU ring buffers, write pointer pages, and doorbell addresses.
v2: fix 32 bit build (Alex)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Allocate one GPU scheduler entity per SDMA instance for remote command
dispatch. Commands to the same peer are serialized through one entity;
different peers can be targeted concurrently by different SDMA engines.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Populate the NPA VM so remote GPUs can access local ring buffers and
write pointer pages at their reserved NPA addresses. Teardown waits for
DMA fences and issues a heavyweight TLB flush.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a ring buffer type enum and address formula for the reserved NPA
space between source and destination GPUs, covering both addressing
modes. Add a GART-adjusted variant so the local SDMA engine can address
remote GPU buffers.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add accessors for the local accelerator ID, addressing mode, and active
peer bitmap, plus a TLB flush helper for the NPA VM. Provide per-ring
size and byte offset helpers for both source-identification and
source-alias addressing modes.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Extend the UALink manager with data structures for cross-GPU interrupt
and TLB shootdown: per-ring state, per-peer channel pairs, and the
top-level remote container.
Declare the public init, teardown, and reset APIs.
Define ring buffer capacity, firmware command and status codes, SMN
mailbox register addresses, and the metadata and writeback buffer layouts
shared between driver and MPNHT firmware.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
ualink_ppod_setup_commit_store() updated accel_state via
check_ppod_state() but never called deactivate_local_vpod() when the
accelerator was already in READY state. A subsequent vpod config commit
would then call activate_local_vpod() again, resulting in
activate_accelerator() calling amdgpu_ualink_manager_start() a second
time on top of already-initialized NPA VM, causing a NULL pointer
dereference in rb_insert_color when drm_sched_entity nodes already in
the SDMA scheduler FIFO rb-tree were double-initialized.
Fix: call deactivate_accelerator() in ualink_ppod_setup_commit_store()
guarded by accel_state >= READY, under mgpu_info.mutex to serialize with
activate_local_vpod() in ualink_vpod_config_commit_store().
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Enable the UALink manager when the pod state transitions to active.
This would enable exchanging of memory across GPUs in a rack-scale
setup.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Map NPA VM to use the reserved VMID.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reserve MMHUB VMID 15 to be used for NPA VM.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Rework the VMID reservation logic by passing in the mask
of VMIDs (for both GFXHUB and MMHUB) available for it to
use. The VMID masks for both GFXHUB and MMHUB are initialized
in the respective ASIC specific files.
This also helps in reserving per-ASIC VMIDs. This is in
preparation for reserving a VMID for NPA added in the next
patch.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Initialize a NPA VM for managing NPA addresses on the exporter
GPU.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a address allocator, which uses DRM MM, for allocating and
deallocating NPA addresses for BOs exported/imported in a rack-scale
setup.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a new address space for NPA address management.
This is needed for sharing buffer objects across GPUs
with each running their own OS. The NPA address space
size can change across different HW generations so the
size is initialized during early init of the driver boot
up process.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
PSP may be disabled during driver initialization. So psp->cmd
is not allocated, so any psp_*() call would deref NULL. Also skip
if sysfs_init failed and ualink.info was never allocated.
Signed-off-by: Jinzhou Su <jinzhou.su@amd.com>
Acked-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Increase the upper bound of the PSP command polling sleep range to allow
more time for commands such as UAL_SET_STATION_CONFIG to complete before
the next status check.
Signed-off-by: Le Ma <le.ma@amd.com>
Reviewed-by: Hawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Helper function to check if psp xgmi ta is supported
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Enable or disable NPA address translation using PSP during accelerator
state transitions in and out of READY state.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set NPA translation configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Set the UALink station configuration to PSP in the commit store
callback.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set the UALink station configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Set the virtual pod configuration to PSP in the commit store callback.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set the virtual pod configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Set the physical pod configuration to PSP in the commit store callback.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set the physical pod configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Query the initial UALink configuration info from PSP at driver
initialization time.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To query UALink configuration information from PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
This patch introduces AMDGPU UALink manager core infrastructure
which will be responsible for sharing buffer objects across
multiple GPUs, with each GPU running their own OS, in a rack scale
setup. The core infrastructure of the ualink manager will include:
- Store BOs exported from a GPU.
- Store BOs imported by a GPU.
- Store connection state with different GPUs.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
<device>/ualink/stations: Directory for staging UALink station/port
configuration attributes:
- station_flags (currently only sets the number of port per station)
- lane_en_bitmap (hexadecimal bitmaps of lanes enabled per station)
- write "true" to commit to apply settings
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
<device>/ualink/config: Directory for staging virtual pod configuration
attributes. Write "true" to commit to apply settings
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
<device>/ualink/setup: Directory for staging physical pod setup
attributes. Write "true" to commit to apply settings
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
<device>/ualink/info: Directory contains current UALink attributes
- link_type: UALoE of UALink
- accel_state: unconfigured, configured, ready, active, error
- accel_id: Accelerator ID
- ppod_id: Physical pod ID (UUID)
- ppod_size: Number of accelerators in physical pod
- bandwidth: Estimated total bandwidth between pairs of accelerators
- latency: Estimated latency between pairs of accelerators
- vpod_id: Virtual pod ID
- vpod_size: Number of accelerators in virtual pod
- vpod_active_accels: List of all accelerator IDs in vpod
- local_accels: List of local accelerator IDs in vpod
- addr_mode: source-aliasing or source-identification
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add interface to get ASP ualink interface version.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The calls are redundant as GFXOFF is not there on GC v12.1.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Wire up the notify_ac_dc pp_funcs callback for swsmu instead of
special-casing is_support_sw_smu in the ACPI event handler. This keeps
AC/DC transition notification through the same callback path as used by
legacy powerplay and legacy-dpm backends.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Asad Kamal <asad.kamal@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
amdgpu_ttm_tt_get_usermm() returns gtt->usertask->mm. By the time KFD
frees its BOs from kfd_process_wq_release(), the process has long since
run exit_mm() and that pointer is NULL, so the cleanup below is simply
skipped on process teardown. mem->user_pages then leaks along with the
page references it still holds, or mem->range on HMM builds.
Check the USERPTR alloc flag instead. It is set when the BO is created
and stays valid no matter what the process does.
Signed-off-by: Perry Yuan <perry.yuan@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
need to pad before __u64 tcc_disabled_mask variable.
This patch fixes 64bit Kernel + 32 bit mesa combination. But at the same
time it will break 32bit Kernel(using this patch) + older 32bit mesa(not
using this patch).
This issue was discussd with alexander.deucher@amd.com,
christian.koenig@amd.com and pierre-eric.pelloux-prayer@amd.com.
Currently today 32 bit kernel + 32 bit userspace and 64 bit kernel and
64 bit userspace work. Mixed 64 bit kernel and 32 bit userspace is
currently broken. Since 32 bit kernel and userspace is probably pretty
rare these days and the data affected by this is not critical, Hence
we can go ahead with this patch.
Fixes: cf21e76a60 ("drm/amdgpu: return tcc_disabled_mask to userspace")
Signed-off-by: Yogesh Mohan Marimuthu <yogesh.mohanmarimuthu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
svm_migrate_vma_to_vram() is called for windows that are already fully
resident in the target GPU's VRAM. migrate_vma_setup() raises an MMU
notifier invalidate before it can discover there is nothing to collect,
and with retry faults enabled svm_range_cpu_invalidate_pagetables()
special cases only MMU_NOTIFY_UNMAP, so MMU_NOTIFY_MIGRATE is handled by
its default arm, reaching svm_range_evict() and then
svm_range_unmap_from_gpus().
The attempt therefore drops the window's GPU mapping and then returns
having migrated nothing, so the next fault on that window is escalated
into another migration instead of being satisfied by mapping alone.
Measured on MI300X (gfx942, SPX/NPS1, xnack+) under an HMM
oversubscription workload, 96 to 97% of to-VRAM migration attempts
collected zero pages, and the driver unmapped a window roughly 35 times
for every migration that moved data.
Test whether every page of the faulting window is already resident in
this node's VRAM and return early if it is. prange->actual_loc alone is
not sufficient: since commit a546a27684 ("drm/amdkfd: Use partial
migrations/mapping for GPU/CPU page faults in SVM") migration is per
window, so a range can report actual_loc == best_loc while part of it has
been evicted back to system memory. 3.3% of attempts were in exactly that
state and did have pages to move, so the per-page test is required rather
than the scalar alone. Instrumented over 100000 attempts on an earlier
run against a different driver build, the per-page test never skipped one
that had pages to move.
The actual_loc comparison is kept alongside the per-page test because
SVM_RANGE_VRAM_DOMAIN records only that a page is in some device's VRAM,
not which device's.
With this applied the fault driven phase of that workload falls from 270
to 348 s down to 45 to 48 s, across 9 patched and 7 unpatched runs with
no overlap at either oversubscription factor tested. page_in at the end
of that phase does not fall with it: 44.5 to 48.3M pages patched against
44.1 to 49.7M unpatched at factor 1.10, and 46.2 to 48.0M against 44.1 to
45.7M at factor 1.02. Those ranges overlap at 1.10, so the claim is that
no data is stranded, not that more moves. Unmaps per productive migration
fall from about 35 to about 2. Retry fault counts do not fall in any arm,
so this is not a refault loop; what changes is how often a fault
escalates into a migration.
The scan is bounded by the migration window, one granule on the fault
path and the whole range on the prefetch path. That is the same order as
the migrate_vma_setup() walk it avoids, and it returns at the first page
that is not resident, so the worst case is one extra pass over a range
that then migrates normally.
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
