Replace the stub implementation of TLB shootdown message with
the actual implementation.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add interrupt handling to process the NPA remote interrupts.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Map NPA addresses into importer's application page tables
using a unique pairing of PTE fields.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
This patch adds connection reset handling. There can be
two case which signal connection reset:
1. Receiving a HELLO message from a remote GPU, when the
connection state is already setup, signals the remote
GPU underwent a reset.
2. If no response received for a NPA-REQ/NPA_REVOKE
message.
In either of the two case, we cleanup all exported and
imported ualink handles exchanged with the remote GPU.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add implementation for cleaning up ualink handles imported
on the importer GPU and the corresponding xarray entries.
This includes cleaning up the NPA BO dmabuf and freeing the
NPA BO.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add implementation for freeing up exporter GPU's UALink
handles and the corresponding xarray entries. Before freeing,
the exporter GPU would send the NPA-REVOKE message to all the
importers and wait for the NPA-RELEASE messages to come back
before freeing the xarray entry.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Convert amdgpu_ualink_invalidate_import_mappings() to lock the BO and
every client VM root PD in a single drm_exec transaction instead of
borrowing the caller's reservation ticket via dma_resv_locking_ctx().
drm_exec owns its ww_acquire_ctx (exec.ticket) for the lifetime of the
transaction, so amdgpu_vm_handle_moved() now uses &exec.ticket.
Also, drop the explicit dma_resv_reserve_fences call as amdgpu_vm_lock_pd()
already reserves 2 + num_fences on the same VM root PD.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
This patch adds implementation to revoke exported memory.
The exporter GPU will send a NPA-REVOKE message to inform the
importer GPUs to stop using the exported memory. Once the importers
stop using the exporter memory, they would send back a NPA-RELEASE
message to inform the exporters to free the exporter memory.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The ualink handle import process involves NPA protocol
message exchange between the exporting and the importing
GPUs in the rack scale setup. The process is as follows:
1. On the importing GPU, check if connection is already setup
with the exporting GPU. If not, then exchange HELLO/HELLO_ACK
messages to setup the connection.
2. Once the connection is setup, then exchange NPA-REQ/NPA-RSP
messages with the exporter GPU to get the NPA address and size
of the BO associated with the ualink handle.
3. On the exporter GPU, upon receipt of NPA-REQ message, do basic
validation to check the ualink handle is valid. If not, send a
NPA-FAIL message back. If its valid, then using the address
allocator, allocate a NPA address, map it into the NPA VM and
send back the address and size in the NPA-RSP to the importing GPU.
4. On the importer GPU, upon receipt of NPA-RSP message, allocate a NPA
BO at the NPA address received in the NPA-RSP message.
5. Finally, generate the corresponding DMABuf for the NPA BO and return
the dmabuf to user-space.
6. We are using refcount to keep track of the importer/exporter xarray
entries. The cleanup functions for these are added in the subsequent
patches.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add functions for managing connection state between GPUs
before import/export of ualink handles occurs.
These functions handle the lifecycle of connections between
GPUs,including handling GPU resets, in the NPA-based memory
sharing model.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
This patch adds the implementation for exporting a ualink
handle associated with a BO. If the BO doesn't have a
ualink handle generated yet, we first allocate a unique
ualink handle, add it to a lookup table and return the
handle to user-space.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a ualink handle to the amdgpu_bo struct. The ualink
handle is a 128-bit unique identifier associated with a
BO and is shared across the network to facilitate NPA based
memory sharing across GPUs in a rack scale setup.
We are only storing the low 64 bits of that handle in the BO
struct as that is gauranteed to be unique.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
NPA DMA-Bufs can only be imported on the same device as the
original BO. In source-identification mode NPA addresses are
importer-specific. Each GPU needs to import the UALink handle
by themselves. Sharing the NPA addresses between devices is
meaningless and potentially dangerous. Put controls to ensure
this is not violated.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Create a drm client for storing the GEM handles corresponding
to NPA BOs. We always want to keep one GEM handle alive for
the NPA BOs during the lifetime of the BO.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a use_lsdma flag to amdgpu_ualink_remote and set it during SDMA
entity initialization. When use_lsdma is set, use amdgpu_lsdma_copy_mem()
instead of SDMA IBs to write the remote command ring data, wptr and
doorbell to their respective NPA GART addresses.
This provides an alternative low-latency path for sending UALink remote
interrupt commands via LSDMA, avoiding SDMA ring/IB overhead for these
small NPA writes.
v2: fix 32 bit build (Alex)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Register the UALink IRQ source with the IH subsystem. Hardware IH
entries are re-queued onto a dedicated soft ring and drained by a work
queue handler. Add the soft ring, worker, and delegate helper to the
IRQ layer.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Sequence the setup steps into a single init path with rollback on
failure, and a teardown that halts firmware first. Hook both paths into
the virtual pod commit so resources rebuild on reconfiguration.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Expose the two public send operations for remote TLB invalidation and
remote interrupt delivery. Set up per-peer GART entries covering ring
buffers, write pointer pages, and doorbells for every active remote GPU.
v2: fix 32 bit build (Ivan)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Compute the writeback buffer addresses for a ring, send a configuration
command to firmware, and mark the ring ready on acknowledgement.
Configuration is deferred to first use.
v2: fix 32 bit build (Ivan)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add 16-dword packet layouts for TLB shootdown, remote interrupt, and
writeback address update. The dispatch path issues three sequential SDMA
copies (ring entry, write pointer, doorbell) and polls the firmware
writeback buffer for the acknowledged sequence number.
v2: fix 32 bit build (Alex)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Allocate pinned memory for firmware metadata, ring buffers, and write
and read pointer pages. Populate per-GPU metadata entries and send them
to MPNHT firmware via SMN mailbox, then poll for a ready acknowledgement.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add helpers to insert and remove NPA addresses in the GART page table
without a TTM buffer object, allowing the local SDMA engine to reach
remote GPU ring buffers, write pointer pages, and doorbell addresses.
v2: fix 32 bit build (Alex)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Allocate one GPU scheduler entity per SDMA instance for remote command
dispatch. Commands to the same peer are serialized through one entity;
different peers can be targeted concurrently by different SDMA engines.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Populate the NPA VM so remote GPUs can access local ring buffers and
write pointer pages at their reserved NPA addresses. Teardown waits for
DMA fences and issues a heavyweight TLB flush.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a ring buffer type enum and address formula for the reserved NPA
space between source and destination GPUs, covering both addressing
modes. Add a GART-adjusted variant so the local SDMA engine can address
remote GPU buffers.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add accessors for the local accelerator ID, addressing mode, and active
peer bitmap, plus a TLB flush helper for the NPA VM. Provide per-ring
size and byte offset helpers for both source-identification and
source-alias addressing modes.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Extend the UALink manager with data structures for cross-GPU interrupt
and TLB shootdown: per-ring state, per-peer channel pairs, and the
top-level remote container.
Declare the public init, teardown, and reset APIs.
Define ring buffer capacity, firmware command and status codes, SMN
mailbox register addresses, and the metadata and writeback buffer layouts
shared between driver and MPNHT firmware.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
ualink_ppod_setup_commit_store() updated accel_state via
check_ppod_state() but never called deactivate_local_vpod() when the
accelerator was already in READY state. A subsequent vpod config commit
would then call activate_local_vpod() again, resulting in
activate_accelerator() calling amdgpu_ualink_manager_start() a second
time on top of already-initialized NPA VM, causing a NULL pointer
dereference in rb_insert_color when drm_sched_entity nodes already in
the SDMA scheduler FIFO rb-tree were double-initialized.
Fix: call deactivate_accelerator() in ualink_ppod_setup_commit_store()
guarded by accel_state >= READY, under mgpu_info.mutex to serialize with
activate_local_vpod() in ualink_vpod_config_commit_store().
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Enable the UALink manager when the pod state transitions to active.
This would enable exchanging of memory across GPUs in a rack-scale
setup.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Map NPA VM to use the reserved VMID.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reserve MMHUB VMID 15 to be used for NPA VM.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Rework the VMID reservation logic by passing in the mask
of VMIDs (for both GFXHUB and MMHUB) available for it to
use. The VMID masks for both GFXHUB and MMHUB are initialized
in the respective ASIC specific files.
This also helps in reserving per-ASIC VMIDs. This is in
preparation for reserving a VMID for NPA added in the next
patch.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Initialize a NPA VM for managing NPA addresses on the exporter
GPU.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a address allocator, which uses DRM MM, for allocating and
deallocating NPA addresses for BOs exported/imported in a rack-scale
setup.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Add a new address space for NPA address management.
This is needed for sharing buffer objects across GPUs
with each running their own OS. The NPA address space
size can change across different HW generations so the
size is initialized during early init of the driver boot
up process.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
PSP may be disabled during driver initialization. So psp->cmd
is not allocated, so any psp_*() call would deref NULL. Also skip
if sysfs_init failed and ualink.info was never allocated.
Signed-off-by: Jinzhou Su <jinzhou.su@amd.com>
Acked-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Increase the upper bound of the PSP command polling sleep range to allow
more time for commands such as UAL_SET_STATION_CONFIG to complete before
the next status check.
Signed-off-by: Le Ma <le.ma@amd.com>
Reviewed-by: Hawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Helper function to check if psp xgmi ta is supported
Signed-off-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Enable or disable NPA address translation using PSP during accelerator
state transitions in and out of READY state.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set NPA translation configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Set the UALink station configuration to PSP in the commit store
callback.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set the UALink station configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Set the virtual pod configuration to PSP in the commit store callback.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set the virtual pod configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Set the physical pod configuration to PSP in the commit store callback.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To set the physical pod configuration to PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Query the initial UALink configuration info from PSP at driver
initialization time.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
To query UALink configuration information from PSP.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
