Allow namespace moves to request a destination device name. Keep requested
names on the same literal-name path as the existing RDMA rename operation,
and keep teardown fallback naming on the trusted kernel-controlled path.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260716132316.1495242-4-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Prepare namespace moves for per-netns names. Check user-initiated moves for
destination-name conflicts before disabling the device, keep same-netns
moves as no-ops, and make teardown moves detach from the exiting namespace
even if fallback naming fails.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260716132316.1495242-3-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Prepare for per-netns RDMA device names by passing the target net
namespace through the name lookup and allocation helpers. Keep current
global uniqueness behaviour.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260716132316.1495242-2-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Add an optional mlx5 driver-namespace UMEM attribute on SRQ create so
userspace can supply the doorbell record umem explicitly, symmetric to
the CQ and QP sides. Resolve it inside mlx5_ib_db_map_user() and use it
as a private DBR page when present; otherwise take the existing UHW
share-or-pin path that preserves per-page DBR sharing across CQ/QP/SRQ
in the same process.
Add mlx5's first UVERBS_OBJECT_SRQ UAPI definition chain to attach the
new attr.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260701124015.64350-4-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Use the per-attribute UMEM helper to pin the SRQ buffer umem on demand.
ib_umem_get_attr_or_va() resolves the new CREATE_SRQ_BUF_UMEM attribute
when present and otherwise falls back to the existing UHW ucmd->buf_addr
VA, preserving the legacy behavior.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260701124015.64350-3-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Apply the per-attribute UMEM model to the SRQ create method. Add an
optional UMEM attribute that backs the SRQ WQE buffer, so userspace can
supply it as either a VA or a dma-buf through a single descriptor,
consistent with the CQ and QP create methods.
mlx5 is the only driver that pins an SRQ WQE buffer via umem; it maps a
single ucmd->buf_addr region through this attribute. No other driver
implements a user SRQ buffer, so none of them use the attribute.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260701124015.64350-2-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Implement returning to userspace applications PDNs of created PDs.
The PDN is used by applications that build work requests outside of the
rdma-core code base. The PDN is used to build work requests that require
mentioning the PD. The HW still ensures PD isolation using PDN attached
to MRs and WRs, therefore the PDN mentioned in the work request must match
the PDN of the used work queue. The work requests can fit only 16 bit PDNs.
Allow users to request short PDNs which are 16 bits.
The capability to request short PDN and get PDN is encoded in the response
of alloc user context IOCTL.
Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
Link: https://patch.msgid.link/20260717200839.495327-1-kotaranov@linux.microsoft.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
erdma_destroy_qp(), erdma_destroy_cq(), erdma_dereg_mr(), and
erdma_destroy_ah() returned early when erdma_post_cmd_wait() failed,
leaking the queue buffers, MTTs, doorbells and the STAG, QPN, CQN and AHN
identifiers. A command timeout clears ERDMA_CMDQ_STATE_OK_BIT and
permanently disables the command queue, so no retry can succeed; the RDMA
core keeps the object after a failed destructor and forced uverbs cleanup
then nulls the pointers, making the resources unreachable.
Warn on failure but release every software-owned resource and return
success, since during terminal destruction the hardware command result is
only diagnostic.
Fixes: 1550557717 ("RDMA/erdma: Add verbs implementation")
Link: https://patch.msgid.link/20260722-b4-qp-and-cq-memory-are-leaked-if-the-d-v1-1-97e223dc1c96@nvidia.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Acked-by: Cheng Xu <chengyou@linux.alibaba.com>
Since reset is actually controlled by the boolean flag rather than
long, switch to 'kstrtobool_from_user()' and use the latter for an
overall simplification of 'ocrdma_dbgfs_ops_write()'.
Signed-off-by: Dmitry Antipov <dmantipov@yandex.ru>
Link: https://patch.msgid.link/20260723071845.568718-1-dmantipov@yandex.ru
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Simplify 'bnxt_re_cc_config_set()' and 'cq_coal_cfg_write()'
by using the convenient 'kstrtou32_from_user()'.
Signed-off-by: Dmitry Antipov <dmantipov@yandex.ru>
Link: https://patch.msgid.link/20260723071629.568675-1-dmantipov@yandex.ru
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Currently EFA WRs support 16-bit request ID, this requires EFA to manage
a translation table to translate the IB WR ID from 64-bits to 16-bits
and translating it back on CQ completion.
Expose a new device capability to handle 64-bit request ID for SQ WRs
allowing userspace to directly post the 64-bit ID to the device.
Reviewed-by: Michael Margolin <mrgolin@amazon.com>
Reviewed-by: Tom Sela <tomsela@amazon.com>
Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
Link: https://patch.msgid.link/20260722113331.2515247-3-ynachum@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Add the support needed to propagate the user requested flags to config
the CQ/QP to support 64-bit SQ request ID to the device.
Reviewed-by: Michael Margolin <mrgolin@amazon.com>
Reviewed-by: Tom Sela <tomsela@amazon.com>
Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
Link: https://patch.msgid.link/20260722113331.2515247-2-ynachum@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
bnxt_re_mmap() rejects VM_WRITE for the DBR_PAGE and TOGGLE_PAGE mmap
flags, but a read-only mapping can still retain VM_MAYWRITE. nd later
be upgraded with mprotect(PROT_WRITE). This can bypass the write check
that only runs at mmap time.
Clear VM_MAYWRITE before vm_insert_page() in the shared DBR/toggle-page
branch, matching the existing policy that userspace writes are not
expected for these pages.
Fixes: ea22248578 ("RDMA/bnxt_re: Update alloc_page uapi for pacing")
Suggested-by: Yousef Alhouseen <alhouseenyousef@gmail.com>
Signed-off-by: Selvin Xavier <selvin.xavier@broadcom.com>
Link: https://patch.msgid.link/20260721115440.24021-5-selvin.xavier@broadcom.com
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Implement completion counters for the EFA device. Each completion
counter is backed by two EFA event counters, one for success
completions and one for error completions.
The driver creates umem for counters from private descriptor ioctl
attributes using core utility.
Read operations are not implemented as the counter values are accessed
directly from userspace through the mapped memory.
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
Link: https://patch.msgid.link/20260722083603.30334-7-mrgolin@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Track completion counter objects in the resource tracking database so
they are visible through the rdma netlink interface. The rdma tool
displays the comp_cntr count in the resource summary.
Add RDMA_RESTRACK_COMP_CNTR type, embed rdma_restrack_entry in
ib_comp_cntr, and add the res_to_dev mapping. Register the resource
on create and remove it on destroy.
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
Link: https://patch.msgid.link/20260722083603.30334-5-mrgolin@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Add a dedicated query interface for completion counter capabilities
via UVERBS_METHOD_QUERY_COMP_CNTR_CAPS on the device object.
The query returns the maximum number of counters, maximum counter
value, and a bitmask of supported QP attach operations. Each field
is an optional ioctl attribute, allowing userspace to request only
the capabilities it needs.
Drivers implement the query_comp_cntr_caps operation to report
device-specific capabilities.
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
Link: https://patch.msgid.link/20260722083603.30334-4-mrgolin@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Reject comp_cntr destroy while it is attached to any QP. Track
attachments using an xarray in ib_qp keyed by the attach op_mask.
Use op bitmask to reject overlapping attaches early.
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
Link: https://patch.msgid.link/20260722083603.30334-3-mrgolin@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Add core infrastructure for Completion Counters, a light-weight
alternative to polling CQ for tracking operation completions.
Define the UVERBS_OBJECT_COMP_CNTR ioctl object with create, destroy,
modify and read methods for both success and error counters. Add a QP
attach method on the QP object to associate a completion counter with a
queue pair.
Add ib_comp_cntr struct, ib_comp_cntr_attach_attr, device ops, and
DECLARE_RDMA_OBJ_SIZE for driver object allocation.
Only userspace Completion Counters are supported at this stage.
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
Link: https://patch.msgid.link/20260722083603.30334-2-mrgolin@amazon.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When accessing a PD via the netlink path the only synchronization
mechanism for the said PD is rdma_restrack_get().
Currently, rdma_restrack_del() is invoked at the end of
ib_dealloc_pd_user(), which is too late, since by that point
vendor-specific resources associated with the PD might already be
freed. This can leave a short window where the PD remains accessible
through restrack, leading to a potential use-after-free.
Fix this by moving the rdma_restrack_begin_del() call to the start of
ib_dealloc_pd_user(), ensuring that the PD is removed from restrack
before its internal resources are released. This guarantees that no new
users hold references to a PD that is in the process of destruction.
In addition, this change preserves the intended inverted order
between create and destroy routines: resources are added to
restrack at the end of successful creation, and hence shall be removed
from the restrack first thing during the destruction flow, which keeps
the lifecycle management consistent and predictable.
Fixes: 91a7c58fce ("RDMA: Restore ability to fail on PD deallocate")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-8-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When accessing a dmah via the netlink path the only synchronization
mechanism for the said dmah is rdma_restrack_get().
Currently, rdma_restrack_del() is invoked at the end of
uverbs_free_dmah(), which is too late, since by that point
vendor-specific resources associated with the dmah might already be
freed. This can leave a short window where the dmah remains accessible
through restrack, leading to a potential use-after-free.
Fix this by moving the rdma_restrack_begin_del() call to the start of
uverbs_free_dmah(), ensuring that the dmah is removed from restrack
before its internal resources are released. This guarantees that no new
users hold references to a dmah that is in the process of destruction.
In addition, this change preserves the intended inverted order
between create and destroy routines: resources are added to
restrack at the end of successful creation, and hence shall be removed
from the restrack first thing during the destruction flow, which keeps
the lifecycle management consistent and predictable.
Fixes: d83edab562 ("RDMA/core: Introduce a DMAH object and its alloc/free APIs")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-7-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When accessing a CQ via the netlink path the only synchronization
mechanism for the said CQ is rdma_restrack_get().
Currently, rdma_restrack_del() is invoked at the end of
ib_free_cq(), which is too late, since by that point
vendor-specific resources associated with the CQ might already be
freed. This can leave a short window where the CQ remains accessible
through restrack, leading to a potential use-after-free.
Fix this by moving the rdma_restrack_del() call to be before the freeing
of the vendor-specific resources ensuring that the CQ is removed from
restrack before its internal resources are released.
This guarantees that no new users hold references to a CQ that is in
the process of destruction.
Fixes: 43d781b9fa ("RDMA: Allow fail of destroy CQ")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-6-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When accessing a counter via the netlink path the only synchronization
mechanism for the said counter is rdma_restrack_get().
Currently, rdma_restrack_del() is invoked at the end of
counter_release(), which is too late, since by that point
vendor-specific resources associated with the counter might already be
freed. This can leave a short window where the counter remains
accessible through restrack, leading to a potential use-after-free.
Fix this by moving the rdma_restrack_del() call to be before the
freeing of the vendor-specific resources, ensuring that the counter is
removed from restrack before its internal resources are released.
This guarantees that no new users hold references to a counter that is
in the process of destruction.
Fixes: 99fa331dc8 ("RDMA/counter: Add "auto" configuration mode support")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-5-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When accessing a SRQ via the netlink path the only synchronization
mechanism for the said SRQ is rdma_restrack_get().
Currently, rdma_restrack_del() is invoked at the end of
ib_destroy_srq_user(), which is too late, since by that point
vendor-specific resources associated with the SRQ might already be
freed. This can leave a short window where the SRQ remains accessible
through restrack, leading to a potential use-after-free.
Fix this by moving the rdma_restrack_begin_del() call to the start of
ib_destroy_srq_user(), ensuring that the SRQ is removed from restrack
before its internal resources are released. This guarantees that no new
users hold references to a SRQ that is in the process of destruction.
In addition, this change preserves the intended inverted order
between create and destroy routines: resources are added to
restrack at the end of successful creation, and hence shall be removed
from the restrack first thing during the destruction flow, which keeps
the lifecycle management consistent and predictable.
Fixes: 48f8a70e89 ("RDMA/restrack: Add support to get resource tracking for SRQ")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-4-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When accessing a CQ via the netlink path the only synchronization
mechanism for the said CQ is rdma_restrack_get().
Currently, rdma_restrack_del() is invoked at the end of
ib_destroy_cq_user(), which is too late, since by that point
vendor-specific resources associated with the CQ might already be
freed. This can leave a short window where the CQ remains accessible
through restrack, leading to a potential use-after-free.
Fix this by moving the rdma_restrack_begin_del() call to the start of
ib_destroy_cq_user(), ensuring that the CQ is removed from restrack
before its internal resources are released. This guarantees that no new
users hold references to a CQ that is in the process of destruction.
In addition, this change preserves the intended inverted order
between create and destroy routines: resources are added to
restrack at the end of successful creation, and hence shall be removed
from the restrack first thing during the destruction flow, which keeps
the lifecycle management consistent and predictable.
Fixes: 08f294a152 ("RDMA/core: Add resource tracking for create and destroy CQs")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-3-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
When querying a QP via the netlink flow the only synchronization
mechanism for the said QP is rdma_restrack_get(), meanwhile during the
QP destroy path rdma_restrack_del() is called at the end of the
ib_destroy_qp_user() function which is too late, since by then the
vendor specific resources for said QP would already be destroyed, and
till the rdma_restrack_del() is called this QP can still be accessed,
which could cause the use after free below.
Fix this by moving the rdma_restrack_begin_del() to the start of the
ib_destroy_qp_user(), which in turn waits for all usages of the QP to be
done then removes it from the database to prevent access to it while it
is being destroyed.
RIP: 0010:ib_query_qp+0x15/0x50 [ib_core]
Code: 48 83 05 5d 8e b9 ff 01 eb b5 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 c7 46 40 00 00 00 00 48 c7 46 78 00 00 00 00 <48> 8b 07 48 8b 80 88 01 00 00 48 85 c0 74 1a 48 83 05 54 91 b9 ff
RSP: 0018:ff11000108a8f2f0 EFLAGS: 00010202
RAX: 0000000000000000 RBX: ff11000108a8f370 RCX: ff11000108a8f370
RDX: 0000000000000000 RSI: ff11000108a8f3d8 RDI: 0000000000000000
RBP: ff1100010de5a000 R08: 0000000000000e80 R09: 0000000000000004
R10: ff110001057a604c R11: 0000000000000000 R12: ff11000108a8f370
R13: ff110001090e8000 R14: 0000000000000000 R15: ff110001057a602c
FS: 00007f2ffd8db6c0(0000) GS:ff110008dc90b000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000010b9a7004 CR4: 0000000000373eb0
Call Trace:
<TASK>
mlx5_ib_gsi_query_qp+0x21/0x50 [mlx5_ib]
mlx5_ib_query_qp+0x689/0x9d0 [mlx5_ib]
ib_query_qp+0x35/0x50 [ib_core]
fill_res_qp_entry_query.isra.0+0x47/0x280 [ib_core]
? __wake_up+0x40/0x50
? netlink_broadcast_filtered+0x15a/0x550
? kobject_uevent_env+0x562/0x710
? ep_poll_callback+0x242/0x270
? __nla_put+0xc/0x20
? nla_put+0x28/0x40
? nla_put_string+0x2e/0x40 [ib_core]
fill_res_qp_entry+0x138/0x190 [ib_core]
res_get_common_dumpit+0x4a5/0x800 [ib_core]
? fill_res_qp_entry_query.isra.0+0x280/0x280 [ib_core]
nldev_res_get_qp_dumpit+0x1e/0x30 [ib_core]
netlink_dump+0x16f/0x450
__netlink_dump_start+0x1ce/0x2e0
rdma_nl_rcv_msg+0x1d3/0x330 [ib_core]
? nldev_res_get_qp_raw_dumpit+0x30/0x30 [ib_core]
rdma_nl_rcv_skb.constprop.0.isra.0+0x108/0x180 [ib_core]
rdma_nl_rcv+0x12/0x20 [ib_core]
netlink_unicast+0x255/0x380
? __alloc_skb+0xfa/0x1e0
netlink_sendmsg+0x1f3/0x420
__sock_sendmsg+0x38/0x60
____sys_sendmsg+0x1e8/0x230
? copy_msghdr_from_user+0xea/0x170
___sys_sendmsg+0x7c/0xb0
? __futex_wait+0x95/0xf0
? __futex_wake_mark+0x40/0x40
? futex_wait+0x67/0x100
? futex_wake+0xac/0x1b0
__sys_sendmsg+0x5f/0xb0
do_syscall_64+0x55/0xb90
entry_SYSCALL_64_after_hwframe+0x4b/0x53
Fixes: 514aee660d ("RDMA: Globally allocate and release QP memory")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-2-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Add rdma_restrack_abort_del(), rdma_restrack_begin_del() and
rdma_restrack_commit_del() functions to allow deleting a resource from
the xarray to effectively prevent future access to it and wait for all
current users to finish while preserving its index in the xarray to
allow to re-insert it if needed with guaranteed success.
This is a preparatory change for subsequent patches in the series
which will use these functions to fix the cleanup flow.
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
Link: https://patch.msgid.link/20260713-restrack-uaf-fix-resub-v2-1-bbe8bb270d51@nvidia.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
mana_table_store_ud_qp() publishes a QP at its send-queue id before
inserting the receive-queue id, dropping the XArray lock between the two
xa_insert_irq() calls. A concurrent completion handler can look up the QP
and take a transient reference. When the second insertion fails, the
rollback erased only the send-queue entry and returned, leaving both the
initial table reference and the transient reference outstanding while RDMA
core frees the QP, causing a use-after-free.
Drain the reference as normal destruction does: drop the initial reference
and wait for qp->free, releasing the QP only after every concurrent lookup
returns its reference.
Fixes: 8001e9257e ("RDMA/mana_ib: extend mana QP table")
Link: https://patch.msgid.link/20260721-if-mana-table-store-qp-qids-partiall-v1-1-8fb3d2d2b559@nvidia.com
Reviewed-by: Konstantin Taranov <kotaranov@microsoft.com>
Reviewed-by: Long Li <longli@microsoft.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Update the CQE structure to include hardware timestamp.
When firmware supports RDMA completion timestamps,
the hardware populates the timestamp field.
Co-developed-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
Link: https://patch.msgid.link/20260610154216.712374-5-abhijit.gangurde@amd.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Enable user space applications to access the PHC state page when
firmware RDMA completion timestamp is supported.
This mapping allows user space to convert RDMA completion timestamps
to system wall time without kernel transitions, minimizing latency
overhead. Applications can directly read the PHC state through mmap,
enabling efficient timestamp correlation for precision timing
applications.
Co-developed-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
Link: https://patch.msgid.link/20260610154216.712374-4-abhijit.gangurde@amd.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Add a page associated with the PHC that can be mapped to user space,
allowing applications to access hardware timestamp information.
In order to synchronize between kernel and user space, a sequence
number is incremented at the beginning and end of each update.
An odd number means the data is being updated while an even number
means the update is complete. To guarantee that the data structure
was accessed atomically, user space will:
repeat:
seq1 = <read sequence>
goto <repeat> if odd
<read PHC state>
seq2 = <read sequence>
if seq1 != seq2 goto repeat
This mechanism acts as a guard against reading invalid state during
concurrent updates.
Co-developed-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
Link: https://patch.msgid.link/20260610154216.712374-3-abhijit.gangurde@amd.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Currently, the driver only registers the PTP Hardware Clock (PHC) if
Ethernet hardware timestamping is supported. Update the registration
logic to register the PHC if the device supports either Ethernet
hardware timestamping or RDMA completion timestamping.
Co-developed-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
Link: https://patch.msgid.link/20260610154216.712374-2-abhijit.gangurde@amd.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Each CEQ interrupt handler only schedules eqc->tasklet. The tasklet calls
erdma_ceq_completion_handler(), which reads the DMA-coherent EQ ring
through get_next_valid_eqe() and updates eq->dbrec through notify_eq().
erdma_ceqs_uninit() frees each CEQ IRQ and then destroys its EQ.
free_irq() prevents another hard IRQ and waits for an in-flight handler,
but it does not drain a tasklet that the handler already scheduled. The
tasklet can therefore access eq->qbuf or eq->dbrec after
erdma_eq_destroy() frees them.
Clearing ceq_cb->ready does not synchronize with a tasklet that already
passed the check at the start of erdma_ceq_completion_handler().
Kill the tasklet after free_irq(), when no handler can schedule it again,
and before erdma_ceq_uninit_one() releases the EQ buffers.
Fixes: f2a0a630b9 ("RDMA/erdma: Add event queue implementation")
Co-developed-by: Ijae Kim <ae878000@gmail.com>
Signed-off-by: Ijae Kim <ae878000@gmail.com>
Signed-off-by: Myeonghun Pak <mhun512@gmail.com>
Link: https://patch.msgid.link/20260721082545.47395-1-mhun512@gmail.com
Acked-by: Cheng Xu <chengyou@linux.alibaba.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
srpt_handle_cmd() maps the initiator-supplied srp_cmd->task_attr into
cmd->sam_task_attr, but then hands a hardcoded TCM_SIMPLE_TAG to
target_init_cmd().
Pass the already mapped cmd->sam_task_attr instead, so target core sees the
attribute the initiator requested.
Fixes: 9474b04313 ("ib_srpt: Convert I/O path to target_submit_cmd + drop legacy ioctx->kref")
Link: https://patch.msgid.link/20260721-b4-scsi-ordering-violation-due-to-hardc-v1-1-07205aab71bb@nvidia.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
RDMA_NLDEV_ATTR_STAT_HWCOUNTERS is a nested attribute whose children are
consumed directly with nla_get_u32(). The top-level policy validates only
the container, so it does not establish the fixed shape of each child.
Require every child payload to be exactly one u32 before reading it.
Fixes: 3c3c1f1416 ("RDMA/nldev: Allow optional-counter status configuration through RDMA netlink")
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Link: https://patch.msgid.link/20260720114918.70323-1-pengpeng@iscas.ac.cn
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Enable the robust udata contract by setting uverbs_robust_udata and
adding proper input validation and output handling to all verbs that
accept struct ib_udata.
For verbs with no driver request or response struct, add
ib_no_udata_io(). For create_ah, which already responds with
ionic_ah_resp, add the missing input validation via
ib_is_udata_in_empty().
Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
Link: https://patch.msgid.link/20260717092345.2533564-1-abhijit.gangurde@amd.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Enable the uverbs robust udata interface in mana_ib by setting
uverbs_robust_udata and converting the driver to the new udata
handling model.
Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
mlx5r_umr_alloc_xlt() allocates physically contiguous scratch buffers
that are DMA mapped only in the DMA_TO_DEVICE direction. kmalloc()
provides the required contiguity and alignment for these sizes while
preserving the existing GFP allocation policy.
The emergency translation buffer has the same requirements. Convert all
of these UMR buffers to kmalloc() and release them with kfree(), which no
longer requires the caller to supply the allocation order.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-4-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
usnic_uiom_get_pages() uses a page-sized array of struct page pointers
as temporary storage for pin_user_pages(). Nothing requires the array to
come directly from the page allocator.
Use kmalloc() for the array and kfree() after the pinning loop.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-3-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
mlx4_alloc_priv_pages() allocates a zeroed, page-sized buffer for a
DMA-to-device page list. kmalloc() provides the required physical
contiguity, and a PAGE_SIZE allocation retains the alignment needed to
keep the list within one page.
Use kzalloc() for the buffer and kfree() on the error and teardown paths.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-2-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
c4iw_reg_user_mr() allocates a page-sized temporary array of DMA
addresses while programming a PBL. The array has no page-specific
requirements, so allocate it with kmalloc() and release it with kfree().
This avoids the casts required by the page allocator and lets the free
operation derive the allocation size from the object.
Link: https://patch.msgid.link/20260715-get_pages-to-kmalloc-v1-1-b0b7fce288be@nvidia.com
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
The irdma driver has been audited to confirm that:
1. Methods which do not accept udata input perform an explicit
check for no (or zero value) input.
2. Methods which do accept input perform the correct validation
to ensure that additional udata beyond the kernel's current
ABI definition is zero, and to enforce the required minimum
length.
3. Methods which do not return udata responses use the proper
helper.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-7-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
The irdma driver maintains backward compatibility with the
legacy i40iw userspace provider by checking the length of
the user response buffer in irdma_create_qp.
Previously, the check relied on udata->outlen < sizeof(uresp).
That is technically okay since there have only ever been two
sizes for the resp struct (legacy and current). However, it
would be a problem if the resp struct is ever expanded in
the future because it would end up triggering the legacy
fallback path for non-legacy irdma providers that just haven't
moved over to the newer expanded struct yet.
Fix this by explicitly checking for the exact legacy resp size.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-6-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Replace the manual udata input copy and validation during
QP creation with the robust helper.
The irdma driver is backwards compatible with the legacy
i40iw userspace provider. The current create_qp ABI contains
two 8 byte fields. The legacy i40iw ABI was the same but
also contained two additional fields which were never actually
used. Furthermore, the i40iw userspace provider never explicitly
zero-initialized those extra fields, so there is a chance that
existing binaries are passing non-zero garbage values down
to the kernel.
Previously, the irdma driver only copied out the first 16
bytes and did not have any check for the rest of the buffer
being zero, so that additional garbage didn't matter.
By switching to ib_copy_validate_udata_in(), we will now be
checking to ensure that data beyond the kernel's definition
of the request is all zero.
In order to avoid breaking legacy binaries, we therefore need
to increase the request structure size to cover those garbage
fields.
- Legacy binaries will continue to pass down a 32 byte request,
with the driver copying the entire 32 bytes out but ignoring
the second 16 bytes, just as before.
- Newer binaries will pass down the normal 16 byte request. The
ib_copy_validate_udata_in() call will allow this to succeed
because we use user_compl_ctx as our minimum length (16 bytes).
- If the request is ever extended, the new fields would be
added after the "don't use" fields and would work as per
the normal uAPI mechanism.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-5-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Replace the use of ib_copy_from_udata() with
ib_copy_validate_udata_in() where applicable.
For each modified call site, the last argument of
ib_copy_validate_udata_in() was determined by taking
the last member of the ABI struct as per its original
definition (i.e., when it was first committed).
Some methods like irdma_create_cq required special care
because the last member of the current ABI def is beyond
that of the legacy i40iw's ABI def which we need to
remain compatible with. In some other cases like modify_qp,
the legacy i40iw provider never provided any udata at all
so the validation is only performed if inlen > 0.
irdma_create_qp is more challenging because the legacy ABI
was actually larger but the additional fields were never used,
and even worse, never initialized in the provider. This will
be handled in a followup commit.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-4-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Methods that may accept udata input but do not provide a
udata response should use the ib_respond_empty_udata()
helper to ensure that user response buffers are cleared.
Since the ib_respond_empty_udata() call itself can fail if
the user intentionally provides a bogus output buffer, it is
called at the beginning of the method to fail early before
mutating any state that would be difficult to unwind.
Additionally, add missing bounds validation for udata->outlen
in irdma_create_srq() to ensure it is large enough to hold the
response struct as per its original (and so far, only) definition.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-3-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Several methods do not accept udata input and do not provide
a udata response. Use the ib_no_udata_io helper to check that
the input buffers are empty and to zero fill any user response
buffers. For methods that do provide a response, enforce the
input buffer is empty using ib_is_udata_in_empty.
The irdma rdma-core provider as well as the legacy i40iw
provider were both checked to ensure they never passed any
udata to these ops.
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260713171257.3131493-2-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Previously, the user QP creation path would only attempt to
populate iwqp->iwpbl if the user-provided req.user_wqe_bufs
field was non-zero. The problem is that iwqp->iwpbl is
unconditionally dereferenced later on in irdma_setup_virt_qp.
While there was a check for iwqp->iwpbl != NULL, this check
would only occur if req.user_wqe_bufs was non-zero. The end
result is that a user could send a zero user_wqe_bufs value
and trigger a null ptr deref.
Fix this by unconditionally calling irdma_get_pbl and bailing
if it fails, similar to the CQ and SRQ paths.
Fixes: b48c24c2d7 ("RDMA/irdma: Implement device supported verb APIs")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
