kernel/linux
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ synced 2026-11-10 20:48:12 +00:00
Code Issues Releases Packages Activity
linux/io_uring/bpf-ops.c

275 lines
6.1 KiB
Raw Permalink Normal View History

io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: implement bpf ops registration Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/1f46bffd76008de49cbafa2ad77d348810a4f69e.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:41 +00:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: implement bpf ops registration Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/1f46bffd76008de49cbafa2ad77d348810a4f69e.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:41 +00:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: restrict ctx access to BPF BPF programs should have no need in looking into struct io_ring_ctx, if anything, most of such cases would be anti patterns like looking up ring indices directly via the context. Replace it with a new empty structure, which is just an alias to struct io_ring_ctx. It'll create a new BTF type and fail verification if a BPF program tries to access it (beyond the first byte). It'll also give more flexibility for the future, and otherwise it can be made aligned with io_ring_ctx as before with struct groups if ever needed or extended in a different way. Fixes: d0e437b76bd3c ("io_uring/bpf-ops: implement loop_step with BPF struct_ops") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/5f6ca3649e9e0bae8667db4357e28dd00cd07901.1780394491.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-06-02 11:08:25 +01:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: restrict ctx access to BPF BPF programs should have no need in looking into struct io_ring_ctx, if anything, most of such cases would be anti patterns like looking up ring indices directly via the context. Replace it with a new empty structure, which is just an alias to struct io_ring_ctx. It'll create a new BTF type and fail verification if a BPF program tries to access it (beyond the first byte). It'll also give more flexibility for the future, and otherwise it can be made aligned with io_ring_ctx as before with struct groups if ever needed or extended in a different way. Fixes: d0e437b76bd3c ("io_uring/bpf-ops: implement loop_step with BPF struct_ops") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/5f6ca3649e9e0bae8667db4357e28dd00cd07901.1780394491.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-06-02 11:08:25 +01:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: restrict ctx access to BPF BPF programs should have no need in looking into struct io_ring_ctx, if anything, most of such cases would be anti patterns like looking up ring indices directly via the context. Replace it with a new empty structure, which is just an alias to struct io_ring_ctx. It'll create a new BTF type and fail verification if a BPF program tries to access it (beyond the first byte). It'll also give more flexibility for the future, and otherwise it can be made aligned with io_ring_ctx as before with struct groups if ever needed or extended in a different way. Fixes: d0e437b76bd3c ("io_uring/bpf-ops: implement loop_step with BPF struct_ops") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/5f6ca3649e9e0bae8667db4357e28dd00cd07901.1780394491.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-06-02 11:08:25 +01:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: restrict ctx access to BPF BPF programs should have no need in looking into struct io_ring_ctx, if anything, most of such cases would be anti patterns like looking up ring indices directly via the context. Replace it with a new empty structure, which is just an alias to struct io_ring_ctx. It'll create a new BTF type and fail verification if a BPF program tries to access it (beyond the first byte). It'll also give more flexibility for the future, and otherwise it can be made aligned with io_ring_ctx as before with struct groups if ever needed or extended in a different way. Fixes: d0e437b76bd3c ("io_uring/bpf-ops: implement loop_step with BPF struct_ops") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/5f6ca3649e9e0bae8667db4357e28dd00cd07901.1780394491.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-06-02 11:08:25 +01:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: restrict ctx access to BPF BPF programs should have no need in looking into struct io_ring_ctx, if anything, most of such cases would be anti patterns like looking up ring indices directly via the context. Replace it with a new empty structure, which is just an alias to struct io_ring_ctx. It'll create a new BTF type and fail verification if a BPF program tries to access it (beyond the first byte). It'll also give more flexibility for the future, and otherwise it can be made aligned with io_ring_ctx as before with struct groups if ever needed or extended in a different way. Fixes: d0e437b76bd3c ("io_uring/bpf-ops: implement loop_step with BPF struct_ops") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/5f6ca3649e9e0bae8667db4357e28dd00cd07901.1780394491.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-06-02 11:08:25 +01:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: add kfunc helpers Add two kfuncs that should cover most of the needs: 1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries. 2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions: - IOU_REGION_SQ returns the submission queue. - IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING. - IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication. It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/967bcc10e94c796eb273998621551b2a21848cde.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:40 +00:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: implement bpf ops registration Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/1f46bffd76008de49cbafa2ad77d348810a4f69e.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:41 +00:00
io_uring/bpf-ops: reject re-registration of an already-bound ops io_install_bpf() only rejects a second registration on the ctx side (ctx->bpf_ops) and sets the per-map back-pointer ops->priv unconditionally. The struct_ops link path never advances a map past BPF_STRUCT_OPS_STATE_READY, so the same io_uring_bpf_ops map can be registered more than once, and bpf_io_reg() re-resolves the target ring via fget(ops->ring_fd) on every call. A caller can therefore point the same ring_fd at a different io_ring_ctx between two BPF_LINK_CREATE calls. The second registration passes the ctx->bpf_ops check (the new ctx has none) and overwrites ops->priv, orphaning the first ctx. Teardown (io_eject_bpf()/bpf_io_unreg()) only reaches a ctx through ops->priv, so the orphaned ctx is never torn down: its ctx->loop_step keeps pointing into the struct_ops trampoline, which is freed once the map is gone. A later io_uring_enter() on the orphaned ring then calls the dangling ctx->loop_step from io_run_loop() -- a use-after-free of freed executable memory, reachable by a task with CAP_BPF + CAP_PERFMON. Reject registration when ops->priv is already set, as hid_bpf_reg() does for its struct_ops. Cc: stable@vger.kernel.org Fixes: 98f37634b12b ("io_uring/bpf-ops: implement bpf ops registration") Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com> Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de> Reviewed-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/20260717154537.129736-1-worapat.kd2@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-07-17 22:45:37 +07:00
io_uring/bpf-ops: implement bpf ops registration Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/1f46bffd76008de49cbafa2ad77d348810a4f69e.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:41 +00:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: implement bpf ops registration Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/1f46bffd76008de49cbafa2ad77d348810a4f69e.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:41 +00:00
io_uring: unify getting ctx from passed in file descriptor io_uring_enter() and io_uring_register() end up having duplicated code for getting a ctx from a passed in file descriptor, for either a registered ring descriptor or a normal file descriptor. Move the io_uring_register_get_file() into io_uring.c and name it a bit more generically, and use it from both callsites rather than have that logic and handling duplicated. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-04-08 11:56:02 -06:00
io_uring/bpf-ops: implement bpf ops registration Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/1f46bffd76008de49cbafa2ad77d348810a4f69e.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:41 +00:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00
io_uring/bpf-ops: implement bpf ops registration Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/1f46bffd76008de49cbafa2ad77d348810a4f69e.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:41 +00:00
io_uring/bpf-ops: implement loop_step with BPF struct_ops Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic. The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/98db437651ce64e9cbeb611c60bf5887259db09f.1772109579.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2026-02-26 12:48:39 +00:00