-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmqGD2AQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpomLD/9rr3SIQvdQRTnSCJlw1cFypSrnRhGAx1g8
v1F7UXuf4Abt4GIerbdXmBTx9zZRctpcOC41ReQk3YKkYuukuvna+Z6tBGf5I5+e
3SR3Zkrj2k4zyNC49g0/QkUJkTTS4tlsBFhNJtY3Q5BBIRMRd/sPvZ+cLnD7g/wH
3zM9IaEMk9xu9mF1nGI7LRl1qY0IOr4ynQpqohl5sEL3TQWgIa0r2B0jhGacBAvC
GGDwIwGDrM6oGwCn6mLN78O5odhLyTufIMNy3EnVw7BcDpLApcnJzYYjGzv+ElCM
WLBJwWYj+PYeoenuJuFhbRihbOPg5gZT3Mn8CUhLxUYzwZvIdFGWOjcGffD9sVu/
JO4nB6A+2AG/wgsb2YVKefhN86FoD0F/2hhdEdgZPJE6a+7+xeyyp9IzyQEWd0cg
Seo06fxE4pwF+dczXT3jzU3LXSZb/Jbh9hzu163VMSWbGYDCpDzfipqZGeN6DWE7
nG2NssAQtjtZrgdggYbZXtwecLZhu3cMj3jbhS4wSUzhQFW42yceJstCx4wpRhW9
m9EXjHQnRFMAb2TjUC3FzEncjbxL+uACjcKhTetd1hLbTI5I8KArZXjam1OdJOHV
urdtsru79LanQLAgRwWYPeKCZGNhX7I2o3OsmVDLQgHIDaZguTcPUVQs3wVui7Dk
IDmmrtwzpg==
=G1l4
-----END PGP SIGNATURE-----
Merge tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe updates via Keith:
- Enable Clang context analysis for the nvme host driver, adding
context annotations across core, fabrics, rdma, tcp and pci
- nvmet reservation state exposed through a new namespace-level
debugfs directory, plus ABI documentation for the host sysfs and
target configfs interfaces
- nvme-tcp host memory disclosure fixes on the read path: reject a
read that transferred too few bytes, don't accept C2HData based
on blk_rq_payload_bytes() alone, and fix the R2T case for a read
command
- Parallelize nvme-rdma I/O queue allocation and startup (Surabhi)
- Apple nvme fixes and quirks: page aligned admin queue buffers,
destroy the admin queue on removal, and various DMA/NVMMU
correctness fixes
- A large pile of nvmet and host fixes for out-of-bounds reads,
refcount/resource leaks, and NULL derefs across auth, zns,
passthru, pci-epf, rdma and configfs
- Various other fixes and cleanups
- MD updates via Yu Kuai:
- llbitmap reshape support, the large series wiring exact bitmap
mapping and reshape lifecycle through raid5 and raid10, growing
the page cache in place, and remapping checkpointed bits as
reshape progresses
- raid5 fixes for lockless max_nr_stripes and recovery_offset
accesses, a reshape deadlock with more failed devices than max
degraded, and bitmap batch counter consistency
- Atomic write handling for raid1/raid10, and removal of the
REQ_NOWAIT support from raid1/10/456
- raid5-ppl use-after-free fix in ppl_do_flush()
- A batch of smaller fixes across md core and the bitmap code
- s390/dasd ESE full-track write support and the surrounding
infrastructure, plus enabling CONTEXT_ANALYSIS for s390/block
- RWF_DONTCACHE support for block devices, built on new task-context
bio completion infrastructure, and wiring it up for the iomap and
buffer dropbehind writeback paths
- Async io_uring zone reset all, plus zone management command cleanups
allowing REQ_NOWAIT and tightening conventional zone rejection
- Block integrity refactoring: lift BIP_CHECK_FLAGS to the shared
header, handle nogenerate/noverify properly in fs-integrity, and drop
the blk-integrity.h include from bdev.c
- Split out a new blk_plug.h header
- ublk improvements: add UBLK_F_IO_DESC_SIZE, split request validation
from io_desc init, reject non-power-of-2 zone sizes in SET_PARAMS,
and a series of hardening fixes around map/unmap and auto buf reg
- null_blk cleanups and configfs serialization fixes
- nbd queue freeze removal on the setup paths, and a new
pre_defined_connections module parameter for pre-created devices
- blk-cgroup fixes for the race between policy activation and blkg
destruction, and accounting per-cpu stats over possible CPUs across
blk-stat, iolatency, iocost and kyber
- Various dio fixes: leak on metadata mapping error, validate user
space vectors during extraction, and set dma_alignment from the
backing file for loop and zloop direct I/O
- bio cleanups
- Various other fixes and cleanups all over
* tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (241 commits)
nbd: add pre_defined_connections module parameter for pre-created devices
nbd: remove queue freeze for newly created nbd from netlink path
nbd: factor out a nbd_genl_foreach_sock
nbd: skip queue freeze when setting size at device startup
nbd: remove queue freeze in nbd_add_socket
nbd: clear queue limits on disconnect
nbd: disallow NBD_SET_SOCK on an active device
nbd: simplify find_fallback() by removing redundant logic
blk-mq: add missing call to srcu_barrier() in blk_mq_free_tag_set()
block: mtip32xx: synchronize ioctls with device removal
ublk: avoid teardown retry loop on xarray allocation failure
null_blk: fix UBSAN shift-out-of-bounds when zone_size is 0 or overflows
block: don't include blk-integrity.h in bdev.c
xfs: avoid double deferrals for RWF_DONTCACHE writes
loop: Fix recently introduced lock inversion
block: set QUEUE_FLAG_DYING unconditionally in blk_mark_disk_dead()
swim3: Add missing MODULE_DESCRIPTION
selftests: ublk: add SET_PARAMS validation test
selftests: ublk: add helper for SET_PARAMS
ublk: reject non-power-of-2 zone sizes in SET_PARAMS
...
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmqGD3YQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgple3D/4wIQyeuf0U2NfM78K4sAX8FjEPzNg/WDzW
LRozg+Sldz2PNpVIbsPqDT9oW+K46zK62ptfF2gTl++dAjP2ER+HD44aXnwLh2JF
Cz54n2PNzR6TY4PeMLwR/fJ7SsAf5pQSRJqsV5aQPCqfEEgviLXczXQP0RcLQdB4
ksqay2xN+MaRGbgTqYYxc53ghMRSxiUI2Hm8boLrKAY9xTIJiZHyjVZmdlv5qp0s
SUCHAnlkTN/zC0zrAi1N+uK+kCTXuQRYQSMbW7LKkKHseqC/4ZwT1HEL20s74cV4
NLGrl2qdAU4euygUADILXKPwz4T0qfZyuIPd4OOkBtoND/uspGQ93r77XJsb8i9J
4G0cNK4y5lppuhp5Qo51a6zLfVVjOoSXSS8hI869ELc2w2iVDmNWccnxxySywBkd
XwKJ/BuRtB+xj9PvxuFgIxBCZTD17Auk8YmY+ifVainrAdNC78/BOiESXR49Gz2i
2Jvhkj2msf7SeAMPuHNox0jTHm3dO2XAktlxNqlaPSYRb5dniZpjVm/0zvvKSKr9
BXULpzU5M+ka2bT7htzSDXOOK43Ospazc8wQwYR54DOiZAcfKizps7VwggWKRRN9
kAOEWIQaMdUMdJ2u/9KZjjMqhdlyarx/Q9R6qeI/Mu7UTzxCi0KbawdqSzl9X/mb
6K6Ks/pmYg==
=N+68
-----END PGP SIGNATURE-----
Merge tag 'for-7.3/io_uring-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring update from Jens Axboe:
"On top of the usual cleanups and fixes, the bigger items in here are:
- zcrx work, most of it centered around adding dynamic area
provisioning, plus a bunch of prep and cleanups leading up to it:
scale refilling with large pages, coalesce same-niov RQEs on
refill, separate the RQ head/tail cache lines and cache the RQ
tail, and rework the area creation locking.
- Fix the futex inflight accounting so that only private futex waits
are marked inflight, and don't mark wake requests as inflight at
all.
- Drop the custom iov copy in the buffer select prep and msg header
copy paths, using the generic helpers instead.
- Fix a folio size overflow in io_vec_fill_bvec(), and account the
pages a compound region really uses in the memmap path.
- Fix an iovec leak in uring_cmd when the async cmd isn't recycled,
skip the blocking task work for io_uring_cmd_issue_blocking(), and
don't skip completion for a synchronous multishot cmd
- Defer eventfd signaling when queued from a wakeup handler
- Fix io-wq worker accounting when canceling creation callbacks
- Annotate remote tasks for kcoverage"
* tag 'for-7.3/io_uring-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (30 commits)
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
io_uring/uring_cmd: don't skip completion for a synchronous multishot cmd
io_uring/memmap: account the pages a compound region really uses
io_uring/zcrx: add dynamic area provisioning
io_uring/zcrx: lock area creation with pp_lock
io_uring/zcrx: keep array of areas
io_uring/zcrx: move freelist lock to struct zcrx
io_uring/zcrx: unmap under netdev lock
io_uring/zcrx: split dmabuf unmap and release
io_uring/zcrx: don't pass ifq_reg to area creation
io_uring/zcrx: add helper for deriving area token
io_uring/zcrx: don't reload skb_shinfo
io_urint/zcrx: narrow var scope in io_zcrx_recv_skb()
io_uring/zcrx: constify area_reg on import
io_uring/zcrx: coalesce same-niov RQEs on refill
io_uring/zcrx: cache RQ tail
io_uring/zcrx: add RQ iterator
io_uring/zcrx: move RQ head/tail to separate cache lines
io_uring/zcrx: scale refilling with large pages
io_uring/io-wq: fix worker accounting when canceling creation callbacks
...
If IORING_URING_CMD_MULTISHOT is set, io_uring_cmd() treats any
non-negative return from ->uring_cmd() as the driver having taken
ownership of the request and returns IOU_ISSUE_SKIP_COMPLETE. But
nothing guarantees that the driver did so, and any handler that just
completes the command inline and returns 0 or a positive result then
leaves the request orphaned, leaking the io_kiocb, the async data,
and the file reference.
The special case isn't needed. ublk returns -EIOCBQUEUED for the
multishot fetch command, which is passed through as-is, and the poll
driven socket timestamp command returns -EAGAIN. Kill it, a multishot
handler that wants to hang on to the request must return -EIOCBQUEUED
or -EAGAIN like any other command.
Fixes: 620a50c927 ("io_uring: uring_cmd: add multishot support")
Cc: stable@vger.kernel.org
Reported-by: syzbot+a4ccdd7ebf452e4d4701@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a7a0194.b50370da.49fe0.0056.GAE@google.com/
Link: https://lore.kernel.org/all/20260811115125.1831170-1-vasilisalmpanis@gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_mem_alloc_compound() allocates get_order(size) pages, which rounds a
region size that is not a power of two up to the next order. The pages
past the region are part of the same allocation and cannot be used for
anything else, but io_create_region() accounts reg->size >> PAGE_SHIFT,
so they are never charged against RLIMIT_MEMLOCK. For a ring with 4096
SQ entries and the default CQ size the region is 37 pages while the
allocation is 64.
Account the tail pages together with the region, and fall back to the
exact sized bulk allocation when they do not fit the limit, so a user
close to their limit still gets the region rather than an error.
io_free_region() gives the same amount back, the compound case being the
one that set IO_REGION_F_SINGLE_REF.
Counting how many 4096 entry rings an unprivileged user can create under
a given RLIMIT_MEMLOCK, before and after:
limit (pages) before after
256 2 2
300 2 2
350 3 2
400 3 3
512 5 4
Five rings under a 512 page limit really pin 640 pages.
Fixes: dfbbfbf191 ("io_uring: introduce concept of memory regions")
Link: https://lore.kernel.org/all/87ik5ncj8d.fsf@mailhost.krisman.be/
Signed-off-by: Ali Ahmet Memis <ali@iusegentoo.com>
Link: https://patch.msgid.link/20260806180044.275543-1-ali@iusegentoo.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Currently, we have only a one area per zcrx instance, and struct
io_zcrx_ifq stores a single pointer. To prepare for adding more areas,
replace it with an array of areas.
We'll be creating them at runtime, and the array is protected by 3
locks: ->pp_lock, ->alloc_lock and ->rq.lock. It takes all of them when
switching arrays, and readers should hold either of them.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/d186adfa0ea26f651804cffef77d477f257eb404.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Until now unmapping and destroying dmabuf were the same thing. To keep
it consistent with non-dmabuf, split it into two separate helpers. Unmap
destroys mappings and attachements as it should, and release only
putting down the dmabuf fd reference.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/bd8c1fd8dac85ce0cf839b389ecd6050761965ca.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
We might want to create an area without having an instance of struct
io_uring_zcrx_ifq_reg. Extract a helper that doesn't have the ifq
registration structure as an argument but takes the buf length
explicitly.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/76751ee4747abb4893058fcaaa36895dfe24a297.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The RQ tail is updated by the user space. Cache it to reduce cache line
bouncing. Refilling now tries to exhaust the previous batch of rqes, but
since it could be too low, the iterator is allowed to recalculate the
rqes to process once after synching the tail value.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/b42d2ed8b0e697110b646573b5da2a5f8e85f92e.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
RQ head and tail are currently put into the same cache line, which can
cause false sharing problems when refill is run on another CPU. Put them
into separate cache lines.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Link: https://patch.msgid.link/f769116a3f5267f06cb9a9a819d482ddea2e0852.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_zcrx_ring_refill() caps the loop by mixing the max number of
allocated netmems and the number of available RQEs together, which
caps the number of entries to process the pp cache size. As a result,
when niovs are heavily fragmented, the refilling logic allocates only a
small number of niovs per call on average and sometimes even none.
Keep a separate counter for the number of processed RQ entries, which is
capped by a roughly calculated from the page size value to keep the
cache full. And separately break if it allocates enough niovs.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/143cf439299728759eb0a840c866363fe99293f0.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
create_worker_cb() reserves an io-wq worker slot only after its
task-work callback runs. If the callback is canceled before then,
io_worker_cancel_cb() still decrements acct->nr_workers. When an
existing worker retires with its creation callback pending, that
worker has already decremented the same account's worker count.
The resulting undercount permits worker creation beyond the account's
configured limit. On an AST2600 OpenBMC system, an unchanged sensor
daemon reached 4,291 threads with the original kernel. With an
equivalent downstream fix, 25 passive samples under its normal
workload showed 6-9 threads.
Decrement nr_workers only when the canceled callback is not
create_worker_cb(). Continuation callbacks still release their reserved
slot, and both callback types retain the existing running-count,
reference-count, and create-state cleanup.
Fixes: 1d5f5ea7cb ("io-wq: remove worker to owner tw dependency")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Vishnu Razdan <vrazdan@openai.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260811-vrazdan-io-wq-b4-submit-v1-1-719ced16c921@openai.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
ddir was only ever forwarded to io_compat_msg_copy_hdr, which
never looked at it. Drop it.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-4-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Similar to commit f4eaf8eda8 ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid the custom copy of a single iovec and just
rely on the iovec api. This lets the compat and native paths share the
buffer-select length lookup
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-3-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Similar to commit f4eaf8eda8 ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid custom copy and just rely on the iovec api.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-2-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_req_local_work_add() signals the CQ ring eventfd inline when it is the
one to push the first entry onto ->work_list. For DEFER_TASKRUN rings that
add is frequently done from a waitqueue wakeup handler, where an
arbitrary waitqueue lock is held.
eventfd_signal_mask() only refuses to recurse when current->in_eventfd
is set, but that bit is set by eventfd_signal_mask() itself. If the wake
chain starts somewhere else, signal goes out inline and can feed back
into epoll.
Add IOU_F_TWQ_IN_WAKE, set it on the task_work add done from the three
waitqueue callbacks, and use it to force io_eventfd_signal() down the
existing call_rcu_hurry() deferral instead of signaling inline.
Fixes: 21a091b970 ("io_uring: signal registered eventfd to process deferred task work")
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20260813133843.2933127-1-4ncienth@gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_import_reg_vec() converts the estimated number of bio_vec entries
into iovec-sized storage when struct bio_vec is larger than struct
iovec. The conversion still multiplies nr_segs by sizeof(struct
bio_vec) in size_t without checking for overflow.
On 32-bit kernels, a registered buffer large enough to make
io_estimate_bvec_size() return 357913942 segments wraps the byte count
from 0x100000008 to 8. io_vec_realloc() then reserves only the input
iovecs plus one extra slot while io_vec_fill_bvec() writes the full
bio_vec array.
Check both the multiplication and the rounding addition before
deriving the replacement iovec count.
Fixes: b4e41050b2 ("io_uring/rsrc: raise registered buffer 1GB limit")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://patch.msgid.link/20260812203042.720348-1-Jeremy.Jean@oss.cyber.gouv.fr
Signed-off-by: Jens Axboe <axboe@kernel.dk>
struct bvec_iter::bi_bvec_done is used an offset in the current bvec,
let's rename it accordingly for better clarity. I also plan to use it
for non-bvec based iteration in the future like dma-buf, so drop the
"bvec" part.
Suggested-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/4e4c21858705a200bd8848ffe4080522e3eb5c1c.1786018753.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
blkdev.h gets included in various places outside the block layer just
for struct blk_plug and related plugging functions.
Split blk_plug into a separate helper to reduce the amount of code
that needs to get rebuilt when blkdev.h changes and to slightly
reduce compile times.
In io_uring this requires pulling in a few other headers explicitly that
previously were implicitly included through blkdev.h.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christian Brauner (Amutable) <brauner@kernel.org>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125524.740996-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_vec_fill_bvec() computes the folio size with a plain int 1:
unsigned long folio_size = 1 << imu->folio_shift;
imu->folio_shift is unsigned int and comes from folio_shift() of the
folio backing the registered buffer, so it can be 32 or more on a 64 bit
kernel. Shifting int 1 that far is undefined, and on x86 and arm64 the
count is taken modulo 32, so a shift of 34 yields 4 rather than 16G.
Every other folio_shift shift in this file already uses 1UL.
The result is that the segment estimate and the fill loop disagree.
io_estimate_bvec_size() sizes the bvec array with the real shift:
max_segs += (iov[i].iov_len >> shift) + 2;
so a 1M iovec on a 16G folio is charged 2 segments, while
io_vec_fill_bvec() then walks the same iovec in folio_size chunks of 4
bytes and writes res_bvec[bvec_idx] a quarter of a million times, past
the end of the array it was given. src_bvec is advanced once per
iteration as well, so imu->bvec is read past its end at the same time.
validate_fixed_range() only checks that the range is inside the
registered buffer and does not bound the segment count.
Reaching it needs a folio with a shift of at least 32, which means a
gigantic hugetlb page: 16G on arm64 with 64K pages, where
CONT_PMD_SHIFT is 34 and hugetlb_add_hstate(CONT_PMD_SHIFT - PAGE_SHIFT)
registers that size, and likewise on powerpc. x86_64 tops out at 1G, so
a shift of 30, which still fits in int and is unaffected.
Use 1UL, as the rest of the file does.
Fixes: 9ef4cbbcb4 ("io_uring: add infra for importing vectored reg buffers")
Cc: stable@vger.kernel.org
Signed-off-by: Ali Ahmet Memis <ali@iusegentoo.com>
Link: https://patch.msgid.link/20260802163030.51005-1-ali@iusegentoo.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
An io_async_cmd carries an iovec array in ->vec.iovec, allocated when the
vec has to grow and kept across recycling through ctx->cmd_cache. On two
paths nothing frees it and io_clean_op()'s kfree(req->async_data) drops
the io_async_cmd without it.
io_req_uring_cleanup() clears the async data flags only when
io_alloc_cache_put() succeeds, and the cache holds IO_ALLOC_CACHE_MAX ==
128 entries, so once it is full the put fails and the vec is left behind.
An NVMe passthrough workload gets there without doing anything unusual:
nvme_uring_cmd_io() returns -EIOCBQUEUED, so the io_async_cmd stays
attached for the lifetime of the command and the live object count tracks
the queue depth. Above 128 the puts start failing.
->cleanup is the last chance to free an inherited vec, since
io_req_uring_cleanup() returns early for an io-wq issued command and is
not called at all for one completed without ever being issued. But
io_clean_op() calls ->cleanup only if REQ_F_NEED_CLEANUP is set, and for
uring_cmd that happens only where the vec has to grow, so a command
reusing a large enough cached vec never sets it. io_rw_alloc_async() and
io_msg_alloc_async() flag an inherited vec for exactly this reason;
io_uring_cmd_prep() does not.
Flag an inherited vec in io_uring_cmd_prep(), and free the vec when the
cache put fails, as io_req_rw_cleanup() does.
The leak is invisible under KASAN, where io_alloc_cache_vec_kasan() frees
the vec unconditionally.
Fixes: 3a4689ac10 ("io_uring/cmd: add iovec cache for commands")
Cc: stable@vger.kernel.org
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Link: https://patch.msgid.link/20260802073518.419-1-worapat.kd2@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Per-task restrictions apply to all rings created by a task. Once
installed, they should not be dropped across exec.
For a task that has used io_uring, the exec cancellation path calls
__io_uring_free(). This frees both the task context and the per-task
restriction, so a ring created after exec is unrestricted.
Split task context cleanup into io_uring_free_tctx(), and use it from
the exec cancellation path. Keep __io_uring_free() for final task
cleanup, where both the context and restriction are released.
Fixes: ed82f35b92 ("io_uring: allow registration of per-task restrictions")
Cc: stable@vger.kernel.org # 7.1+
Signed-off-by: Kyumin Lee <fyonglkm@gmail.com>
Link: https://patch.msgid.link/20260730192734.459247-1-fyonglkm@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring_cmd_issue_blocking() is only called from blk_cmd_complete(),
which is already a task work callback. However, it queues another task
work item, to call io_queue_iowq(). Just call io_queue_iowq() directly
to skip the CPU cost and latency of the redundant task work proxying.
Signed-off-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260702184847.1709378-1-csander@purestorage.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
import_zcrx() attaches an existing ifq to another ring. It never calls
zcrx_set_ring_ctx() and so never takes the ->master_ctx reference, but
its error path still passes @ctx to zcrx_unregister(), which clears
->master_ctx and drops its percpu_ref whenever ifq->master_ctx == ctx.
That condition is reachable. A ring that registers an ifq with a
non-zero event type_mask gets ->master_ctx pointed at itself, and
nothing stops it from exporting that ifq with ZCRX_CTRL_EXPORT and
importing the resulting fd back into the same ring. Failing the import
after the refcount bumps -- an argument page mapped PROT_READ makes the
copy_to_user() in import_zcrx() return -EFAULT -- then clears the
->master_ctx owned by the original registration, which is still live.
Refcounts stay balanced and nothing is freed early, so there is no
splat. The ring silently stops receiving ZCRX_EVENT_ALLOC_FAIL and
ZCRX_EVENT_COPY: zcrx_send_notif() returns early on a NULL
->master_ctx, and ->master_ctx is only ever set on a freshly allocated
ifq, so it cannot be restored without tearing the ring down.
Pass NULL instead, matching zcrx_box_release() and the zcrx_export()
error path. io_register_zcrx() only gets away with passing @ctx
because zcrx_set_ring_ctx() runs after its last goto err.
Fixes: 00d9148127 ("io_uring/zcrx: share an ifq between rings")
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Link: https://patch.msgid.link/20260730162741.1125-1-worapat.kd2@gmail.com
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
[axboe: add pavel edit]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Turns out that my naive assumption that surely nobody is using io_uring
to manage nested epoll instances was wrong - libuv is in fact doing that
very thing. Since we have users in the wild using that feature, we
cannot remove it from upstream. Revert the commit that introduced that
limitation.
This reverts commit cfa1539b24.
Link: https://lore.kernel.org/io-uring/605939add4dfb674bd7ab39ef8dbd3fd22e754e4.camel@xry111.site/
Reported-by: Xi Ruoyao <xry111@xry111.site>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Inflight tracking of futex wait requests exists to ensure that do_exit()
-> io_uring_files_cancel() cancels them before the mm goes away, as a
private futex wait depends on the mm private futex hash staying alive
for the duration of the request.
Shared futexes have no such dependency. A FLAGS_SHARED request always
resolves to either an inode based key or an mm-shared key, both of which
fail futex_key_is_private() and hence always hash into the global futex
hash, whose lifetime isn't tied to the mm.
Only mark vectored futex waits as inflight if the futex is private.
Cc: stable@vger.kernel.org
Fixes: 079afb081c ("io_uring/futex: mark wait requests as inflight")
Link: https://lore.kernel.org/io-uring/CANGjgdn=R_qyUdE=j9za+vkmqcxacbP-84OHXF4nZ4ho9qRyVg@mail.gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Commit 079afb081c ("io_uring/futex: mark wait requests as inflight")
added inflight tracking to ensure that do_exit() ->
io_uring_files_cancel() finds and cancels pending futex waits before the
mm goes away, as a private futex wait depends on the mm private futex
hash staying alive for the duration of the request. However, as
io_futex_prep() is shared between FUTEX_WAIT and FUTEX_WAKE, wake
requests got marked as inflight as well.
A futex wake executes fully inline at issue time and never depends on
the mm staying alive after completion, hence there's no need to track
it. Kill it.
Cc: stable@vger.kernel.org
Fixes: 079afb081c ("io_uring/futex: mark wait requests as inflight")
Reported-by: Chengfeng Lin <lin2530632123@gmail.com>
Link: https://lore.kernel.org/io-uring/CANGjgdn=R_qyUdE=j9za+vkmqcxacbP-84OHXF4nZ4ho9qRyVg@mail.gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_ring_buffers_peek() builds an iovec array from provided buffers, and
that in turn can be handed off to a lower level provider. Be prudent and
cap the total size to MAX_RW_COUNT, which is the Linux default for how
much IO do to in a single call.
No bugs here, but it's a good preventative measure to avoid truncation
issues.
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Commit:
6a8afb9fff ("io_uring/net: allow multishot receive per-invocation cap")
changed how io_mshot_prep_retry() set sr->len, and added the same
initialization in io_mshot_prep_retry(). But it neglected to touch the
send path, which may also uses the mshot retry path. Ensure that
sr->mshot_len always gets initialized correctly.
Fixes: 6a8afb9fff ("io_uring/net: allow multishot receive per-invocation cap")
Cc: stable@vger.kernel.org
Reported-by: Sung Keum <kambodi127@gmail.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmpkDfYQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpgJgEACAoCs3hC/reUovTYQB8+68AyyBmF5xeJMI
uxYKcVEZGjOo6gTjSm2GYEBa6P85kQtbWEKEfuxXxV2gT1S/3yulV+uy8NfI/lO7
NsKYRHszaa/AFr2RbeZeYLU9TN/qFbZinfxukpW1zWrfwg83IhGGe6QN53G1XB2v
+pO0bqSZ/qKkJ4IXGubLxSxUnDWg6bav4XUeEkjSlE6mzp1+BgpesLLOsZNvvLST
5QTGt5EfWFBgo9LdYHz/ky7MnJjkioFDY+c10sLp2+6ghtyMTZcACB1W4gSSyJG+
QsLYJ6Z25f7zo7WIci42pCDDDhYWQgDFc7r9+uJ1ISFKXLGnEyONnBaxmk2LTWdX
g9V03+pI17vTOQNU0XlkpajUykOmMCkhZWA8jwlOAFJXtz+pdV1vSKeSZt9tEB4j
aEd6IjoBUrPLsv7WIlomf/h9quHfdjMogA6eyaXc4227XzKWo63XlGe4HxBC6yQQ
4Lq8QH8dta6BOm+5APQXNQ5g/EB0OVLDkgh8EzSTDChgoJd4hVwoHnGbGidZSAcR
mgSl4SG/XdXMIwhqAH5iiI+7YZ71ALnoRkLujED90c+lGHrrY0T7H99XaHprp6xa
NOUbz5CAZf+bWPfXbLpJG+QTh/evh9zxfJ30VprHByl99EJFxjK7rW3BJvOAokjs
1O635JJo7A==
=pQDU
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.2-20260724' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a missing ERESTARTSYS conversion in the read paths, which got
messed up back when some code consolidation was done for read
multishot support
- zcrx UAPI rename, dropping the abbreviated "notif" naming in favor of
"event" for consistency and to be less ambiguous for users. This was
added for 7.2, so let's rename it while we still can. No functional
or code changes, just a strict rename
* tag 'io_uring-7.2-20260724' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/zcrx: rename notif to event
io_uring/zcrx: rename ZCRX_NOTIF_NO_BUFFERS
io_uring/zcrx: drop "notif" from stats struct names
io_uring/rw: fix missing ERESTARTSYS conversion in read paths
"Notification" is too long and the abbreviated version is used in
several places, which is inconsistent and more ambiguous for users.
Rename it to event, which is easier to keep consistent. To keep the
change small, only change uapi/ + do necessary fix ups, and the rest of
internals can be adjusted in the next release.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/f95ca6717da3c8d3649a1a7f0d883a563f545052.1784726895.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Both read and write may receive internal restart error codes from
the filesystem layer and should be converted to -EINTR. However,
when multishot read support was added, the error code normalization
was lost for both io_read() and io_read_mshot().
Extract the conversion into io_fixup_restart_res() and apply it
in all three locations: io_rw_done(), io_read(), and io_read_mshot().
Fixes: a08d195b58 ("io_uring/rw: split io_read() into a helper")
Cc: stable@vger.kernel.org
Signed-off-by: Yitang Yang <yi1tang.yang@gmail.com>
Link: https://patch.msgid.link/20260722124551.130563-1-yi1tang.yang@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmpbBbsQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpgpFD/9NnN5A9eMm+4ZeC41yeynykN1QOl5wCqoT
zi7CIizuwtsCQAjgpFUahMosvkLg3P77S/MJxqzlGVPhaSvp/QISxr7oH9WqyoYf
jEfi3Dx8XfdxK+5GG0dC7s4iiWuxZha7iGt4W2DZmjXWWu0ofCKB0FTVK7J/AkC0
7DA4hMmts643cE9pwmD1Bbuj7SYmqjneDhgwe/JaSuCiuuclN5L359E1rnxzc+dS
9McCmwPjwF0f+Vs9IWyHb1KNMBBmC2T0n2zvzKoJKL72mQ0v3IaOlGkUEDgp7EpT
TIK5rEKPl/Wpw970BOlzdYgD01vDiWeLuQQQlGfAaLheUUImLoFPo1rf/q8nCgFN
CVEWDSZZWRnwH6FjFCa5JwOS0B9YcamYzhkTck1O+g7KcNB6ZCU1zMX8+pCMEVgr
Ju5J2uoazT2txEZEgCSSUYLuRRSpz5P5s/VmFEXNxeRAhndlGhyNq16kVnRqyLfA
OzVS5uYeqSOlOYbY8pL4QkGaxBUr0CSBOnhCT7q9d8KYXnh9CHReFQcy9wPclSfd
D+F5v4CPqWN0tg2rrYBgo3Zpt4Mwb1JEiyOAaIapAKQT2T9tNtp1fubq4PA9c1ub
VkSRutPGeUWNC1fEjs67I9V3V0bzbR8j+AMC9AAFFqaqpn2B5NGHqvlcYdaNuuje
Dr7yXvROWw==
=Ue52
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.2-20260717' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a use-after-free in the bpf-ops struct_ops path, where the same
io_uring_bpf_ops map could be registered more than once.
- Fix the deferred iovec free for the provided-buffer grow path, which
could leave the caller with a dangling iovec and result in repeated
frees. Follow-up to the earlier fix in this series.
- Zero-check the unused addr3/pad2 SQE fields for unlinkat
* tag 'io_uring-7.2-20260717' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/bpf-ops: reject re-registration of an already-bound ops
io_uring/fs: check unused sqe fields for unlinkat
io_uring/kbuf: free the replaced iovec after a successful grow
io_install_bpf() only rejects a second registration on the ctx side
(ctx->bpf_ops) and sets the per-map back-pointer ops->priv
unconditionally. The struct_ops link path never advances a map past
BPF_STRUCT_OPS_STATE_READY, so the same io_uring_bpf_ops map can be
registered more than once, and bpf_io_reg() re-resolves the target ring
via fget(ops->ring_fd) on every call. A caller can therefore point the
same ring_fd at a different io_ring_ctx between two BPF_LINK_CREATE
calls.
The second registration passes the ctx->bpf_ops check (the new ctx has
none) and overwrites ops->priv, orphaning the first ctx. Teardown
(io_eject_bpf()/bpf_io_unreg()) only reaches a ctx through ops->priv, so
the orphaned ctx is never torn down: its ctx->loop_step keeps pointing
into the struct_ops trampoline, which is freed once the map is gone. A
later io_uring_enter() on the orphaned ring then calls the dangling
ctx->loop_step from io_run_loop() -- a use-after-free of freed
executable memory, reachable by a task with CAP_BPF + CAP_PERFMON.
Reject registration when ops->priv is already set, as hid_bpf_reg()
does for its struct_ops.
Cc: stable@vger.kernel.org
Fixes: 98f37634b1 ("io_uring/bpf-ops: implement bpf ops registration")
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/20260717154537.129736-1-worapat.kd2@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Zero check unused SQE fields addr3 and pad2 for unlinkat. They're
not needed now, but could be used sometime in the future.
Signed-off-by: Yi Xie <xieyi@kylinos.cn>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260714030306.64820-1-xieyi@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The provided-buffer validation fix deferred freeing a cached iovec
until validation completed. However, the deferred free uses arg->iovs.
After a grow, that points to the newly allocated array. Without a grow,
it points to the cached array that remains in use.
This leaves the caller with a dangling iovec in both cases and can
result in repeated frees. Only free org_iovs when arg->iovs actually
replaced it.
Fixes: cd053d788c ("io_uring: fix dangling iovec after provided-buffer bundle grow failure")
Assisted-by: Codex:gpt-5.3-codex-spark
Signed-off-by: Jaeyeong Lee <iostreampy@proton.me>
Link: https://patch.msgid.link/20260712142612.188695595-iostreampy@proton.me
Signed-off-by: Jens Axboe <axboe@kernel.dk>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAmpRch8QHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgpqORD/9xYHpNwkeIXfG+6giRGCF03u0eGlreUsIl
uCGdqfNKjKZqWOw6q7uzNHll7/2SmwOniB32mdo6jpUgXeE3/RJ4U37kZ8ypIyO8
1iOSM5zz5ZNQQuHa/HZMqMRy9mqBa5WIJqRO4CpRk9uaqq6flU2CEq4/y3SdOc0S
9u+UxzVhmoYuL7iQhaLg6xQSrNxtwynVXqIru/bN1Ft01sCo3Yf3w0SmNnmsrr34
jG1tkn0J7QtwkaN81poX8WTqm0D7McFc5xcq8wWM+BDnpvllPxQ+J+J3+zMfvAbN
tbb3FiXkc5SeYTOrBu8FWwPMn8reNpUBgEGjrcZWyRIkHdzxrDhwvC9tyQs85l4w
j93PSsJnlprVPI4LWGfglCaNBZ85hFFKpm83rAFJgJjLuyzRFSF7+/EKrHFP58sK
v8oYq/iXpQunu1YLfBIAjW4OnVkqSYM95Xax4l5yFSgeCS+hNd/naRgvqzrM4puq
tspIH9zjZCyRftRuVebW4V/cxfceSp3kg5cPyW/PRonO9iGZ1Q2scK0bovNO0QKC
gDjax1qancwp/GlT65jgdLp9lAx1TIofPz2gP22lGlZe732bi3gvX5rGPo1bT98v
1NJXrEIqLHOuId308+pzpvl4j4YyJAC3llThVg4L2TZb6jy+S5qYnbBnRkQoNPVi
iMP+XrWPjw==
=vHWI
-----END PGP SIGNATURE-----
Merge tag 'io_uring-7.2-20260710' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Restore full RCU read section in io_req_local_work_add(), which was
mistakenly dropped with the DEFER_TASKRUN rework in this merge
window. Revert the commit that grabbed the RCU read lock in
io_ctx_mark_taskrun(), as that's no longer required with the previous
fix.
- Fix a dangling iovec after a provided-buffer bundle grow failure,
also an issue introduced in this merge window.
- Reject IORING_CQE_F_32 flag pass-through in MSG_RING to rings that
weren't setup with CQE32 or CQE_MIXED.
- Return -EINVAL rather than -ENOMEM from get_unmapped_area() when mmap
validation fails, matching io_uring_mmap().
* tag 'io_uring-7.2-20260710' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
Revert "io_uring: grab RCU read lock marking task run"
io_uring: restore RCU read section in io_req_local_work_add()
io_uring: fix dangling iovec after provided-buffer bundle grow failure
io_uring/uring_cmd: fix uring_cmd.c comments
io_uring/msg_ring: reject CQE32 flag pass-through to normal rings
io_uring/memmap: return -EINVAL from get_unmapped_area() on bad mmap
This reverts commit ed64f5c546.
Since commit:
648790e095 ("io_uring: restore RCU read section in io_req_local_work_add()")
io_ctx_mark_taskrun() is only ever called with the RCU read lock
already held, like previously. Hence's there's no need for this commit
anymore, which grabbed the RCU read lock inside io_ctx_mark_taskrun().
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The task-work refactor that moved io_req_local_work_add() out of
io_uring.c into the new io_uring/tw.c dropped the whole-body
guard(rcu)() that used to cover the function body.
For DEFER_TASKRUN rings the ring teardown still relies on that RCU read
section pairing with its grace period:
/* pairs with RCU read section in io_req_local_work_add() */
if (ctx->flags & IORING_SETUP_DEFER_TASKRUN)
synchronize_rcu();
io_ring_ctx_free(ctx);
io_req_local_work_add() keeps dereferencing ctx after mpscq_push() has
published the request to the work list (ctx->cq_wait_nr, and
ctx->submitter_task in the final wake_up_state()), without holding a ctx
reference across that window. The RCU read section was the only thing
guaranteeing an in-flight adder had finished touching ctx before
io_ring_ctx_free() ran; synchronize_rcu() only waits for readers that
are actually inside an RCU read-side critical section. With the guard
gone the grace period no longer pairs with anything on the add side, so
ctx can be freed and reused while io_req_local_work_add() is still using
it.
Fixes: d46ab2c98a ("io_uring: switch local task_work to a mpscq")
Signed-off-by: Woraphat Khiaodaeng <worapat.kd2@gmail.com>
Link: https://patch.msgid.link/20260709035100.2269-1-worapat.kd2@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
