2026-06-04 13:55:08 -06:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2008-11-26 12:03:54 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2026-03-17 12:00:29 +01:00
|
|
|
|
2026-04-01 09:02:14 +02:00
|
|
|
|
2025-01-20 02:29:04 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-02-02 19:15:33 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2014-07-02 16:29:19 -05:00
|
|
|
|
2025-05-13 12:20:49 +08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2024-07-03 10:38:42 -04:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
driver core: add devname module aliases to allow module on-demand auto-loading
This adds:
alias: devname:<name>
to some common kernel modules, which will allow the on-demand loading
of the kernel module when the device node is accessed.
Ideally all these modules would be compiled-in, but distros seems too
much in love with their modularization that we need to cover the common
cases with this new facility. It will allow us to remove a bunch of pretty
useless init scripts and modprobes from init scripts.
The static device node aliases will be carried in the module itself. The
program depmod will extract this information to a file in the module directory:
$ cat /lib/modules/2.6.34-00650-g537b60d-dirty/modules.devname
# Device nodes to trigger on-demand module loading.
microcode cpu/microcode c10:184
fuse fuse c10:229
ppp_generic ppp c108:0
tun net/tun c10:200
dm_mod mapper/control c10:235
Udev will pick up the depmod created file on startup and create all the
static device nodes which the kernel modules specify, so that these modules
get automatically loaded when the device node is accessed:
$ /sbin/udevd --debug
...
static_dev_create_from_modules: mknod '/dev/cpu/microcode' c10:184
static_dev_create_from_modules: mknod '/dev/fuse' c10:229
static_dev_create_from_modules: mknod '/dev/ppp' c108:0
static_dev_create_from_modules: mknod '/dev/net/tun' c10:200
static_dev_create_from_modules: mknod '/dev/mapper/control' c10:235
udev_rules_apply_static_dev_perms: chmod '/dev/net/tun' 0666
udev_rules_apply_static_dev_perms: chmod '/dev/fuse' 0666
A few device nodes are switched to statically allocated numbers, to allow
the static nodes to work. This might also useful for systems which still run
a plain static /dev, which is completely unsafe to use with any dynamic minor
numbers.
Note:
The devname aliases must be limited to the *common* and *single*instance*
device nodes, like the misc devices, and never be used for conceptually limited
systems like the loop devices, which should rather get fixed properly and get a
control node for losetup to talk to, instead of creating a random number of
device nodes in advance, regardless if they are ever used.
This facility is to hide the mess distros are creating with too modualized
kernels, and just to hide that these modules are not compiled-in, and not to
paper-over broken concepts. Thanks! :)
Cc: Greg Kroah-Hartman <gregkh@suse.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Chris Mason <chris.mason@oracle.com>
Cc: Alasdair G Kergon <agk@redhat.com>
Cc: Tigran Aivazian <tigran@aivazian.fsnet.co.uk>
Cc: Ian Kent <raven@themaw.net>
Signed-Off-By: Kay Sievers <kay.sievers@vrfy.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
2010-05-20 18:07:20 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-19 15:49:56 +01:00
|
|
|
|
|
|
|
|
|
2006-12-06 20:33:20 -08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2017-03-03 11:04:04 +02:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2025-01-22 13:55:27 -08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-10-01 10:07:05 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2012-10-26 19:48:07 +04:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2012-10-26 19:48:07 +04:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2018-10-01 10:07:06 +02:00
|
|
|
|
2026-06-09 09:58:51 +00:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2017-03-03 11:04:04 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-03-03 11:04:04 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-14 11:59:41 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
|
|
|
|
|
2026-07-16 11:31:43 -07:00
|
|
|
|
|
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
|
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2013-03-21 18:02:28 +04:00
|
|
|
|
2026-07-16 11:31:43 -07:00
|
|
|
|
|
|
|
|
|
2026-07-16 11:31:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2013-03-21 18:02:28 +04:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2018-11-09 15:52:16 +01:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2018-11-09 15:52:16 +01:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
|
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2006-04-10 22:54:59 -07:00
|
|
|
|
|
|
|
|
|
2024-09-06 16:34:53 +02:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2013-03-21 18:02:28 +04:00
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2013-03-21 18:02:28 +04:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2025-06-10 13:52:29 +09:00
|
|
|
|
2013-03-21 18:02:28 +04:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2006-06-25 05:48:50 -07:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2006-06-25 05:48:50 -07:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2006-04-11 21:16:09 +02:00
|
|
|
|
2013-03-21 18:02:36 +04:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2006-04-11 21:16:09 +02:00
|
|
|
|
2013-03-21 18:02:36 +04:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2006-04-11 21:16:09 +02:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2006-04-11 21:16:09 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2013-03-21 18:02:04 +04:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2006-02-04 23:27:40 -08:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2017-03-03 11:04:04 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2013-03-21 18:02:36 +04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-21 18:02:36 +04:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2015-07-01 16:25:56 +02:00
|
|
|
|
2006-06-25 05:48:52 -07:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2006-02-04 23:27:40 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-21 09:34:25 +01:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-21 09:34:25 +01:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2018-09-11 13:11:56 +03:00
|
|
|
|
|
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-22 13:48:30 +01:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2025-01-20 02:29:02 +01:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-03 14:47:38 +08:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
2023-01-08 17:00:23 -08:00
|
|
|
|
2018-06-18 15:53:19 +01:00
|
|
|
|
|
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
2018-06-18 15:53:19 +01:00
|
|
|
|
|
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-18 15:53:19 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 13:18:27 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:05 +01:00
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:05 +01:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
2026-07-09 08:37:05 +02:00
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-15 17:24:48 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
2025-09-15 17:24:48 -07:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-24 10:47:23 +02:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 12:08:00 +01:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-18 15:53:19 +01:00
|
|
|
|
2026-03-17 12:08:00 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 12:08:00 +01:00
|
|
|
|
2018-06-18 15:53:19 +01:00
|
|
|
|
2026-03-17 12:00:29 +01:00
|
|
|
|
|
|
|
|
|
2026-03-17 13:09:20 +01:00
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:00:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 13:09:20 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 16:32:37 +01:00
|
|
|
|
2026-03-17 13:09:20 +01:00
|
|
|
|
2026-03-17 14:34:20 +01:00
|
|
|
|
|
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
|
|
|
|
|
2026-03-17 14:34:20 +01:00
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-18 18:01:27 +01:00
|
|
|
|
2026-03-17 13:09:20 +01:00
|
|
|
|
|
|
|
|
|
2026-03-17 12:00:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 12:08:00 +01:00
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:08:00 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 12:08:00 +01:00
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:08:00 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
|
|
|
|
|
2026-05-22 16:38:05 +08:00
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-01 16:24:51 +08:00
|
|
|
|
|
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
2026-04-02 22:57:49 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-04-02 22:57:49 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
|
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-04-02 22:57:49 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-04-02 22:57:49 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 22:57:49 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
|
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-17 12:44:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-03-19 15:40:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
|
|
|
|
|
2018-06-21 09:34:25 +01:00
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
|
|
|
|
|
2026-03-31 13:18:27 +02:00
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2026-03-31 13:18:27 +02:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2018-07-31 13:25:25 +03:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
|
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
2026-04-08 10:25:10 -07:00
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
2026-04-08 10:25:10 -07:00
|
|
|
|
|
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-08 10:25:10 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
2026-04-08 10:25:10 -07:00
|
|
|
|
|
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
|
|
|
|
|
2026-04-08 10:25:10 -07:00
|
|
|
|
|
|
|
|
|
2026-03-17 14:48:33 +01:00
|
|
|
|
|
|
|
|
|
2026-04-08 10:25:10 -07:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:53 -07:00
|
|
|
|
2006-01-16 22:14:26 -08:00
|
|
|
|
2006-06-25 05:48:50 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2015-07-01 16:26:07 +02:00
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2019-10-21 09:11:40 +02:00
|
|
|
|
2024-07-03 10:38:42 -04:00
|
|
|
|
2018-11-08 12:05:25 +03:00
|
|
|
|
|
|
|
|
|
2021-08-04 13:22:58 +02:00
|
|
|
|
2018-11-08 12:05:25 +03:00
|
|
|
|
|
|
|
|
|
2021-08-04 13:22:58 +02:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2018-11-08 12:05:25 +03:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2018-11-08 12:05:25 +03:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-08 12:05:31 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-11-08 12:05:31 +03:00
|
|
|
|
2020-02-13 09:16:07 +01:00
|
|
|
|
2026-03-26 16:11:52 +01:00
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-06-21 09:33:40 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2018-11-08 12:05:42 +03:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2018-11-08 12:05:42 +03:00
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
2018-11-08 12:05:42 +03:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
|
|
|
|
|
2025-03-25 18:29:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:00 +02:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2015-07-01 16:26:00 +02:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:26:00 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:26:03 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2025-03-25 18:29:31 +01:00
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2016-07-19 03:08:27 -04:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:26:00 +02:00
|
|
|
|
2007-10-16 23:31:04 -07:00
|
|
|
|
|
|
|
|
|
2026-03-16 14:10:00 +01:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2026-08-12 20:58:36 -07:00
|
|
|
|
2026-03-16 14:10:00 +01:00
|
|
|
|
|
|
|
|
|
2025-03-25 18:29:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-16 23:31:04 -07:00
|
|
|
|
2006-06-25 05:48:50 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-08-12 20:58:36 -07:00
|
|
|
|
2007-10-16 23:31:04 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2013-02-04 13:04:44 +00:00
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-23 13:52:31 +08:00
|
|
|
|
2019-09-10 15:04:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2022-11-10 15:46:33 +01:00
|
|
|
|
|
|
|
|
|
2020-02-13 09:16:07 +01:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:09 +02:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2014-12-12 09:49:05 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
|
|
|
|
|
2026-03-16 14:10:00 +01:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-12-12 09:49:05 +01:00
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:09 +02:00
|
|
|
|
2015-01-06 10:45:35 +01:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
|
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2014-12-12 09:49:05 +01:00
|
|
|
|
2019-09-10 15:04:08 +02:00
|
|
|
|
2019-09-10 15:04:09 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2014-12-12 09:49:05 +01:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2014-12-12 09:49:05 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:07 +01:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2025-01-20 02:29:07 +01:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2025-01-20 02:29:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-15 17:24:48 -07:00
|
|
|
|
2025-01-20 02:29:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2018-08-27 18:29:56 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2015-07-01 16:25:56 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2025-01-20 02:29:07 +01:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:07 +01:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-08-27 18:29:56 +03:00
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
2026-03-30 12:36:59 +02:00
|
|
|
|
2018-08-27 18:29:56 +03:00
|
|
|
|
|
|
|
|
|
2008-02-06 01:38:39 -08:00
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2019-09-10 15:04:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2019-09-10 15:04:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2019-09-10 15:04:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2010-07-12 14:41:40 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2026-03-30 12:58:19 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
|
|
|
|
|
2010-07-12 14:41:40 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2010-07-12 14:41:40 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2010-07-12 14:41:40 +02:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2010-07-12 14:41:40 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:53 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2025-02-20 12:16:59 -08:00
|
|
|
|
2025-01-20 02:29:00 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-04-03 22:06:08 -04:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-10-21 22:46:42 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2016-09-27 10:45:12 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-10-16 16:47:32 +01:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2015-04-03 22:06:08 -04:00
|
|
|
|
2022-06-09 10:28:36 -04:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
2015-04-03 22:06:08 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-01-16 22:14:28 -08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-08 16:38:28 +08:00
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
|
|
|
|
|
2021-09-08 16:38:28 +08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
2025-01-20 02:29:01 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
|
|
|
|
|
2025-08-05 18:22:51 +01:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-18 21:16:42 +02:00
|
|
|
|
2020-05-19 14:50:37 +02:00
|
|
|
|
mm: multi-gen LRU: groundwork
Evictable pages are divided into multiple generations for each lruvec.
The youngest generation number is stored in lrugen->max_seq for both
anon and file types as they are aged on an equal footing. The oldest
generation numbers are stored in lrugen->min_seq[] separately for anon
and file types as clean file pages can be evicted regardless of swap
constraints. These three variables are monotonically increasing.
Generation numbers are truncated into order_base_2(MAX_NR_GENS+1) bits
in order to fit into the gen counter in folio->flags. Each truncated
generation number is an index to lrugen->lists[]. The sliding window
technique is used to track at least MIN_NR_GENS and at most
MAX_NR_GENS generations. The gen counter stores a value within [1,
MAX_NR_GENS] while a page is on one of lrugen->lists[]. Otherwise it
stores 0.
There are two conceptually independent procedures: "the aging", which
produces young generations, and "the eviction", which consumes old
generations. They form a closed-loop system, i.e., "the page reclaim".
Both procedures can be invoked from userspace for the purposes of working
set estimation and proactive reclaim. These techniques are commonly used
to optimize job scheduling (bin packing) in data centers [1][2].
To avoid confusion, the terms "hot" and "cold" will be applied to the
multi-gen LRU, as a new convention; the terms "active" and "inactive" will
be applied to the active/inactive LRU, as usual.
The protection of hot pages and the selection of cold pages are based
on page access channels and patterns. There are two access channels:
one through page tables and the other through file descriptors. The
protection of the former channel is by design stronger because:
1. The uncertainty in determining the access patterns of the former
channel is higher due to the approximation of the accessed bit.
2. The cost of evicting the former channel is higher due to the TLB
flushes required and the likelihood of encountering the dirty bit.
3. The penalty of underprotecting the former channel is higher because
applications usually do not prepare themselves for major page
faults like they do for blocked I/O. E.g., GUI applications
commonly use dedicated I/O threads to avoid blocking rendering
threads.
There are also two access patterns: one with temporal locality and the
other without. For the reasons listed above, the former channel is
assumed to follow the former pattern unless VM_SEQ_READ or VM_RAND_READ is
present; the latter channel is assumed to follow the latter pattern unless
outlying refaults have been observed [3][4].
The next patch will address the "outlying refaults". Three macros, i.e.,
LRU_REFS_WIDTH, LRU_REFS_PGOFF and LRU_REFS_MASK, used later are added in
this patch to make the entire patchset less diffy.
A page is added to the youngest generation on faulting. The aging needs
to check the accessed bit at least twice before handing this page over to
the eviction. The first check takes care of the accessed bit set on the
initial fault; the second check makes sure this page has not been used
since then. This protocol, AKA second chance, requires a minimum of two
generations, hence MIN_NR_GENS.
[1] https://dl.acm.org/doi/10.1145/3297858.3304053
[2] https://dl.acm.org/doi/10.1145/3503222.3507731
[3] https://lwn.net/Articles/495543/
[4] https://lwn.net/Articles/815342/
Link: https://lkml.kernel.org/r/20220918080010.2920238-6-yuzhao@google.com
Signed-off-by: Yu Zhao <yuzhao@google.com>
Acked-by: Brian Geffon <bgeffon@google.com>
Acked-by: Jan Alexander Steffens (heftig) <heftig@archlinux.org>
Acked-by: Oleksandr Natalenko <oleksandr@natalenko.name>
Acked-by: Steven Barrett <steven@liquorix.net>
Acked-by: Suleiman Souhlal <suleiman@google.com>
Tested-by: Daniel Byrne <djbyrne@mtu.edu>
Tested-by: Donald Carr <d@chaos-reins.com>
Tested-by: Holger Hoffstätte <holger@applied-asynchrony.com>
Tested-by: Konstantin Kharlamov <Hi-Angel@yandex.ru>
Tested-by: Shuang Zhai <szhai2@cs.rochester.edu>
Tested-by: Sofia Trinh <sofia.trinh@edi.works>
Tested-by: Vaibhav Jain <vaibhav@linux.ibm.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hillf Danton <hdanton@sina.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michael Larabel <Michael@MichaelLarabel.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Mike Rapoport <rppt@linux.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Qi Zheng <zhengqi.arch@bytedance.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2022-09-18 02:00:02 -06:00
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-11 13:47:50 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2016-09-27 10:45:12 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2020-05-20 17:58:16 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2024-08-28 15:55:17 +02:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2026-01-11 12:48:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2026-05-18 22:28:06 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2011-03-22 16:30:52 -07:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2021-11-25 14:05:18 +01:00
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2021-11-25 14:05:18 +01:00
|
|
|
|
2021-11-02 11:10:37 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-05-18 22:28:06 -07:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2022-11-01 10:53:23 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2014-07-07 15:28:51 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2019-10-16 16:47:32 +01:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2020-09-18 10:36:50 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-05-18 22:28:07 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2026-03-31 14:50:24 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
fuse: add zero-copy over io-uring
Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-08-14 11:59:45 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2010-10-26 14:22:27 -07:00
|
|
|
|
fuse: add zero-copy over io-uring
Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-08-14 11:59:45 -07:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2022-03-07 16:30:44 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2022-03-07 16:30:44 +01:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-11-26 12:03:54 +01:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-08 16:38:28 +08:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2024-10-24 10:18:09 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 14:50:24 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2008-11-26 12:03:54 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:00 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: add zero-copy over io-uring
Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-08-14 11:59:45 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2026-03-05 17:05:25 -08:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2026-03-05 17:05:25 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2026-03-05 17:05:25 -08:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:03 +02:00
|
|
|
|
2026-04-30 04:47:46 -07:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2019-06-05 15:50:43 -04:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:03 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-29 17:09:07 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-05 15:50:43 -04:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2015-07-01 16:26:03 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2026-04-30 04:47:46 -07:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2015-07-01 16:26:03 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:53 -07:00
|
|
|
|
|
|
|
|
|
2018-06-21 09:33:40 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
fuse: require /dev/fuse reads to have enough buffer capacity (take 2)
[ This retries commit d4b13963f217 ("fuse: require /dev/fuse reads to have
enough buffer capacity"), which was reverted. In this version we require
only `sizeof(fuse_in_header) + sizeof(fuse_write_in)` instead of 4K for
FUSE request header room, because, contrary to libfuse and kernel client
behaviour, GlusterFS actually provides only so much room for request
header. ]
A FUSE filesystem server queues /dev/fuse sys_read calls to get filesystem
requests to handle. It does not know in advance what would be that request
as it can be anything that client issues - LOOKUP, READ, WRITE, ... Many
requests are short and retrieve data from the filesystem. However WRITE and
NOTIFY_REPLY write data into filesystem.
Before getting into operation phase, FUSE filesystem server and kernel
client negotiate what should be the maximum write size the client will ever
issue. After negotiation the contract in between server/client is that the
filesystem server then should queue /dev/fuse sys_read calls with enough
buffer capacity to receive any client request - WRITE in particular, while
FUSE client should not, in particular, send WRITE requests with >
negotiated max_write payload. FUSE client in kernel and libfuse
historically reserve 4K for request header. However an existing filesystem
server - GlusterFS - was found which reserves only 80 bytes for header room
(= `sizeof(fuse_in_header) + sizeof(fuse_write_in)`).
Since
`sizeof(fuse_in_header) + sizeof(fuse_write_in)` ==
`sizeof(fuse_in_header) + sizeof(fuse_read_in)` ==
`sizeof(fuse_in_header) + sizeof(fuse_notify_retrieve_in)`
is the absolute minimum any sane filesystem should be using for header
room, the contract is that filesystem server should queue sys_reads with
`sizeof(fuse_in_header) + sizeof(fuse_write_in)` + max_write buffer.
If the filesystem server does not follow this contract, what can happen
is that fuse_dev_do_read will see that request size is > buffer size,
and then it will return EIO to client who issued the request but won't
indicate in any way that there is a problem to filesystem server.
This can be hard to diagnose because for some requests, e.g. for
NOTIFY_REPLY which mimics WRITE, there is no client thread that is
waiting for request completion and that EIO goes nowhere, while on
filesystem server side things look like the kernel is not replying back
after successful NOTIFY_RETRIEVE request made by the server.
We can make the problem easy to diagnose if we indicate via error return to
filesystem server when it is violating the contract. This should not
practically cause problems because if a filesystem server is using shorter
buffer, writes to it were already very likely to cause EIO, and if the
filesystem is read-only it should be too following FUSE_MIN_READ_BUFFER
minimum buffer size.
Please see [1] for context where the problem of stuck filesystem was hit
for real (because kernel client was incorrectly sending more than
max_write data with NOTIFY_REPLY; see also previous patch), how the
situation was traced and for more involving patch that did not make it
into the tree.
[1] https://marc.info/?l=linux-fsdevel&m=155057023600853&w=2
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Tested-by: Sander Eikelenboom <linux@eikelenboom.it>
Cc: Han-Wen Nienhuys <hanwen@google.com>
Cc: Jakob Unterwurzacher <jakobunt@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-07-08 17:03:31 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
fuse: require /dev/fuse reads to have enough buffer capacity (take 2)
[ This retries commit d4b13963f217 ("fuse: require /dev/fuse reads to have
enough buffer capacity"), which was reverted. In this version we require
only `sizeof(fuse_in_header) + sizeof(fuse_write_in)` instead of 4K for
FUSE request header room, because, contrary to libfuse and kernel client
behaviour, GlusterFS actually provides only so much room for request
header. ]
A FUSE filesystem server queues /dev/fuse sys_read calls to get filesystem
requests to handle. It does not know in advance what would be that request
as it can be anything that client issues - LOOKUP, READ, WRITE, ... Many
requests are short and retrieve data from the filesystem. However WRITE and
NOTIFY_REPLY write data into filesystem.
Before getting into operation phase, FUSE filesystem server and kernel
client negotiate what should be the maximum write size the client will ever
issue. After negotiation the contract in between server/client is that the
filesystem server then should queue /dev/fuse sys_read calls with enough
buffer capacity to receive any client request - WRITE in particular, while
FUSE client should not, in particular, send WRITE requests with >
negotiated max_write payload. FUSE client in kernel and libfuse
historically reserve 4K for request header. However an existing filesystem
server - GlusterFS - was found which reserves only 80 bytes for header room
(= `sizeof(fuse_in_header) + sizeof(fuse_write_in)`).
Since
`sizeof(fuse_in_header) + sizeof(fuse_write_in)` ==
`sizeof(fuse_in_header) + sizeof(fuse_read_in)` ==
`sizeof(fuse_in_header) + sizeof(fuse_notify_retrieve_in)`
is the absolute minimum any sane filesystem should be using for header
room, the contract is that filesystem server should queue sys_reads with
`sizeof(fuse_in_header) + sizeof(fuse_write_in)` + max_write buffer.
If the filesystem server does not follow this contract, what can happen
is that fuse_dev_do_read will see that request size is > buffer size,
and then it will return EIO to client who issued the request but won't
indicate in any way that there is a problem to filesystem server.
This can be hard to diagnose because for some requests, e.g. for
NOTIFY_REPLY which mimics WRITE, there is no client thread that is
waiting for request completion and that EIO goes nowhere, while on
filesystem server side things look like the kernel is not replying back
after successful NOTIFY_RETRIEVE request made by the server.
We can make the problem easy to diagnose if we indicate via error return to
filesystem server when it is violating the contract. This should not
practically cause problems because if a filesystem server is using shorter
buffer, writes to it were already very likely to cause EIO, and if the
filesystem is read-only it should be too following FUSE_MIN_READ_BUFFER
minimum buffer size.
Please see [1] for context where the problem of stuck filesystem was hit
for real (because kernel client was incorrectly sending more than
max_write data with NOTIFY_REPLY; see also previous patch), how the
situation was traced and for more involving patch that did not make it
into the tree.
[1] https://marc.info/?l=linux-fsdevel&m=155057023600853&w=2
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Tested-by: Sander Eikelenboom <linux@eikelenboom.it>
Cc: Han-Wen Nienhuys <hanwen@google.com>
Cc: Jakob Unterwurzacher <jakobunt@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-07-08 17:03:31 +00:00
|
|
|
|
|
|
|
|
|
2006-01-06 00:19:40 -08:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-04-10 22:54:53 -07:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:03 +02:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:03 +02:00
|
|
|
|
2017-11-09 21:23:35 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2017-11-09 21:23:35 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-05 17:05:25 -08:00
|
|
|
|
|
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
|
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
|
|
|
|
|
2010-12-07 20:16:56 +01:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:26:02 +02:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2015-07-01 16:26:02 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
|
|
|
|
|
2017-09-12 16:57:53 +02:00
|
|
|
|
2006-01-06 00:19:40 -08:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2006-01-06 00:19:40 -08:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2006-01-06 00:19:40 -08:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2006-01-06 00:19:40 -08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2021-06-22 09:15:35 +02:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2021-06-22 09:15:35 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2006-01-06 00:19:40 -08:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:26:04 +02:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
2007-10-16 23:31:05 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2007-10-16 23:31:05 -07:00
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
|
|
|
|
|
2018-09-25 12:28:55 +03:00
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
2026-06-05 11:26:47 +02:00
|
|
|
|
2018-09-28 16:43:22 +02:00
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2015-07-01 16:26:05 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-01-12 05:22:16 +01:00
|
|
|
|
|
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2026-03-11 21:02:41 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-01-12 05:22:16 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2026-03-11 21:02:41 +01:00
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-04-22 09:31:23 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2026-04-22 09:31:23 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2022-05-22 14:59:25 -04:00
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2025-02-20 12:16:59 -08:00
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-09-17 22:56:25 -04:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2026-02-21 16:37:42 -08:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-20 12:16:59 -08:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fs/pipe: add simpler helpers for common cases
The fix to atomically read the pipe head and tail state when not holding
the pipe mutex has caused a number of headaches due to the size change
of the involved types.
It turns out that we don't have _that_ many places that access these
fields directly and were affected, but we have more than we strictly
should have, because our low-level helper functions have been designed
to have intimate knowledge of how the pipes work.
And as a result, that random noise of direct 'pipe->head' and
'pipe->tail' accesses makes it harder to pinpoint any actual potential
problem spots remaining.
For example, we didn't have a "is the pipe full" helper function, but
instead had a "given these pipe buffer indexes and this pipe size, is
the pipe full". That's because some low-level pipe code does actually
want that much more complicated interface.
But most other places literally just want a "is the pipe full" helper,
and not having it meant that those places ended up being unnecessarily
much too aware of this all.
It would have been much better if only the very core pipe code that
cared had been the one aware of this all.
So let's fix it - better late than never. This just introduces the
trivial wrappers for "is this pipe full or empty" and to get how many
pipe buffers are used, so that instead of writing
if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
the places that literally just want to know if a pipe is full can just
say
if (pipe_is_full(pipe))
instead. The existing trivial cases were converted with a 'sed' script.
This cuts down on the places that access pipe->head and pipe->tail
directly outside of the pipe code (and core splice code) quite a lot.
The splice code in particular still revels in doing the direct low-level
accesses, and the fuse fuse_dev_splice_write() code also seems a bit
unnecessarily eager to go very low-level, but it's at least a bit better
than it used to be.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2025-03-06 18:25:35 -10:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2016-09-17 22:56:25 -04:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
2016-09-17 22:56:25 -04:00
|
|
|
|
2014-01-22 19:36:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-09-17 22:56:25 -04:00
|
|
|
|
2017-02-16 15:08:20 +01:00
|
|
|
|
2016-09-17 22:56:25 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2016-09-17 22:56:25 -04:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 15:29:47 +03:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2018-07-17 19:00:34 +03:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 14:50:24 +02:00
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
2026-07-09 08:37:05 +02:00
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-09 08:37:05 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
2024-07-25 10:53:34 -07:00
|
|
|
|
|
|
|
|
|
2025-01-20 02:28:54 +01:00
|
|
|
|
2024-07-25 10:53:34 -07:00
|
|
|
|
|
|
|
|
|
2026-06-09 09:58:51 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-09 08:37:05 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-09 09:58:51 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
fuse: give wakeup hints to the scheduler for synchronous requests
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2026-07-31 15:01:02 +08:00
|
|
|
|
2024-01-09 17:24:42 +08:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2025-01-20 02:29:02 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
2013-07-30 22:50:01 -04:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
2018-09-11 13:12:05 +03:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:00 +01:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2025-01-20 02:29:01 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-06-21 09:34:25 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-06-21 09:33:40 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2008-11-26 12:03:55 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2008-11-26 12:03:55 +01:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2008-11-26 12:03:55 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-31 14:50:24 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-16 11:31:43 -07:00
|
|
|
|
|
|
|
|
|
2026-03-31 14:50:24 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2025-09-01 17:16:26 +02:00
|
|
|
|
2008-11-26 12:03:55 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-22 09:15:35 +02:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:02 +01:00
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-09-11 13:12:05 +03:00
|
|
|
|
|
|
|
|
|
2018-09-25 12:52:42 +03:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
|
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2006-06-25 05:48:54 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:26:04 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2019-09-10 15:04:11 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-01-20 02:29:00 +01:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
2015-07-01 16:26:04 +02:00
|
|
|
|
2015-07-01 16:25:58 +02:00
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2015-07-01 16:26:07 +02:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2018-11-08 12:05:36 +03:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
2022-05-22 14:59:25 -04:00
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2025-02-20 12:16:59 -08:00
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-06 07:53:25 -10:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
2018-07-17 19:00:33 +03:00
|
|
|
|
|
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
|
|
|
|
|
|
2025-03-06 07:53:25 -10:00
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
|
2026-02-21 16:37:42 -08:00
|
|
|
|
2018-07-17 19:00:33 +03:00
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2018-07-17 19:00:33 +03:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-06 07:53:25 -10:00
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
2019-04-05 14:02:10 -07:00
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-06 07:53:25 -10:00
|
|
|
|
2019-08-19 09:53:50 +03:00
|
|
|
|
|
|
|
|
|
2025-03-06 07:53:25 -10:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2019-04-05 14:02:10 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-20 12:16:59 -08:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2015-04-03 22:06:08 -04:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2025-05-12 15:58:30 -07:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2019-01-12 02:39:05 +01:00
|
|
|
|
2019-04-05 14:02:10 -07:00
|
|
|
|
2021-11-02 11:10:37 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-01-12 02:39:05 +01:00
|
|
|
|
2016-09-27 10:45:12 +02:00
|
|
|
|
2018-07-17 19:00:34 +03:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-07-03 01:02:18 -04:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2018-02-11 14:34:03 -08:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-06 12:15:20 +03:00
|
|
|
|
2025-01-20 02:28:54 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2015-07-01 16:26:04 +02:00
|
|
|
|
2020-04-20 17:59:34 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:25:59 +02:00
|
|
|
|
|
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
2015-07-01 16:25:59 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-01 10:43:50 +02:00
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
2015-07-01 16:25:59 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2015-07-01 16:26:01 +02:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2015-07-01 16:25:59 +02:00
|
|
|
|
2018-07-26 16:13:12 +02:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
2015-07-01 16:25:59 +02:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
|
|
|
|
|
2025-01-22 13:55:27 -08:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-08-27 18:29:56 +03:00
|
|
|
|
2026-03-31 15:50:29 +02:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2018-07-26 16:13:12 +02:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:06 +02:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2015-07-01 16:25:59 +02:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:02 +02:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
2015-07-01 16:26:02 +02:00
|
|
|
|
2018-07-26 16:13:12 +02:00
|
|
|
|
2017-01-12 12:04:04 -08:00
|
|
|
|
2018-07-26 16:13:12 +02:00
|
|
|
|
2015-07-01 16:26:02 +02:00
|
|
|
|
2019-06-05 15:50:43 -04:00
|
|
|
|
fuse: fix deadlock with aio poll and fuse_iqueue::waitq.lock
When IOCB_CMD_POLL is used on the FUSE device, aio_poll() disables IRQs
and takes kioctx::ctx_lock, then fuse_iqueue::waitq.lock.
This may have to wait for fuse_iqueue::waitq.lock to be released by one
of many places that take it with IRQs enabled. Since the IRQ handler
may take kioctx::ctx_lock, lockdep reports that a deadlock is possible.
Fix it by protecting the state of struct fuse_iqueue with a separate
spinlock, and only accessing fuse_iqueue::waitq using the versions of
the waitqueue functions which do IRQ-safe locking internally.
Reproducer:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <linux/aio_abi.h>
int main()
{
char opts[128];
int fd = open("/dev/fuse", O_RDWR);
aio_context_t ctx = 0;
struct iocb cb = { .aio_lio_opcode = IOCB_CMD_POLL, .aio_fildes = fd };
struct iocb *cbp = &cb;
sprintf(opts, "fd=%d,rootmode=040000,user_id=0,group_id=0", fd);
mkdir("mnt", 0700);
mount("foo", "mnt", "fuse", 0, opts);
syscall(__NR_io_setup, 1, &ctx);
syscall(__NR_io_submit, ctx, 1, &cbp);
}
Beginning of lockdep output:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.3.0-rc5 #9 Not tainted
-----------------------------------------------------
syz_fuse/135 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
000000003590ceda (&fiq->waitq){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: aio_poll fs/aio.c:1751 [inline]
000000003590ceda (&fiq->waitq){+.+.}, at: __io_submit_one.constprop.0+0x203/0x5b0 fs/aio.c:1825
and this task is already holding:
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: spin_lock_irq include/linux/spinlock.h:363 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: aio_poll fs/aio.c:1749 [inline]
0000000075037284 (&(&ctx->ctx_lock)->rlock){..-.}, at: __io_submit_one.constprop.0+0x1f4/0x5b0 fs/aio.c:1825
which would create a new lock dependency:
(&(&ctx->ctx_lock)->rlock){..-.} -> (&fiq->waitq){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&ctx->ctx_lock)->rlock){..-.}
[...]
Reported-by: syzbot+af05535bb79520f95431@syzkaller.appspotmail.com
Reported-by: syzbot+d86c4426a01f60feddc7@syzkaller.appspotmail.com
Fixes: bfe4037e722e ("aio: implement IOCB_CMD_POLL")
Cc: <stable@vger.kernel.org> # v4.19+
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
2019-09-08 20:15:18 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:02 +02:00
|
|
|
|
2026-03-31 13:49:22 +02:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:02 +02:00
|
|
|
|
2025-01-20 02:28:54 +01:00
|
|
|
|
2025-01-20 02:29:04 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2025-01-20 02:29:04 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2006-01-16 22:14:41 -08:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2018-11-09 15:52:16 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2025-01-20 02:29:04 +01:00
|
|
|
|
2026-03-24 16:12:28 +01:00
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
|
|
|
|
|
2009-04-14 10:54:53 +09:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2026-03-11 21:02:41 +01:00
|
|
|
|
2026-03-11 22:05:17 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
2026-03-12 12:19:10 +01:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
|
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2018-09-11 13:12:14 +03:00
|
|
|
|
|
|
|
|
|
2018-07-26 16:13:11 +02:00
|
|
|
|
|
|
|
|
|
2026-03-12 12:19:10 +01:00
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
|
|
|
|
|
2026-03-12 12:19:10 +01:00
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
|
|
|
|
|
2026-03-12 12:19:10 +01:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:09 +02:00
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2006-04-10 22:54:52 -07:00
|
|
|
|
2026-03-11 22:05:17 +01:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2009-04-14 10:54:53 +09:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2006-04-10 22:54:52 -07:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2006-04-10 22:54:52 -07:00
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2006-04-10 22:54:52 -07:00
|
|
|
|
|
|
|
|
|
2023-08-31 23:07:55 +03:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2021-01-25 15:30:51 +00:00
|
|
|
|
2026-03-12 12:19:10 +01:00
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2023-08-31 23:07:55 +03:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2024-07-19 21:19:02 -04:00
|
|
|
|
|
|
|
|
|
2023-08-31 23:07:55 +03:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2023-08-31 23:07:55 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 20:19:55 +02:00
|
|
|
|
|
|
|
|
|
2022-08-21 14:02:15 -04:00
|
|
|
|
2026-04-02 20:19:55 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-31 23:07:55 +03:00
|
|
|
|
2026-04-02 14:49:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 22:57:49 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-12 12:19:10 +01:00
|
|
|
|
|
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
|
|
|
|
|
2023-09-11 17:09:27 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2023-09-11 17:09:27 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2023-09-11 17:09:27 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2023-09-11 17:09:27 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-30 14:27:23 +02:00
|
|
|
|
2023-09-11 17:09:27 +03:00
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2026-03-11 21:02:41 +01:00
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
2026-03-31 16:32:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
2023-08-31 23:07:55 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-09-11 17:09:27 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-08-31 23:07:55 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-13 12:20:49 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-22 13:10:44 +02:00
|
|
|
|
2025-05-13 12:20:49 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-01 09:02:14 +02:00
|
|
|
|
2025-05-13 12:20:49 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-03-28 01:56:42 -08:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2015-01-12 05:22:16 +01:00
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
2010-05-25 15:06:07 +02:00
|
|
|
|
2015-04-03 21:53:39 -04:00
|
|
|
|
fuse: support splice() writing to fuse device
Allow userspace filesystem implementation to use splice() to write to
the fuse device. The semantics of using splice() are:
1) buffer the message header and data in a temporary pipe
2) with a *single* splice() call move the message from the temporary pipe
to the fuse device
The READ reply message has the most interesting use for this, since
now the data from an arbitrary file descriptor (which could be a
regular file, a block device or a socket) can be tranferred into the
fuse device without having to go through a userspace buffer. It will
also allow zero copy moving of pages.
One caveat is that the protocol on the fuse device requires the length
of the whole message to be written into the header. But the length of
the data transferred into the temporary pipe may not be known in
advance. The current library implementation works around this by
using vmplice to write the header and modifying the header after
splicing the data into the pipe (error handling omitted):
struct fuse_out_header out;
iov.iov_base = &out;
iov.iov_len = sizeof(struct fuse_out_header);
vmsplice(pip[1], &iov, 1, 0);
len = splice(input_fd, input_offset, pip[1], NULL, len, 0);
/* retrospectively modify the header: */
out.len = len + sizeof(struct fuse_out_header);
splice(pip[0], NULL, fuse_chan_fd(req->ch), NULL, out.len, flags);
This works since vmsplice only saves a pointer to the data, it does
not copy the data itself.
Since pipes are currently limited to 16 pages and messages need to be
spliced atomically, the length of the data is limited to 15 pages (or
60kB for 4k pages).
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
2010-05-25 15:06:06 +02:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
2006-04-10 22:54:52 -07:00
|
|
|
|
2015-07-01 16:26:08 +02:00
|
|
|
|
2018-09-11 21:59:08 +02:00
|
|
|
|
2025-01-20 02:29:10 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-13 12:20:49 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
2009-04-14 10:54:53 +09:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-07-20 10:11:58 +09:00
|
|
|
|
2005-09-09 13:10:27 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
