2019-05-23 11:14:39 +02:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-07-25 12:06:13 -05:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 17:04:11 +09:00
|
|
|
|
2011-07-03 16:20:30 -04:00
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2021-08-10 12:26:05 -04:00
|
|
|
|
2016-02-02 21:46:40 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-06-12 22:16:35 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-03-30 21:55:22 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-06-12 22:16:35 -06:00
|
|
|
|
|
|
|
|
|
2009-03-30 21:55:22 -06:00
|
|
|
|
2009-06-12 22:16:35 -06:00
|
|
|
|
2009-01-22 16:42:57 +01:00
|
|
|
|
2010-02-24 14:22:22 -06:00
|
|
|
|
2018-11-21 18:03:22 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2009-06-12 22:16:35 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:22 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:45 +08:00
|
|
|
|
2021-10-19 15:01:48 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2021-06-04 13:53:44 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2015-01-15 13:33:31 +02:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:26 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:42 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-20 02:10:44 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-11-07 15:49:24 -06:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
virtio_ring: shadow available ring flags & index
Improves cacheline transfer flow of available ring header.
Virtqueues are implemented as a pair of rings, one producer->consumer
avail ring and one consumer->producer used ring; preceding the
avail ring in memory are two contiguous u16 fields -- avail->flags
and avail->idx. A producer posts work by writing to avail->idx and
a consumer reads avail->idx.
The flags and idx fields only need to be written by a producer CPU
and only read by a consumer CPU; when the producer and consumer are
running on different CPUs and the virtio_ring code is structured to
only have source writes/sink reads, we can continuously transfer the
avail header cacheline between 'M' states between cores. This flow
optimizes core -> core bandwidth on certain CPUs.
(see: "Software Optimization Guide for AMD Family 15h Processors",
Section 11.6; similar language appears in the 10h guide and should
apply to CPUs w/ exclusive caches, using LLC as a transfer cache)
Unfortunately the existing virtio_ring code issued reads to the
avail->idx and read-modify-writes to avail->flags on the producer.
This change shadows the flags and index fields in producer memory;
the vring code now reads from the shadows and only ever writes to
avail->flags and avail->idx, allowing the cacheline to transfer
core -> core optimally.
In a concurrent version of vring_bench, the time required for
10,000,000 buffer checkout/returns was reduced by ~2% (average
across many runs) on an AMD Piledriver (15h) CPU:
(w/o shadowing):
Performance counter stats for './vring_bench':
5,451,082,016 L1-dcache-loads
...
2.221477739 seconds time elapsed
(w/ shadowing):
Performance counter stats for './vring_bench':
5,405,701,361 L1-dcache-loads
...
2.168405376 seconds time elapsed
The further away (in a NUMA sense) virtio producers and consumers are
from each other, the more we expect to benefit. Physical implementations
of virtio devices and implementations of virtio where the consumer polls
vring avail indexes (vhost) should also benefit.
Signed-off-by: Venkatesh Srinivas <venkateshs@google.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2015-11-10 16:21:07 -08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-02-01 17:13:57 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:45 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
virtio_ring: shadow available ring flags & index
Improves cacheline transfer flow of available ring header.
Virtqueues are implemented as a pair of rings, one producer->consumer
avail ring and one consumer->producer used ring; preceding the
avail ring in memory are two contiguous u16 fields -- avail->flags
and avail->idx. A producer posts work by writing to avail->idx and
a consumer reads avail->idx.
The flags and idx fields only need to be written by a producer CPU
and only read by a consumer CPU; when the producer and consumer are
running on different CPUs and the virtio_ring code is structured to
only have source writes/sink reads, we can continuously transfer the
avail header cacheline between 'M' states between cores. This flow
optimizes core -> core bandwidth on certain CPUs.
(see: "Software Optimization Guide for AMD Family 15h Processors",
Section 11.6; similar language appears in the 10h guide and should
apply to CPUs w/ exclusive caches, using LLC as a transfer cache)
Unfortunately the existing virtio_ring code issued reads to the
avail->idx and read-modify-writes to avail->flags on the producer.
This change shadows the flags and index fields in producer memory;
the vring code now reads from the shadows and only ever writes to
avail->flags and avail->idx, allowing the cacheline to transfer
core -> core optimally.
In a concurrent version of vring_bench, the time required for
10,000,000 buffer checkout/returns was reduced by ~2% (average
across many runs) on an AMD Piledriver (15h) CPU:
(w/o shadowing):
Performance counter stats for './vring_bench':
5,451,082,016 L1-dcache-loads
...
2.221477739 seconds time elapsed
(w/ shadowing):
Performance counter stats for './vring_bench':
5,405,701,361 L1-dcache-loads
...
2.168405376 seconds time elapsed
The further away (in a NUMA sense) virtio producers and consumers are
from each other, the more we expect to benefit. Physical implementations
of virtio devices and implementations of virtio where the consumer polls
vring avail indexes (vhost) should also benefit.
Signed-off-by: Venkatesh Srinivas <venkateshs@google.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2015-11-10 16:21:07 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2013-10-29 09:38:50 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2022-03-28 18:58:16 +08:00
|
|
|
|
2018-11-21 18:03:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:35 -08:00
|
|
|
|
virtio: new feature to detect IOMMU device quirk
The interaction between virtio and IOMMUs is messy.
On most systems with virtio, physical addresses match bus addresses,
and it doesn't particularly matter which one we use to program
the device.
On some systems, including Xen and any system with a physical device
that speaks virtio behind a physical IOMMU, we must program the IOMMU
for virtio DMA to work at all.
On other systems, including SPARC and PPC64, virtio-pci devices are
enumerated as though they are behind an IOMMU, but the virtio host
ignores the IOMMU, so we must either pretend that the IOMMU isn't
there or somehow map everything as the identity.
Add a feature bit to detect that quirk: VIRTIO_F_IOMMU_PLATFORM.
Any device with this feature bit set to 0 needs a quirk and has to be
passed physical addresses (as opposed to bus addresses) even though
the device is behind an IOMMU.
Note: it has to be a per-device quirk because for example, there could
be a mix of passed-through and virtual virtio devices. As another
example, some devices could be implemented by an out of process
hypervisor backend (in case of qemu vhost, or vhost-user) and so support
for an IOMMU needs to be coded up separately.
It would be cleanest to handle this in IOMMU core code, but that needs
per-device DMA ops. While we are waiting for that to be implemented, use
a work-around in virtio core.
Note: a "noiommu" feature is a quirk - add a wrapper to make
that clear.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-04-18 12:58:14 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:35 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
virtio: new feature to detect IOMMU device quirk
The interaction between virtio and IOMMUs is messy.
On most systems with virtio, physical addresses match bus addresses,
and it doesn't particularly matter which one we use to program
the device.
On some systems, including Xen and any system with a physical device
that speaks virtio behind a physical IOMMU, we must program the IOMMU
for virtio DMA to work at all.
On other systems, including SPARC and PPC64, virtio-pci devices are
enumerated as though they are behind an IOMMU, but the virtio host
ignores the IOMMU, so we must either pretend that the IOMMU isn't
there or somehow map everything as the identity.
Add a feature bit to detect that quirk: VIRTIO_F_IOMMU_PLATFORM.
Any device with this feature bit set to 0 needs a quirk and has to be
passed physical addresses (as opposed to bus addresses) even though
the device is behind an IOMMU.
Note: it has to be a per-device quirk because for example, there could
be a mix of passed-through and virtual virtio devices. As another
example, some devices could be implemented by an out of process
hypervisor backend (in case of qemu vhost, or vhost-user) and so support
for an IOMMU needs to be coded up separately.
It would be cleanest to handle this in IOMMU core code, but that needs
per-device DMA ops. While we are waiting for that to be implemented, use
a work-around in virtio core.
Note: a "noiommu" feature is a quirk - add a wrapper to make
that clear.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-04-18 12:58:14 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:35 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-06-24 19:17:04 -04:00
|
|
|
|
virtio: new feature to detect IOMMU device quirk
The interaction between virtio and IOMMUs is messy.
On most systems with virtio, physical addresses match bus addresses,
and it doesn't particularly matter which one we use to program
the device.
On some systems, including Xen and any system with a physical device
that speaks virtio behind a physical IOMMU, we must program the IOMMU
for virtio DMA to work at all.
On other systems, including SPARC and PPC64, virtio-pci devices are
enumerated as though they are behind an IOMMU, but the virtio host
ignores the IOMMU, so we must either pretend that the IOMMU isn't
there or somehow map everything as the identity.
Add a feature bit to detect that quirk: VIRTIO_F_IOMMU_PLATFORM.
Any device with this feature bit set to 0 needs a quirk and has to be
passed physical addresses (as opposed to bus addresses) even though
the device is behind an IOMMU.
Note: it has to be a per-device quirk because for example, there could
be a mix of passed-through and virtual virtio devices. As another
example, some devices could be implemented by an out of process
hypervisor backend (in case of qemu vhost, or vhost-user) and so support
for an IOMMU needs to be coded up separately.
It would be cleanest to handle this in IOMMU core code, but that needs
per-device DMA ops. While we are waiting for that to be implemented, use
a work-around in virtio core.
Note: a "noiommu" feature is a quirk - add a wrapper to make
that clear.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-04-18 12:58:14 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:40 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:35 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-02-07 12:59:16 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-12-01 11:20:18 +00:00
|
|
|
|
2019-02-07 12:59:16 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-10-31 00:38:21 +02:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:26 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:26 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:26 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:26 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:55 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
|
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
2012-10-19 14:03:33 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-12-01 15:32:49 +01:00
|
|
|
|
2012-10-19 14:03:33 +01:00
|
|
|
|
treewide: kmalloc() -> kmalloc_array()
The kmalloc() function has a 2-factor argument form, kmalloc_array(). This
patch replaces cases of:
kmalloc(a * b, gfp)
with:
kmalloc_array(a * b, gfp)
as well as handling cases of:
kmalloc(a * b * c, gfp)
with:
kmalloc(array3_size(a, b, c), gfp)
as it's slightly less ugly than:
kmalloc_array(array_size(a, b), c, gfp)
This does, however, attempt to ignore constant size factors like:
kmalloc(4 * 1024, gfp)
though any constants defined via macros get caught up in the conversion.
Any factors with a sizeof() of "unsigned char", "char", and "u8" were
dropped, since they're redundant.
The tools/ directory was manually excluded, since it has its own
implementation of kmalloc().
The Coccinelle script used for this was:
// Fix redundant parens around sizeof().
@@
type TYPE;
expression THING, E;
@@
(
kmalloc(
- (sizeof(TYPE)) * E
+ sizeof(TYPE) * E
, ...)
|
kmalloc(
- (sizeof(THING)) * E
+ sizeof(THING) * E
, ...)
)
// Drop single-byte sizes and redundant parens.
@@
expression COUNT;
typedef u8;
typedef __u8;
@@
(
kmalloc(
- sizeof(u8) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(__u8) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(char) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(unsigned char) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(u8) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(__u8) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(char) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(unsigned char) * COUNT
+ COUNT
, ...)
)
// 2-factor product with sizeof(type/expression) and identifier or constant.
@@
type TYPE;
expression THING;
identifier COUNT_ID;
constant COUNT_CONST;
@@
(
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (COUNT_ID)
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * COUNT_ID
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (COUNT_CONST)
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * COUNT_CONST
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (COUNT_ID)
+ COUNT_ID, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * COUNT_ID
+ COUNT_ID, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (COUNT_CONST)
+ COUNT_CONST, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * COUNT_CONST
+ COUNT_CONST, sizeof(THING)
, ...)
)
// 2-factor product, only identifiers.
@@
identifier SIZE, COUNT;
@@
- kmalloc
+ kmalloc_array
(
- SIZE * COUNT
+ COUNT, SIZE
, ...)
// 3-factor product with 1 sizeof(type) or sizeof(expression), with
// redundant parens removed.
@@
expression THING;
identifier STRIDE, COUNT;
type TYPE;
@@
(
kmalloc(
- sizeof(TYPE) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(THING) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
)
// 3-factor product with 2 sizeof(variable), with redundant parens removed.
@@
expression THING1, THING2;
identifier COUNT;
type TYPE1, TYPE2;
@@
(
kmalloc(
- sizeof(TYPE1) * sizeof(TYPE2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kmalloc(
- sizeof(THING1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(THING1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
)
// 3-factor product, only identifiers, with redundant parens removed.
@@
identifier STRIDE, SIZE, COUNT;
@@
(
kmalloc(
- (COUNT) * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
)
// Any remaining multi-factor products, first at least 3-factor products,
// when they're not all constants...
@@
expression E1, E2, E3;
constant C1, C2, C3;
@@
(
kmalloc(C1 * C2 * C3, ...)
|
kmalloc(
- (E1) * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- (E1) * (E2) * E3
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- (E1) * (E2) * (E3)
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- E1 * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
)
// And then all remaining 2 factors products when they're not all constants,
// keeping sizeof() as the second factor argument.
@@
expression THING, E1, E2;
type TYPE;
constant C1, C2, C3;
@@
(
kmalloc(sizeof(THING) * C2, ...)
|
kmalloc(sizeof(TYPE) * C2, ...)
|
kmalloc(C1 * C2 * C3, ...)
|
kmalloc(C1 * C2, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (E2)
+ E2, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * E2
+ E2, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (E2)
+ E2, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * E2
+ E2, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- (E1) * E2
+ E1, E2
, ...)
|
- kmalloc
+ kmalloc_array
(
- (E1) * (E2)
+ E1, E2
, ...)
|
- kmalloc
+ kmalloc_array
(
- E1 * E2
+ E1, E2
, ...)
)
Signed-off-by: Kees Cook <keescook@chromium.org>
2018-06-12 13:55:00 -07:00
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2014-10-22 15:42:09 +03:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:48 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:48 +08:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:48 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:48 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
treewide: Remove uninitialized_var() usage
Using uninitialized_var() is dangerous as it papers over real bugs[1]
(or can in the future), and suppresses unrelated compiler warnings
(e.g. "unused variable"). If the compiler thinks it is uninitialized,
either simply initialize the variable or make compiler changes.
In preparation for removing[2] the[3] macro[4], remove all remaining
needless uses with the following script:
git grep '\buninitialized_var\b' | cut -d: -f1 | sort -u | \
xargs perl -pi -e \
's/\buninitialized_var\(([^\)]+)\)/\1/g;
s:\s*/\* (GCC be quiet|to make compiler happy) \*/$::g;'
drivers/video/fbdev/riva/riva_hw.c was manually tweaked to avoid
pathological white-space.
No outstanding warnings were found building allmodconfig with GCC 9.3.0
for x86_64, i386, arm64, arm, powerpc, powerpc64le, s390x, mips, sparc64,
alpha, and m68k.
[1] https://lore.kernel.org/lkml/20200603174714.192027-1-glider@google.com/
[2] https://lore.kernel.org/lkml/CA+55aFw+Vbj0i=1TGqCR5vQkCzWJ0QxK6CernOU6eedsudAixw@mail.gmail.com/
[3] https://lore.kernel.org/lkml/CA+55aFwgbgqhbp1fkxvRKEpzyR5J8n1vKT1VZdz9knmPuXhOeg@mail.gmail.com/
[4] https://lore.kernel.org/lkml/CA+55aFz2500WfbKXAx8s67wrm9=yVJu65TpLgN_ybYNv0VEOKA@mail.gmail.com/
Reviewed-by: Leon Romanovsky <leonro@mellanox.com> # drivers/infiniband and mlx4/mlx5
Acked-by: Jason Gunthorpe <jgg@mellanox.com> # IB
Acked-by: Kalle Valo <kvalo@codeaurora.org> # wireless drivers
Reviewed-by: Chao Yu <yuchao0@huawei.com> # erofs
Signed-off-by: Kees Cook <keescook@chromium.org>
2020-06-03 13:09:38 -07:00
|
|
|
|
2010-07-26 16:55:30 +09:30
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2017-02-07 06:15:13 +02:00
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
2014-03-13 11:23:40 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:22 +08:00
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
|
|
|
|
|
2022-03-28 18:58:16 +08:00
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2017-08-10 17:56:51 +01:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2017-08-10 17:56:51 +01:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-03-28 18:58:17 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2008-07-25 12:06:04 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2008-07-25 12:06:04 -05:00
|
|
|
|
2016-08-02 14:16:31 +00:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2014-09-11 10:17:37 +09:30
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:48 +08:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2014-09-11 10:17:37 +09:30
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:48 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2014-10-22 15:42:09 +03:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
2021-10-26 15:31:00 +02:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:48 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
2014-09-11 10:17:38 +09:30
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2017-07-19 16:54:45 +08:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
|
|
|
|
|
2013-03-18 13:22:19 +10:30
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
|
|
|
|
|
2015-02-11 15:01:13 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-10-16 23:56:15 +10:30
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-09-06 16:59:01 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-08-03 07:18:51 +03:00
|
|
|
|
2019-11-14 13:46:46 +01:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2013-03-20 15:37:09 +10:30
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2011-05-20 02:10:44 +03:00
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2012-01-20 16:17:08 +08:00
|
|
|
|
|
|
|
|
|
2013-03-18 13:22:19 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:22 +08:00
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2016-11-22 13:51:50 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2012-10-16 23:56:14 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2012-10-16 23:56:14 +10:30
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2017-02-07 06:15:13 +02:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
|
|
|
|
|
2017-02-07 06:15:13 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2017-02-07 06:15:13 +02:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2017-02-07 06:15:13 +02:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2016-02-02 21:46:36 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-05-02 21:50:43 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-10-25 15:28:53 +02:00
|
|
|
|
2013-03-18 13:22:19 +10:30
|
|
|
|
2009-10-25 15:28:53 +02:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2011-05-20 02:10:44 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2015-12-17 12:20:39 +02:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2015-12-17 12:20:39 +02:00
|
|
|
|
2011-05-20 02:10:44 +03:00
|
|
|
|
2018-11-21 18:03:22 +08:00
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-07-09 13:19:18 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2016-08-31 14:00:04 +02:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
virtio_ring: shadow available ring flags & index
Improves cacheline transfer flow of available ring header.
Virtqueues are implemented as a pair of rings, one producer->consumer
avail ring and one consumer->producer used ring; preceding the
avail ring in memory are two contiguous u16 fields -- avail->flags
and avail->idx. A producer posts work by writing to avail->idx and
a consumer reads avail->idx.
The flags and idx fields only need to be written by a producer CPU
and only read by a consumer CPU; when the producer and consumer are
running on different CPUs and the virtio_ring code is structured to
only have source writes/sink reads, we can continuously transfer the
avail header cacheline between 'M' states between cores. This flow
optimizes core -> core bandwidth on certain CPUs.
(see: "Software Optimization Guide for AMD Family 15h Processors",
Section 11.6; similar language appears in the 10h guide and should
apply to CPUs w/ exclusive caches, using LLC as a transfer cache)
Unfortunately the existing virtio_ring code issued reads to the
avail->idx and read-modify-writes to avail->flags on the producer.
This change shadows the flags and index fields in producer memory;
the vring code now reads from the shadows and only ever writes to
avail->flags and avail->idx, allowing the cacheline to transfer
core -> core optimally.
In a concurrent version of vring_bench, the time required for
10,000,000 buffer checkout/returns was reduced by ~2% (average
across many runs) on an AMD Piledriver (15h) CPU:
(w/o shadowing):
Performance counter stats for './vring_bench':
5,451,082,016 L1-dcache-loads
...
2.221477739 seconds time elapsed
(w/ shadowing):
Performance counter stats for './vring_bench':
5,405,701,361 L1-dcache-loads
...
2.168405376 seconds time elapsed
The further away (in a NUMA sense) virtio producers and consumers are
from each other, the more we expect to benefit. Physical implementations
of virtio devices and implementations of virtio where the consumer polls
vring avail indexes (vhost) should also benefit.
Signed-off-by: Venkatesh Srinivas <venkateshs@google.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2015-11-10 16:21:07 -08:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2013-07-09 13:19:18 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-05-20 02:11:14 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-08-31 14:00:04 +02:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2016-08-31 14:00:04 +02:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
virtio_ring: shadow available ring flags & index
Improves cacheline transfer flow of available ring header.
Virtqueues are implemented as a pair of rings, one producer->consumer
avail ring and one consumer->producer used ring; preceding the
avail ring in memory are two contiguous u16 fields -- avail->flags
and avail->idx. A producer posts work by writing to avail->idx and
a consumer reads avail->idx.
The flags and idx fields only need to be written by a producer CPU
and only read by a consumer CPU; when the producer and consumer are
running on different CPUs and the virtio_ring code is structured to
only have source writes/sink reads, we can continuously transfer the
avail header cacheline between 'M' states between cores. This flow
optimizes core -> core bandwidth on certain CPUs.
(see: "Software Optimization Guide for AMD Family 15h Processors",
Section 11.6; similar language appears in the 10h guide and should
apply to CPUs w/ exclusive caches, using LLC as a transfer cache)
Unfortunately the existing virtio_ring code issued reads to the
avail->idx and read-modify-writes to avail->flags on the producer.
This change shadows the flags and index fields in producer memory;
the vring code now reads from the shadows and only ever writes to
avail->flags and avail->idx, allowing the cacheline to transfer
core -> core optimally.
In a concurrent version of vring_bench, the time required for
10,000,000 buffer checkout/returns was reduced by ~2% (average
across many runs) on an AMD Piledriver (15h) CPU:
(w/o shadowing):
Performance counter stats for './vring_bench':
5,451,082,016 L1-dcache-loads
...
2.221477739 seconds time elapsed
(w/ shadowing):
Performance counter stats for './vring_bench':
5,405,701,361 L1-dcache-loads
...
2.168405376 seconds time elapsed
The further away (in a NUMA sense) virtio producers and consumers are
from each other, the more we expect to benefit. Physical implementations
of virtio devices and implementations of virtio where the consumer polls
vring avail indexes (vhost) should also benefit.
Signed-off-by: Venkatesh Srinivas <venkateshs@google.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2015-11-10 16:21:07 -08:00
|
|
|
|
2011-05-20 02:11:14 +03:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2015-12-17 12:20:39 +02:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2015-12-17 12:20:39 +02:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
2011-05-20 02:11:14 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2010-01-18 19:15:23 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2010-01-18 19:15:23 +05:30
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2010-01-18 19:15:23 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2010-01-18 19:15:23 +05:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-08 14:33:22 +02:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:54 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:54 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:54 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:54 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:56 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-10-20 19:23:22 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-10-20 19:23:23 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-03-12 15:06:53 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:44 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-11-14 13:46:46 +01:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
treewide: Remove uninitialized_var() usage
Using uninitialized_var() is dangerous as it papers over real bugs[1]
(or can in the future), and suppresses unrelated compiler warnings
(e.g. "unused variable"). If the compiler thinks it is uninitialized,
either simply initialize the variable or make compiler changes.
In preparation for removing[2] the[3] macro[4], remove all remaining
needless uses with the following script:
git grep '\buninitialized_var\b' | cut -d: -f1 | sort -u | \
xargs perl -pi -e \
's/\buninitialized_var\(([^\)]+)\)/\1/g;
s:\s*/\* (GCC be quiet|to make compiler happy) \*/$::g;'
drivers/video/fbdev/riva/riva_hw.c was manually tweaked to avoid
pathological white-space.
No outstanding warnings were found building allmodconfig with GCC 9.3.0
for x86_64, i386, arm64, arm, powerpc, powerpc64le, s390x, mips, sparc64,
alpha, and m68k.
[1] https://lore.kernel.org/lkml/20200603174714.192027-1-glider@google.com/
[2] https://lore.kernel.org/lkml/CA+55aFw+Vbj0i=1TGqCR5vQkCzWJ0QxK6CernOU6eedsudAixw@mail.gmail.com/
[3] https://lore.kernel.org/lkml/CA+55aFwgbgqhbp1fkxvRKEpzyR5J8n1vKT1VZdz9knmPuXhOeg@mail.gmail.com/
[4] https://lore.kernel.org/lkml/CA+55aFz2500WfbKXAx8s67wrm9=yVJu65TpLgN_ybYNv0VEOKA@mail.gmail.com/
Reviewed-by: Leon Romanovsky <leonro@mellanox.com> # drivers/infiniband and mlx4/mlx5
Acked-by: Jason Gunthorpe <jgg@mellanox.com> # IB
Acked-by: Kalle Valo <kvalo@codeaurora.org> # wireless drivers
Reviewed-by: Chao Yu <yuchao0@huawei.com> # erofs
Signed-off-by: Kees Cook <keescook@chromium.org>
2020-06-03 13:09:38 -07:00
|
|
|
|
|
|
|
|
|
2021-10-20 19:23:23 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-03-28 18:58:16 +08:00
|
|
|
|
2021-10-20 19:23:23 +08:00
|
|
|
|
|
|
|
|
|
2022-01-06 07:57:46 -05:00
|
|
|
|
|
|
|
|
|
2021-10-20 19:23:23 +08:00
|
|
|
|
2022-01-06 07:57:46 -05:00
|
|
|
|
2021-10-20 19:23:23 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:44 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:47 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:54 +08:00
|
|
|
|
2021-06-04 13:53:47 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:44 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-02-24 19:03:54 +08:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:44 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:28 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
virtio_ring: fix stalls for packed rings
When VIRTIO_F_RING_EVENT_IDX is negotiated, virtio devices can
use virtqueue_enable_cb_delayed_packed to reduce the number of device
interrupts. At the moment, this is the case for virtio-net when the
napi_tx module parameter is set to false.
In this case, the virtio driver selects an event offset and expects that
the device will send a notification when rolling over the event offset
in the ring. However, if this roll-over happens before the event
suppression structure update, the notification won't be sent. To address
this race condition the driver needs to check wether the device rolled
over the offset after updating the event suppression structure.
With VIRTIO_F_RING_PACKED, the virtio driver did this by reading the
flags field of the descriptor at the specified offset.
Unfortunately, checking at the event offset isn't reliable: if
descriptors are chained (e.g. when INDIRECT is off) not all descriptors
are overwritten by the device, so it's possible that the device skipped
the specific descriptor driver is checking when writing out used
descriptors. If this happens, the driver won't detect the race condition
and will incorrectly expect the device to send a notification.
For virtio-net, the result will be a TX queue stall, with the
transmission getting blocked forever.
With the packed ring, it isn't easy to find a location which is
guaranteed to change upon the roll-over, except the next device
descriptor, as described in the spec:
Writes of device and driver descriptors can generally be
reordered, but each side (driver and device) are only required to
poll (or test) a single location in memory: the next device descriptor after
the one they processed previously, in circular order.
while this might be sub-optimal, let's do exactly this for now.
Cc: stable@vger.kernel.org
Cc: Jason Wang <jasowang@redhat.com>
Fixes: f51f982682e2a ("virtio_ring: leverage event idx in packed ring")
Signed-off-by: Marvin Liu <yong.liu@intel.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2019-10-22 01:10:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:46 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-01-23 17:50:26 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:46 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2020-12-04 17:23:36 +03:00
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-12-04 17:23:00 +03:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2020-12-04 17:23:00 +03:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-05-26 11:12:11 +08:00
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-08-02 15:44:09 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
2010-04-12 16:19:07 +03:00
|
|
|
|
2010-01-18 19:15:23 +05:30
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2018-11-21 18:03:19 +08:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2008-02-04 23:49:57 -05:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-02-04 23:50:05 -05:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-03-06 18:32:29 +02:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2009-06-12 22:16:35 -06:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2012-10-16 23:56:14 +10:30
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2012-01-12 15:44:42 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2021-04-13 01:19:16 -04:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2018-11-21 18:03:26 +08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:43 +10:30
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2017-02-07 06:15:13 +02:00
|
|
|
|
|
|
|
|
|
2011-05-20 02:10:44 +03:00
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
2019-01-23 17:50:26 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
virtio_ring: shadow available ring flags & index
Improves cacheline transfer flow of available ring header.
Virtqueues are implemented as a pair of rings, one producer->consumer
avail ring and one consumer->producer used ring; preceding the
avail ring in memory are two contiguous u16 fields -- avail->flags
and avail->idx. A producer posts work by writing to avail->idx and
a consumer reads avail->idx.
The flags and idx fields only need to be written by a producer CPU
and only read by a consumer CPU; when the producer and consumer are
running on different CPUs and the virtio_ring code is structured to
only have source writes/sink reads, we can continuously transfer the
avail header cacheline between 'M' states between cores. This flow
optimizes core -> core bandwidth on certain CPUs.
(see: "Software Optimization Guide for AMD Family 15h Processors",
Section 11.6; similar language appears in the 10h guide and should
apply to CPUs w/ exclusive caches, using LLC as a transfer cache)
Unfortunately the existing virtio_ring code issued reads to the
avail->idx and read-modify-writes to avail->flags on the producer.
This change shadows the flags and index fields in producer memory;
the vring code now reads from the shadows and only ever writes to
avail->flags and avail->idx, allowing the cacheline to transfer
core -> core optimally.
In a concurrent version of vring_bench, the time required for
10,000,000 buffer checkout/returns was reduced by ~2% (average
across many runs) on an AMD Piledriver (15h) CPU:
(w/o shadowing):
Performance counter stats for './vring_bench':
5,451,082,016 L1-dcache-loads
...
2.221477739 seconds time elapsed
(w/ shadowing):
Performance counter stats for './vring_bench':
5,405,701,361 L1-dcache-loads
...
2.168405376 seconds time elapsed
The further away (in a NUMA sense) virtio producers and consumers are
from each other, the more we expect to benefit. Physical implementations
of virtio devices and implementations of virtio where the consumer polls
vring avail indexes (vhost) should also benefit.
Signed-off-by: Venkatesh Srinivas <venkateshs@google.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2015-11-10 16:21:07 -08:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2016-08-31 14:00:04 +02:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
|
|
|
|
|
virtio_ring: shadow available ring flags & index
Improves cacheline transfer flow of available ring header.
Virtqueues are implemented as a pair of rings, one producer->consumer
avail ring and one consumer->producer used ring; preceding the
avail ring in memory are two contiguous u16 fields -- avail->flags
and avail->idx. A producer posts work by writing to avail->idx and
a consumer reads avail->idx.
The flags and idx fields only need to be written by a producer CPU
and only read by a consumer CPU; when the producer and consumer are
running on different CPUs and the virtio_ring code is structured to
only have source writes/sink reads, we can continuously transfer the
avail header cacheline between 'M' states between cores. This flow
optimizes core -> core bandwidth on certain CPUs.
(see: "Software Optimization Guide for AMD Family 15h Processors",
Section 11.6; similar language appears in the 10h guide and should
apply to CPUs w/ exclusive caches, using LLC as a transfer cache)
Unfortunately the existing virtio_ring code issued reads to the
avail->idx and read-modify-writes to avail->flags on the producer.
This change shadows the flags and index fields in producer memory;
the vring code now reads from the shadows and only ever writes to
avail->flags and avail->idx, allowing the cacheline to transfer
core -> core optimally.
In a concurrent version of vring_bench, the time required for
10,000,000 buffer checkout/returns was reduced by ~2% (average
across many runs) on an AMD Piledriver (15h) CPU:
(w/o shadowing):
Performance counter stats for './vring_bench':
5,451,082,016 L1-dcache-loads
...
2.221477739 seconds time elapsed
(w/ shadowing):
Performance counter stats for './vring_bench':
5,405,701,361 L1-dcache-loads
...
2.168405376 seconds time elapsed
The further away (in a NUMA sense) virtio producers and consumers are
from each other, the more we expect to benefit. Physical implementations
of virtio devices and implementations of virtio where the consumer polls
vring avail indexes (vhost) should also benefit.
Signed-off-by: Venkatesh Srinivas <venkateshs@google.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2015-11-10 16:21:07 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:49 +08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:24 +08:00
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2020-12-04 17:23:36 +03:00
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2021-06-04 13:53:49 +08:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2021-06-04 13:53:49 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-03-06 18:32:29 +02:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-03-06 18:32:29 +02:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2017-03-06 18:32:29 +02:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
2008-02-04 23:50:05 -05:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2021-07-21 17:26:46 +03:00
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2021-07-21 17:26:46 +03:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
2020-02-24 15:26:43 -06:00
|
|
|
|
2021-06-04 13:53:50 +08:00
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2008-02-04 23:50:05 -05:00
|
|
|
|
2007-10-22 11:03:40 +10:00
|
|
|
|
2008-07-25 12:06:13 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-05-11 18:11:45 +01:00
|
|
|
|
|
|
|
|
|
2011-05-20 02:10:44 +03:00
|
|
|
|
|
|
|
|
|
2014-12-01 15:52:40 +02:00
|
|
|
|
|
|
|
|
|
2020-06-24 18:24:33 -04:00
|
|
|
|
virtio: new feature to detect IOMMU device quirk
The interaction between virtio and IOMMUs is messy.
On most systems with virtio, physical addresses match bus addresses,
and it doesn't particularly matter which one we use to program
the device.
On some systems, including Xen and any system with a physical device
that speaks virtio behind a physical IOMMU, we must program the IOMMU
for virtio DMA to work at all.
On other systems, including SPARC and PPC64, virtio-pci devices are
enumerated as though they are behind an IOMMU, but the virtio host
ignores the IOMMU, so we must either pretend that the IOMMU isn't
there or somehow map everything as the identity.
Add a feature bit to detect that quirk: VIRTIO_F_IOMMU_PLATFORM.
Any device with this feature bit set to 0 needs a quirk and has to be
passed physical addresses (as opposed to bus addresses) even though
the device is behind an IOMMU.
Note: it has to be a per-device quirk because for example, there could
be a mix of passed-through and virtual virtio devices. As another
example, some devices could be implemented by an out of process
hypervisor backend (in case of qemu vhost, or vhost-user) and so support
for an IOMMU needs to be coded up separately.
It would be cleanest to handle this in IOMMU core code, but that needs
per-device DMA ops. While we are waiting for that to be implemented, use
a work-around in virtio core.
Note: a "noiommu" feature is a quirk - add a wrapper to make
that clear.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2016-04-18 12:58:14 +03:00
|
|
|
|
2018-11-21 18:03:30 +08:00
|
|
|
|
|
|
|
|
|
2019-01-23 17:50:26 +08:00
|
|
|
|
|
|
|
|
|
2008-07-25 12:06:13 -05:00
|
|
|
|
|
|
|
|
|
2014-10-07 16:39:42 +02:00
|
|
|
|
2008-07-25 12:06:13 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-01-12 15:44:42 +10:30
|
|
|
|
|
|
|
|
|
2019-04-23 18:25:12 +08:00
|
|
|
|
2012-01-12 15:44:42 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-19 08:10:59 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2011-10-19 08:10:59 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-10-29 09:40:19 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-07-21 17:26:45 +03:00
|
|
|
|
2013-10-29 09:40:19 +10:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-04-28 11:15:08 +09:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2014-04-28 11:15:08 +09:30
|
|
|
|
|
|
|
|
|
2021-07-21 17:26:45 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-04-28 11:15:08 +09:30
|
|
|
|
2021-07-21 17:26:47 +03:00
|
|
|
|
2014-04-28 11:15:08 +09:30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2014-10-07 16:39:47 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
2014-10-07 16:39:47 +02:00
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-10-07 16:39:47 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2014-10-07 16:39:47 +02:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:25 +08:00
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:27 +08:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
2018-11-21 18:03:21 +08:00
|
|
|
|
2016-02-02 21:46:37 -08:00
|
|
|
|
|
|
|
|
|
2014-10-07 16:39:47 +02:00
|
|
|
|
2008-02-04 23:50:05 -05:00
|
|
|
|