On systems with NVIDIA Olympus cores, a Device-nGnR* load can be
observed by a peripheral before an older, non-overlapping Device-nGnR*
store to the same peripheral. This breaks the program-order guarantee
that software expects for Device-nGnR* accesses and can leave a
peripheral in an incorrect state.
The erratum can occur only when all of the following apply:
- A PE executes a Device-nGnR* store followed by a younger
Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program
order, and there are no intervening Device-nGnR* loads.
- There is no DSB or full DMB between the store and the load.
- Specific microarchitectural and timing conditions occur.
Insert a DMB OSH immediately before each raw MMIO load on affected CPUs.
As a full barrier, DMB OSH orders the older Device store before the
younger Device load and prevents the erroneous observation.
Add the barrier directly to the __raw_read*() helpers, independently of
the existing device-load-acquire alternative. On affected CPUs this adds
one DMB OSH per raw MMIO load, including each load used by
memcpy_fromio(). On unaffected CPUs the alternative remains a NOP.
Co-developed-by: Vikram Sethi <vsethi@nvidia.com>
Signed-off-by: Vikram Sethi <vsethi@nvidia.com>
Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
Link: https://lore.kernel.org/all/akPQ8F3OgER621UP@willie-the-truck/
Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Change to using KVM_X86_OP() instead of KVM_X86_OP_OPTIONAL() for the
.pi_update_irte() hook in kvm-x86-ops.h since now both VMX and SVM have
implemented it.
For the Fixes tag:
This hook was introduced for VMX posted-interrupt support. SVM later
added its implementation, but at this point KVM_X86_OP* had not been
introduced yet.
Initially KVM introduced KVM_X86_OP_NULL (and KVM_X86_OP) and used it
for this hook. But this was correct, because the use of KVM_X86_OP_NULL
was "to mark calls that do not follow the [svm|vmx]_func_name naming
convention" and the VMX one was named pi_update_irte(), i.e., did not
follow the convention. See commit 9af5471bdb ("KVM: x86: introduce
definitions to support static calls for kvm_x86_ops").
KVM later removed KVM_X86_OP_NULL (due to "the naming convention is not
in use anymore"), and added KVM_X86_OP_OPTIONAL for the hooks that can
be NULL pointer. It used KVM_X86_OP_OPTIONAL for this hook, but should
use KVM_X86_OP instead. See commit e4fc23bad8 ("KVM: x86: remove
KVM_X86_OP_NULL and mark optional kvm_x86_ops").
Note this hook was named .update_pi_irte() when it was introduced, but
got renamed to .pi_update_irte() at some point between the above two
commits.
Fixes: e4fc23bad8 ("KVM: x86: remove KVM_X86_OP_NULL and mark optional kvm_x86_ops")
Signed-off-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260715090505.601174-1-kai.huang@intel.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
The driver explicitly sets the .driver_data member of struct
platform_device_id to zero without relying on that value. Drop this
unused assignment.
While touching this array use a named initializer for .name.
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
Signed-off-by: Will Deacon <will@kernel.org>
sctp_auth_chunk_verify() returns true unconditionally when
chunk->auth_chunk is NULL, silently skipping authentication.
This is incorrect when:
1. skb_clone() failed in the BH receive path, leaving auth_chunk
NULL. In sctp_endpoint_bh_rcv() asoc is NULL for new
connections, so the early sctp_auth_recv_cid() check cannot
catch this.
2. No AUTH chunk precedes COOKIE-ECHO, so skb_clone() is never
called and auth_chunk remains NULL.
Fix by checking sctp_auth_recv_cid() when auth_chunk is NULL:
if authentication is required, return false to drop the chunk;
otherwise continue normally.
Fixes: bbd0d59809 ("[SCTP]: Implement the receive and verification of AUTH chunk")
Signed-off-by: Qing Luo <luoqing@kylinos.cn>
Acked-by: Xin Long <lucien.xin@gmail.com>
Link: https://patch.msgid.link/20260721015532.120157-2-l1138897701@163.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
When replacing the source list of an MDB remote entry, all existing
sources are first marked for deletion and vxlan_mdb_remote_srcs_add()
is then called to add the new source list. Sources present in the new
list have their deletion mark cleared, and any sources left marked
afterwards are removed.
If vxlan_mdb_remote_srcs_add() fails partway through, its error path
deletes all entries on the remote's source list. That rollback is only
correct for its other caller, vxlan_mdb_remote_add(), where the remote
was just allocated and the list contains solely entries added during
the call. On the replace path the list also holds pre-existing sources,
so a failed replace tears them down together with their (S, G)
forwarding entries instead of leaving the entry unchanged.
This is reachable from an existing (*, G) remote. An EXCLUDE filter
that loses sources starts forwarding traffic that should be blocked,
while an INCLUDE filter that loses sources drops traffic that should be
forwarded.
Mark entries created during the current pass with a new
VXLAN_SGRP_F_NEW flag. On failure, delete only those entries and clear
the deletion mark on the pre-existing ones, so a failed replace leaves
the source list untouched. Retain the flag until the whole operation
succeeds and then clear it. Also stop vxlan_mdb_remote_src_add() from
deleting a pre-existing entry it only looked up when adding that
entry's forwarding entry fails.
Fixes: a3a48de5ea ("vxlan: mdb: Add MDB control path support")
Cc: stable@vger.kernel.org
Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Reviewed-by: Antoine Tenart <atenart@kernel.org>
Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org>
Link: https://patch.msgid.link/20260720160428.249356-1-jamestiotio@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Joe Damato says:
====================
Extend packet socket selftests
I was looking around the packet socket code and noticed there were a few
features that could be tested by extending the existing packet socket
tests.
I extended the test to test stats, drops, and auxdata and re-ran the tests.
The existing and new tests passed.
====================
Link: https://patch.msgid.link/20260720122714.759175-1-joe@dama.to
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Extend the packet socket selftest, adding a recvmsg path, to test
PACKET_AUXDATA. Check basic attributes of tpacket_auxdata.
Signed-off-by: Joe Damato <joe@dama.to>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Link: https://patch.msgid.link/20260720122714.759175-4-joe@dama.to
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Extend psock_snd to test drops by setting a tiny receive buffer and
sending a large burst of packets.
Signed-off-by: Joe Damato <joe@dama.to>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Link: https://patch.msgid.link/20260720122714.759175-3-joe@dama.to
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Update the existing packet socket test to include a test for the sockopt
PACKET_STATISTICS.
Signed-off-by: Joe Damato <joe@dama.to>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Since commit 1b9707e6f1 ("net: stmmac: enable RPS and RBU
interrupts"), suspending causes an interrupt storm from the RPS
interrupt.
Fix this by adding a deinit_chan() op to stmmac_dma_ops, which
masks all default dma channel interrupts. This is called from
stmmac_stop_all_dma(), so interrupts don't trigger while suspending.
Fixes: 1b9707e6f1 ("net: stmmac: enable RPS and RBU interrupts")
Suggested-by: Andrew Lunn <andrew@lunn.ch>
Suggested-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: Luis Lang <luis.la@mail.de>
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Tested-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Reviewed-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Link: https://patch.msgid.link/20260720111534.163416-1-luis.la@mail.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
arm64 implements CPPC FFH feedback-counter reads using AMU counters.
Because those counters must be sampled on the target CPU, reading the
delivered and reference counters separately widens the observation window
between them.
Implement the paired FFH feedback-counter read hook on arm64 and sample
both AMU counters together before decoding the requested CPC register
values.
Also factor the FFH bitfield extraction logic into a helper and reuse
it from the existing single-counter FFH read path.
Tested-by: Sumit Gupta <sumitg@nvidia.com>
Reviewed-by: Sumit Gupta <sumitg@nvidia.com>
Tested-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Reviewed-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Signed-off-by: Pengjie Zhang <zhangpengjie2@huawei.com>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Reviewed-by: Jeremy Linton <jeremy.linton@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
cppc_get_perf_ctrs() reads the delivered and reference performance
counters one at a time.
Allow architectures to provide both FFH feedback counters in one
operation when that either narrows the sampling window or avoids extra
cross-CPU reads. Add a small FFH-specific hook for that case and fall
back to the existing per-register reads when unsupported.
Tested-by: Sumit Gupta <sumitg@nvidia.com>
Reviewed-by: Sumit Gupta <sumitg@nvidia.com>
Tested-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Reviewed-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Signed-off-by: Pengjie Zhang <zhangpengjie2@huawei.com>
Acked-by: Rafael J. Wysocki (Intel) <rafael@kernel.org>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Reviewed-by: Jeremy Linton <jeremy.linton@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
After executing cpufreq sptest, the system governor
will be overwritten with the governor switched
during the test. Restore this setting to maintain
consistency before and after the test.
Signed-off-by: Yiwei Lin <s921975628@gmail.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Link: https://patch.msgid.link/20260707163647.6646-1-s921975628@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
The cpufreq selftests are always executed through main.sh, which verifies
that the test is run as root before dispatching any test case.
Therefore, invoking sudo inside quick_shuffle() is redundant and may cause
failures in environments where sudo is unavailable.
Signed-off-by: Jinseok Kim <always.starving0@gmail.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Link: https://patch.msgid.link/20260706143857.3306-2-always.starving0@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
switch_show_governor() assigns the current governor and frequency
to local variables before switching governors.
However, these variables are never referenced afterwards. The function
does not restore the previous governor or use the saved frequency, as
backup_governor() and restore_governor() already handle state preservation
elsewhere.
Signed-off-by: Jinseok Kim <always.starving0@gmail.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Link: https://patch.msgid.link/20260706143857.3306-1-always.starving0@gmail.com
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Before converting to the phylink interface, the init function would have
set a non-reserved I/F mode in the maccfg2 register. After converting to
phylink, 0 is written as mode, which is a reserved value (although it's
the hardware default). Without a valid mode, a SGMII link is never
established between the MAC and the PHY and thus .link_up() is never
called which could set the correct mode according to the actual speed.
Fix it by setting the maximum speed of the phy_interface_t in use in
.mac_config() - just like the driver did before the phylink conversion.
Fixes: 5d93cfcf73 ("net: dpaa: Convert to phylink")
Suggested-by: Sean Anderson <sean.anderson@linux.dev>
Signed-off-by: Michael Walle <mwalle@kernel.org>
Reviewed-by: Sean Anderson <sean.anderson@linux.dev>
Reviewed-by: Sean Anderson <sean.anderson@linux.dev>
Link: https://patch.msgid.link/20260717132401.2653252-1-mwalle@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Return the bitmap size, rather than size + 1, when
bitmap_find_next_zero_area_off() cannot find a suitable area. This
matches the conventional find_bit() failure sentinel and still lets
callers detect failure with an out-of-range check.
Document the public failure contract as a value greater than or equal
to the bitmap size, without requiring callers to depend on the exact
sentinel.
Signed-off-by: Yury Norov <ynorov@nvidia.com>
bitmap_find_next_zero_area() uses an out-of-range return value to
indicate failure. Check for values greater than or equal to the bitmap
size so the caller does not depend on the exact failure sentinel.
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
bitmap_find_next_zero_area() uses an out-of-range return value to
indicate failure. Accept only offsets strictly below the bitmap size so
the callers do not depend on the exact failure sentinel.
Signed-off-by: Yury Norov <ynorov@nvidia.com>
bitmap_find_next_zero_area() uses an out-of-range return value to
indicate failure. Check for values greater than or equal to the bitmap
size so the caller does not depend on the exact failure sentinel.
Acked-by: Madhavan Srinivasan <maddy@linux.ibm.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
bitmap_find_next_zero_area() uses an out-of-range return value to
indicate failure. Check for values greater than or equal to the bitmap
size so the caller does not depend on the exact failure sentinel.
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
__nodemask_pr_numnodes() currently returns MAX_NUMNODES as the field
width for '%*pb[l]' nodemask printing. MAX_NUMNODES is a compile-time
upper bound and can be much larger than the runtime node id range,
resulting in excessive zero padding in bitmap-form output.
For example, /proc/<pid>/status prints Mems_allowed with '%*pb' using
the nodemask_pr_args() helper. On systems built with MAX_NUMNODES=1024
but booted with a much smaller possible-node range, this produces:
Mems_allowed: 00000000,00000000,...,00000003
Switch to nr_node_ids, matching the behavior of cpumask_pr_args() which
uses nr_cpu_ids. This reduces the output width from MAX_NUMNODES bits
to the runtime node id range:
Mems_allowed: 3
Visible impact on in-tree users:
- Bitmap format ('%*pb') users:
* /proc/<pid>/status Mems_allowed (format changes as shown above)
- List format ('%*pbl') users, output is unchanged, as list formatter
only prints set bit ranges:
* /sys/devices/system/node/{possible,online,has_normal_memory, ...}
* NVMe multipath sysfs numa_nodes
* memory tier sysfs nodelist
* cpuset cgroup mems and effective_mems files
* /proc/<pid>/status Mems_allowed_list
* mempolicy strings in /proc/<pid>/numa_maps
* SLUB debugfs output
* Kernel log messages printing nodemasks
Move nr_node_ids and nr_online_nodes declarations earlier in the file
to allow __nodemask_pr_numnodes() to use nr_node_ids.
Cc: Yury Norov <yury.norov@gmail.com>
Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm@kvack.org
Signed-off-by: Li RongQing <lirongqing@baidu.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Erhard reports failure of bitmap tests on powerpc:
test_bitmap: loaded.
test_bitmap: [lib/test_bitmap.c:397] bitmaps contents differ: expected "1,3-4,9", got "1,3-4,9,65-71,73-79,81-87,89-95,97-99"
test_bitmap: parselist('0-2047:128/256'): 912
test_bitmap: scnprintf("%*pbl", '0-32767'): 5977
test_bitmap: test_bitmap_read_perf: 1191082
test_bitmap: test_bitmap_write_perf: 1270153
test_bitmap: failed 1 out of 208655 tests
It happens mainly when CONFIG_INIT_STACK_ALL_PATTERN is set.
Commit 6b5a4b6873 ("bitmap: Add test for out-of-boundary
modifications for scatter & gather") extended the test to
out-of-boundary bits, but those bits were left uninitialised.
Properly initialise the entire result bitmap before the test.
[Yury: minor commit message tweaks]
Reported-by: Erhard Furtner <erhard_f@mailbox.org>
Closes: https://lore.kernel.org/all/ca3547ae-8b79-43a2-a758-23ec980bfd9a@mailbox.org
Fixes: 6b5a4b6873 ("bitmap: Add test for out-of-boundary modifications for scatter & gather")
Signed-off-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
These callbacks are sysfs show paths. Use sysfs_emit() and
cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Link: https://lore.kernel.org/all/akANJ-AT7nHpRMq-@yury/
Acked-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
These callbacks are sysfs show paths.
Use sysfs_emit() and cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
sdma_get_cpu_to_sde_map() is used by a sysfs show callback.
Use sysfs_emit() and cpumask_pr_args() to emit the mask.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
cpumask_show() is a sysfs show callback. Use sysfs_emit() and
cpumask_pr_args() to emit the mask.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
cpumask_show() is a sysfs show callback. Use sysfs_emit() and
cpumask_pr_args() to emit the mask.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
These callbacks are sysfs show paths. Use sysfs_emit() and
cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
show_cpus_attr() is a sysfs show callback. Use sysfs_emit() and
cpumask_pr_args() to emit the mask.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
These callbacks are sysfs show paths.
Use sysfs_emit() and cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
These callbacks are sysfs show paths.
Use sysfs_emit() and cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
These callbacks are sysfs show paths.
Use sysfs_emit() and cpumask_pr_args() to emit the masks.
This prepares for removing cpumap_print_to_pagebuf().
Signed-off-by: Yury Norov <ynorov@nvidia.com>
The *_bit_le functions use a signed integer for the bit number.
However, the *_bit functions can use an unsigned long. This causes
problems if there is a large bitmap and a bit number > 0x80000000 is
passed in. Since that is a negative int, it will get sign extended to a
long when getting passed to the *_bit function, turning it into a huge
bit number. This usually ends up with the memory address wrapping around
and the function accessing memory before the start of the bitmap.
Avoid this by making the *_bit_le functions take an unsigned long.
This can be triggered by faking an almost 4TB dm-mirror device, which
uses bitmaps to track the mirror regions:
$ dmsetup create bigzero --table '0 8589934590 zero'
$ dmsetup create mymirror --table '0 8589934590 mirror core 2 2 nosync 2 /dev/mapper/bigzero 0 /dev/mapper/bigzero 0'
This will access memory before the start of the sync_bits bitmap, and
likely hit the guard page of the previously allocated clean_bits bitmap,
causing a kernel panic with the old code.
I looked and didn't see any crazy code using the signed int to
intentionally try and access bits before some address within the bitmap.
Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Finding a contiguous free region in a highly fragmented
bitmap is not easy and may require many repeated attempts.
Therefore, find_next_bit(map, end, index) is not the optimal choice.
This is because there may be multiple scattered free regions
within the range [index, end) and none of them will meet the length
requirement of @nr.
Instead, it's sufficient to directly find the last bit within
the range [index, end), thus reducing unnecessary repeated calls.
An example of a bitmap:
Bits 0-3: cleared(4 bits)
Bits 4-5: set (2 bits)
Bits 6-8: cleared(4 bits)
Bits 9-10: set (2 bits)
Bits 11-20: cleared(10 bits)
The goal is to find a 10-bit free region.
The old code logic is as follows:
find_next_zero_bit(start = 0, find bit 0) -> find_next_bit(find bit 4) ->
next loop ->
find_next_zero_bit(start = 5, find bit 6) -> find_next_bit(find bit 9) ->
next loop ->
find_next_zero_bit(start = 10, find bit 11) -> success
The new code logic is as follows:
find_next_zero_bit(start = 0, find bit 0) -> find_last_bit(find bit 9) ->
next loop ->
find_next_zero_bit(start = 10, find bit 11) -> success
Performance test results on my hardware(use lib/find_bit_benchmark.c):
before after change p-value
dense 1211 688 -43.2% 8.3e-11
sparse 13.3 13.4 0.8% 0.27
Yury:
The less micro-benchmark kselftest/dmabuf-heaps/dmabuf-heap gives
even better numbers:
Metric Before After Change
Trace span 194.0 ms 87.1 ms -55.1%
Total CMA alloc time 48.46 ms 16.11 ms -66.8%
Avg alloc latency 184.94 us 61.49 us -66.8%
Median alloc latency 73.72 us 20.59 us -72.1%
p90 alloc latency 329.76 us 55.63 us -83.1%
p99 alloc latency 1866.76 us 859.83 us -53.9%
Max alloc latency 4821.91 us 2324.41 us -51.8%
By request size:
Request Before Avg After Avg Change
1 page 79.68 us 34.47 us -56.7%
256 pages 285.50 us 87.30 us -69.4%
Co-developed-by: Yury Norov <yury.norov@gmail.com>
Signed-off-by: Yury Norov <yury.norov@gmail.com>
Signed-off-by: sunyi <279644543@qq.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Switch to pr_info("... %pbl"), and drop the temporary buffer allocation.
This prepares for removing cpumap_print_to_pagebuf().
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
While the GCC and Clang compilers already define __ASSEMBLER__
automatically when compiling assembly code, __ASSEMBLY__ is a
macro that only gets defined by the Makefiles in the kernel.
This can be very confusing when switching between userspace
and kernelspace coding, or when dealing with uapi headers that
rather should use __ASSEMBLER__ instead. So let's standardize now
on the __ASSEMBLER__ macro that is provided by the compilers.
This is a completely mechanical patch (done with a simple "sed -i"
statement).
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
scmi_driver_register() requests protocol devices before registering the
driver. If driver_register() fails, those requests remain in the global
IDR and retain pointers to the module's ID table. Once the failed module
load releases that storage, later request matching or SCMI device creation
can dereference the stale pointers.
Unrequest the complete protocol table before returning the registration
failure. At this point table registration succeeded, so every entry is
owned by the current registration attempt.
Fixes: d3cd7c525f ("firmware: arm_scmi: Refactor protocol device creation")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://patch.msgid.link/20260722173521.2184378-2-sudeep.holla@kernel.org
Signed-off-by: Sudeep Holla <sudeep.holla@kernel.org>
scmi_protocol_table_register() can leave earlier requests registered when
a later entry in the same ID table fails. Each request retains a pointer
to the driver's ID table, so a failed module load can leave a dangling
pointer after the module storage is released.
Unrequest only the successfully registered prefix, in reverse order,
before returning the failure. Leave the failed entry and the remaining
entries untouched because matching requests can be owned by another
driver.
Fixes: 2858f6e5f0 ("firmware: arm_scmi: Add multiple protocols registration support")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://patch.msgid.link/20260722173521.2184378-1-sudeep.holla@kernel.org
Signed-off-by: Sudeep Holla <sudeep.holla@kernel.org>
Implement the "ccm(aes)" crypto_aead algorithm using the corresponding
library functions.
Among other benefits, this allows the architecture-optimized AES-CCM
code to be migrated into the library while still leaving it accessible
via crypto_aead, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of this algorithm haven't yet
been migrated into the library. It will be boosted once that happens.
Link: https://patch.msgid.link/20260715221153.246410-14-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Implement the "gcm(aes)" and "rfc4106(gcm(aes))" crypto_aead algorithms
using the corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-GCM
code to be migrated into the library while still leaving it accessible
via crypto_aead, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of these algorithms haven't yet
been migrated into the library. It will be boosted once that happens.
Link: https://patch.msgid.link/20260715221153.246410-13-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Implement the "xts(aes)" crypto_skcipher algorithm using the
corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-XTS
code to be migrated into the library while still leaving it accessible
via crypto_skcipher, eliminating lots of boilerplate code.
Fast paths similar to what x86_64 uses (to eliminate the scatterlist
walking overhead) are included. So we'll get that optimization for all
architectures.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of this algorithm haven't yet
been migrated into the library. It will be boosted once that happens.
Link: https://patch.msgid.link/20260715221153.246410-12-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Implement the "ctr(aes)" and "xctr(aes)" crypto_skcipher algorithms
using the corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-CTR and
AES-XCTR code to be migrated into the library while still leaving it
accessible via crypto_skcipher, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of these algorithms haven't yet
been migrated into the library. It will be boosted once that happens.
Link: https://patch.msgid.link/20260715221153.246410-11-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Implement the "cbc(aes)" and "cts(cbc(aes))" crypto_skcipher algorithms
using the corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-CBC and
AES-CBC-CTS code to be migrated into the library while still leaving it
accessible via crypto_skcipher, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of these algorithms haven't yet
been migrated into the library. It will be boosted once that happens.
Link: https://patch.msgid.link/20260715221153.246410-10-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Implement the "ecb(aes)" crypto_skcipher algorithm using the
corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-ECB
code to be migrated into the library while still leaving it accessible
via crypto_skcipher, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of this algorithm haven't yet
been migrated into the library. It will be boosted once that happens.
Link: https://patch.msgid.link/20260715221153.246410-9-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Add support for AES-CCM to the crypto library.
This will be used to provide a streamlined implementation of the
"ccm(aes)" crypto_aead algorithm. Most users of "ccm(aes)" will also be
able to switch to the library, which as usual will be faster and
simpler, e.g.:
- fs/smb/client/
- fs/smb/server/
- net/mac80211/
- net/mac802154/
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As in the AES-GCM API, incremental operation is supported. It has to be
used carefully, especially when decrypting, but it makes the API general
enough to work well for all users.
The AES-CCM library code calls aes_cbcmac_blocks() directly, bypassing
the higher-level aes_cbcmac_init(), aes_cbcmac_update(), and
aes_cbcmac_final(). The latter set of functions is useful only for
AES-CCM, so they don't make sense to keep around and will be removed
once the "ccm(aes)" crypto_aead starts using the AES-CCM library.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Add support for AES-GCM to the crypto library.
This will be used to provide streamlined implementations of the
"gcm(aes)" and "rfc4106(gcm(aes))" crypto_aead algorithms. Most users
of these will also be able to switch to the library, which as usual will
be faster and simpler, e.g.:
- drivers/net/macsec.c
- fs/smb/client/
- fs/smb/server/
- net/ceph/messenger_v2.c
- net/mac80211/ (for both GMAC and GCMP)
- net/tipc/crypto.c
- security/keys/trusted-keys/trusted_dcp.c
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As usual, the architecture-optimized AES-GCM code will be migrated into
the library as well (using the hooks provided in this commit as well as
the GHASH ones), eliminating lots of repetitive boilerplate code.
Incremental en/decryption is supported. Incremental operation is a bit
controversial in AEAD APIs because users have to be careful not to
consume any decrypted data that hasn't been authenticated yet. But I do
think it's the right choice here. It's not fundamentally different from
the existing incremental MAC APIs, and it's the only approach that's
general enough to work well for all users in the kernel:
- An array of virtually-addressed buffers (like that used by
BoringSSL's EVP_AEAD_CTX_sealv() and EVP_AEAD_CTX_openv()) doesn't
work in the kernel in general, since in some cases the data for a
single AES-GCM message is contained in a large number of highmem
pages that each need to be mapped into memory individually. That
can be done efficiently only by using CPU-local mappings, but there
is a limited number of those.
Ceph messenger v2 is a great example, as it can send or receive up
to 32 MiB in a single AES-GCM message. And it needs the
en/decrypted data to go into a (potentially large) number of bvecs
provided by a custom iterator, as well as into four
virtually-addressed buffers, two of which can be large buffers in
the vmalloc region.
Even just allocating an array big enough to store all the pointers
can be problematic in the kernel. There are cases in which
decryption runs in GFP_NOIO context or even in softirq context,
where memory allocations are not as reliable as they normally are.
- Meanwhile, 'struct scatterlist' (the choice of crypto_aead) has
turned out to be really inconvenient for anyone who *does* just have
virtually-addressed buffers. This is especially true if they can be
in the vmalloc region, including the stack, as in that case the
conversion to a scatterlist has to be done page-by-page.
And even for users who have all of their data in bare 'struct page',
none of them actually use 'struct scatterlist' as their native data
structure anyway. They actually use skbs, bvecs, or other formats.
- iov_iter is attractive, but ultimately not general enough either
(considering the Ceph case for example), but also too general in
some ways (like having support for userspace addresses). Additional
iter types like ITER_SKB would help a bit, but bloating iov_iter
with more types would reduce performance elsewhere in the kernel.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-7-ebiggers@kernel.org
Link: https://patch.msgid.link/20260722021730.16897-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
