Add few more cpumask helpers that are required by the Rust abstraction.
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Acked-by: Yury Norov [NVIDIA] <yury.norov@gmail.com>
Describe the two new vcpu flags that control NV, together with
the capabilities that advertise them.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Link: https://lore.kernel.org/r/20250514103501.2225951-18-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Since we're (almost) feature complete, let's allow userspace to
request KVM_ARM_VCPU_EL2* by bumping KVM_VCPU_MAX_FEATURES up.
We also now advertise the features to userspace with new capabilities.
It's going to be great...
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com>
Link: https://lore.kernel.org/r/20250514103501.2225951-17-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Now that we have to handle TLBI S1E2 in the core code, plumb the
sysinsn dispatch code into it, so that these instructions don't
just UNDEF anymore.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-15-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
A TLBI by VA for S1 must take effect on our pseudo-TLB for VNCR
and potentially knock the fixmap mapping. Even worse, that TLBI
must be able to work cross-vcpu.
For that, we track on a per-VM basis if any VNCR is mapped, using
an atomic counter. Whenever a TLBI S1E2 occurs and that this counter
is non-zero, we take the long road all the way back to the core code.
There, we iterate over all vcpus and check whether this particular
invalidation has any damaging effect. If it does, we nuke the pseudo
TLB and the corresponding fixmap.
Yes, this is costly.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-14-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Since we now have a way to map the guest's VNCR_EL2 on the host,
we can point the host's VNCR_EL2 to it and go full circle!
Note that we unconditionally assign the fixmap to VNCR_EL2,
irrespective of the guest's version being mapped or not. We want
to take a fault on first access, so the fixmap either contains
something guranteed to be either invalid or a guest mapping.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-13-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
During an invalidation triggered by an MMU notifier, we need to
make sure we can drop the *host* mapping that would have been
translated by the stage-2 mapping being invalidated.
For the moment, the invalidation is pretty brutal, as we nuke
the full IPA range, and therefore any VNCR_EL2 mapping.
At some point, we'll be more light-weight, and the code is able
to deal with something more targetted.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-12-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Now that we can handle faults triggered through VNCR_EL2, we need
to map the corresponding page at EL2. But where, you'll ask?
Since each CPU in the system can run a vcpu, we need a per-CPU
mapping. For that, we carve a NR_CPUS range in the fixmap, giving
us a per-CPU va at which to map the guest's VNCR's page.
The mapping occurs both on vcpu load and on the back of a fault,
both generating a request that will take care of the mapping.
That mapping will also get dropped on vcpu put.
Yes, this is a bit heavy handed, but it is simple. Eventually,
we may want to have a per-VM, per-CPU mapping, which would avoid
all the TLBI overhead.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-11-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
As VNCR_EL2.BADDR contains a VA, it is bound to trigger faults.
These faults can have multiple source:
- We haven't mapped anything on the host: we need to compute the
resulting translation, populate a TLB, and eventually map
the corresponding page
- The permissions are out of whack: we need to tell the guest about
this state of affairs
Note that the kernel doesn't support S1POE for itself yet, so
the particular case of a VNCR page mapped with no permissions
or with write-only permissions is not correctly handled yet.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-10-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Plug VNCR_EL2 in the vcpu_sysreg enum, define its RES0/RES1 bits,
and make it accessible to userspace when the VM is configured to
support FEAT_NV2.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-9-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
FEAT_NV2 introduces an interesting problem for NV, as VNCR_EL2.BADDR
is a virtual address in the EL2&0 (or EL2, but we thankfully ignore
this) translation regime.
As we need to replicate such mapping in the real EL2, it means that
we need to remember that there is such a translation, and that any
TLBI affecting EL2 can possibly affect this translation.
It also means that any invalidation driven by an MMU notifier must
be able to shoot down any such mapping.
All in all, we need a data structure that represents this mapping,
and that is extremely close to a TLB. Given that we can only use
one of those per vcpu at any given time, we only allocate one.
No effort is made to keep that structure small. If we need to
start caching multiple of them, we may want to revisit that design
point. But for now, it is kept simple so that we can reason about it.
Oh, and add a braindump of how things are supposed to work, because
I will definitely page this out at some point. Yes, pun intended.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-8-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
We currently check for HCR_EL2.NV being set to decide whether we
need to repaint PSTATE.M to say EL2 instead of EL1 on exit.
However, this isn't correct when L2 is itself a hypervisor, and
that L1 as set its own HCR_EL2.NV. That's because we "flatten"
the state and inherit parts of the guest's own setup. In that case,
we shouldn't adjust PSTATE.M, as this is really EL1 for both us
and the guest.
Instead of trying to try and work out how we ended-up with HCR_EL2.NV
being set by introspecting both the host and guest states, use
a per-CPU flag to remember the context (HYP or not), and use that
information to decide whether PSTATE needs tweaking.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-7-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
As we are about to expand out TLB invalidation capabilities to support
recursive virtualisation, move the decoding of a TLBI by range into
a helper that returns the base, the range and the ASID.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-6-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
We currently completely ignore any sort of ASID tagging during a S1
walk, as AT doesn't care about it.
However, such information is required if we are going to create
anything that looks like a TLB from this walk.
Let's capture it both the nG and ASID information while walking
the page tables.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-5-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
The address translation infrastructure is currently pretty tied to
the AT emulation.
However, we also need to features that require the use of VAs, such
as VNCR_EL2 (and maybe one of these days SPE), meaning that we need
a slightly more generic infrastructure.
Start this by introducing a new helper (__kvm_translate_va()) that
performs a S1 walk for a given translation regime, EL and PAN
settings.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-4-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
If running a NV guest on an ARMv8.4-NV capable system, let's
allocate an additional page that will be used by the hypervisor
to fulfill system register accesses.
Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com>
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-3-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Now that we're about to emulate VNCR_EL2, we need its full layout.
Add it to the sysreg file.
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Link: https://lore.kernel.org/r/20250514103501.2225951-2-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Bspec comment on PSR2_SU_STATUS:
"This register has been tied-off since DG2/ADL-P (it returns zeros only)
and it has been removed on Xe2_LPD."
v2: fix inversed logic
Bspec: 69889
Signed-off-by: Jouni Högander <jouni.hogander@intel.com>
Reviewed-by: Mika Kahola <mika.kahola@intel.com>
Link: https://lore.kernel.org/r/20250516063019.2126702-1-jouni.hogander@intel.com
The tee info reg for teev2 is the same as teev1.
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The bootloader info reg for pspv5 is the same as pspv4.
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
'struct ahash_request' has a flexible array at the end, so it must be the
last member in a struct, to avoid overwriting other struct members.
Therefore, move 'fallback_req' to the end of the 'sun8i_ce_hash_reqctx'
struct.
Fixes: 56f6d5aee8 ("crypto: sun8i-ce - support hash algorithms")
Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Current driver uses static LMTST region allocated by firmware.
Firmware allocated memory for LMTST is available in PF/VF BAR2.
Using this memory have performance impact as this is mapped as
device memory. There is another option to allocate contiguous
memory at run time and map this in LMT MAP table with the
help of AF driver. With this patch dynamic allocated memory
is used for LMTST.
Also add myself as maintainer for crypto octeontx2 driver
Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Function otx2_cptlf_set_dev_info() initializes common
fields of cptlfs data-struct. This function is called
every time a cptlf is initialized but this needs be done
once for a cptlf block. So this initialization is moved
to early device probe code to avoid redundant initialization.
Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Only add ecb to the cipher name if it isn't already ecb.
Also use memcmp instead of strncmp since these strings are all
stored in an array of length CRYPTO_MAX_ALG_NAME.
Fixes: f1c131b454 ("crypto: xts - Convert to skcipher")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Only add ecb to the cipher name if it isn't already ecb.
Also use memcmp instead of strncmp since these strings are all
stored in an array of length CRYPTO_MAX_ALG_NAME.
Fixes: 700cb3f5fe ("crypto: lrw - Convert to skcipher")
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202505151503.d8a6cf10-lkp@intel.com
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add ahash support to hmac so that drivers that can't do hmac in
hardware do not have to implement duplicate copies of hmac.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Soon hmac will support ahash. For compatibility hmac still supports
shash so it is possible for two hmac algorithms to be registered at
the same time. The shash algorithm will have the driver name
"hmac-shash(XXX-driver)". Due to a quirk in the API, there is no way
to locate the shash algorithm using the name "hmac(XXX-driver)". It
has to be addressed as either "hmac(XXX)" or "hmac-shash(XXX-driver)".
Looking it up with "hmac(XXX-driver)" will simply trigger the creation
of another instance, and on the second instantiation this will fail
with EEXIST.
Catch the error EEXIST along with ENOENT since it is expected.
If a real shash algorithm came this way, it would be addressed using
the proper name "hmac-shash(XXX-driver)".
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add support to crypto_inst_setname for having a driver template
name that differs from the algorithm template name.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add export_import and import_core so that hmac can be used as a
fallback by block-only drivers.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The shash desc needs to be zeroed after use in setkey as it is
not finalised (finalisation automatically zeroes it).
Also remove the final function as it's been superseded by finup.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Provide an option to handle the partial blocks in the ahash API.
Almost every hash algorithm has a block size and are only able
to hash partial blocks on finalisation.
As a first step disable virtual address support for algorithms
with state sizes larger than HASH_MAX_STATESIZE. This is OK as
virtual addresses are currently only used on synchronous fallbacks.
This means ahash_do_req_chain only needs to handle synchronous
fallbacks, removing the complexities of saving the request state.
Also move the saved request state into the ahash_request object
as nesting is no longer possible.
Add a scatterlist to ahash_request to store the partial block.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add export_core and import_core hooks. These are intended to be
used by algorithms which are wrappers around block-only algorithms,
but are not themselves block-only, e.g., hmac.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The core export and import functions are targeted at implementors
so move them into internal/hash.h.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Convert the Marvell CESA binding to DT schema format. The
marvell-cesa.txt and mv_cesa.txt are duplicate bindings.
The clock properties are quite varied for each platform hence the
if/then schemas. The old binding was fairly accurate with reality.
The original binding didn't mention there is 1 interrupt per CESA
engine. Based on users, there's a maximum of 2 engines.
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Convert the HiSilicon HIP06/7 Security Accelerator binding to DT schema
format. It's a straight forward conversion.
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Convert the Broadcom SPUM/SPU2 binding to DT schema format. It's a
straight forward conversion.
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Convert the Axis Crypto engine binding to DT schema format. It's a
straight forward conversion.
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Convert the AMD Cryptographic Coprocessor binding to DT schema format.
It's a straight forward conversion.
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The mediatek,eip97-crypto binding is half abandoned. The driver was
dropped in 2020 as the Mediatek platforms use InsideSecure block and
the driver for it. All the platforms except MT7623 were updated. A
patch to update it was submitted, but never addressed the review
comments.
Link: https://lore.kernel.org/all/20210303080923.16761-1-vic.wu@mediatek.com/
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The fsl,sec-v6.0 binding is the same as the fsl,sec-v4.0 binding, so add
it to the existing schema and drop the old .txt binding.
The compatibles in the .txt binding don't match the 1 user. Follow the
user for the schema.
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Enable the reporting of error counters through sysfs for QAT GEN6
devices and update the ABI documentation.
This enables the reporting of the following:
- errors_correctable - hardware correctable errors that allow the
system to recover without data loss.
- errors_nonfatal: errors that can be isolated to specific in-flight
requests.
- errors_fatal: errors that cannot be contained to a request,
requiring a Function Level Reset (FLR) upon occurrence.
Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Enable the reporting and handling of errors for QAT GEN6 devices.
Errors are categorized as correctable, non-fatal, or fatal. Error
handling involves reading the error source registers (ERRSOU0 to ERRSOU3)
to determine the source of the error and then decoding the actual source
reading specific registers.
The action taken depends on the error type:
- Correctable and Non-Fatal errors. These error are logged, cleared and
the corresponding counter is incremented.
- Fatal errors. These errors are logged, cleared and a Function Level
Reset (FLR) is scheduled.
This reports and handles the following errors:
- Accelerator engine (AE) correctable errors
- Accelerator engine (AE) uncorrectable errors
- Chassis push-pull (CPP) errors
- Host interface (HI) parity errors
- Internal memory parity errors
- Receive interface (RI) errors
- Transmit interface (TI) errors
- Interface for system-on-chip (SoC) fabric (IOSF) primary command
parity errors
- Shared RAM and slice module (SSM) errors
Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If accept(2) is called on socket type algif_hash with
MSG_MORE flag set and crypto_ahash_import fails,
sk2 is freed. However, it is also freed in af_alg_release,
leading to slab-use-after-free error.
Fixes: fe869cdb89 ("crypto: algif_hash - User-space interface for hash operations")
Cc: <stable@vger.kernel.org>
Signed-off-by: Ivan Pravdin <ipravdin.official@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
