| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Add a test case which reproduces the bug fixed by commit 24f07e8c07a0
("lib/crypto: arm64: Fix lost Poly1305 carry when resuming NEON state").
It is written in a fairly general way, such that it should also be able
to reproduce similar bugs in other implementations.
Update the comment on test_poly1305_allones_keys_and_message() to more
accurately describe what it does and differentiate it from the new test.
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Link: https://patch.msgid.link/20261009125354.315476-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
When poly1305_blocks_neon() resumes from a base 2^26 accumulator and the
next update contains an odd number of full 16-byte blocks, it processes
one leading block through poly1305_mult() so that the remaining NEON
work has an even block count. That requires converting the accumulator
to the base 2^64 form used by poly1305_mult(). The final ADC of that
conversion stores the carry into d2, but the following accumulation and
poly1305_mult() use h2. If the conversion carries across bit 128, the
carry is dropped and the emitted tag is wrong.
This was introduced by Cryptogams commit 03dc4adf91c8
("arm/poly1305-armv*.pl: optimize branches."), which removed the
reduction that consumed d2 shortly before Linux imported the code.
OpenSSL's copy did not take that change.
The carry is possible because the NEON code stores the accumulator [h0,
h1, h2, h3, h4] in a lazily reduced, redundant base 2^26 form: its final
carry chain leaves h0, h2, h3 < 2^26, but h1 and h4 may slightly exceed
2^26 - 1. For example, the reachable state
h0 = 4
h1 = 2^26
h2 = 2^26 - 1
h3 = 2^26 - 1
h4 = 2^24 - 1
represents 4 + 2^26*2^26 + (2^26-1)*2^52 + (2^26-1)*2^78 +
(2^24-1)*2^104 = 2^128 + 4, despite the bit representing the 2^128 term
(bit 24 of h4) not being set; the extra comes from the h1 limb being
above the normal base 2^26 range. When this value is converted into the
non-redundant base 2^64 form, a carry into the new 2^128 bit (in bit 0
of h2) is necessary to preserve the value.
Given the above bounds, a carry into bit 128 happens exactly when
h1 >= 2^26, h2 = h3 = 2^26 - 1, and h4 mod 2^24 = 2^24 - 1.
Store the carry into h2, matching the other base 2^26 to base 2^64
conversions in poly1305_blocks() and poly1305_emit().
This bug causes an incorrect tag to be emitted. However, this is
unlikely to happen in practice where Poly1305 is used with a random key,
since the random key tends to distribute the accumulator values
throughout their valid range. There isn't an obvious way for an
attacker without knowledge of the r_key to cause [h0, h1, h2, h3, h4] to
land on the precise values where this bug occurs.
[EB: Rewrote the commit message to more clearly describe the bug and
its impact, and simplified explanation about why there's a carry bit.]
Fixes: f569ca164751 ("crypto: arm64/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementation")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://patch.msgid.link/20261009125354.315476-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Use memzero_explicit() instead of a plain struct assignment to guarantee
zeroization of the 'struct poly1305_desc_ctx'.
Note that dead store elimination was only theoretically possible with
link-time optimization here. But it's still worth fixing.
Fixes: a1d93064094c ("crypto: poly1305 - expose init/update/final library interface")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-c3a6bd76-f6ab-4203-892a-cd06f7aa0c34
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261006083432.16328-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
By using __cleanup(aes_cmac_zeroize_key) for clearing the key data, we
can save one line of code here.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260807125845.1477067-6-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the AES-GCM library API.
It consists of:
- All the shared test cases from aead-test-template.h. These include
extensive consistency tests, a "Monte-Carlo test", and a benchmark.
- Tests against hardcoded AES-GCM test vectors from external sources.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the AES-CCM library API.
It consists of:
- All the shared test cases from aead-test-template.h. These include
extensive consistency tests, a "Monte-Carlo test", and a benchmark.
- Tests against hardcoded AES-CCM test vectors from external sources.
- Tests for CCM-specific message length validation.
To generate the expected aes_ccm_monte_carlo_checksum[] value, add a
script gen-aead-testvecs.py which computes it using python-cryptography.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add aead-test-template.h which contains shared test logic for AEAD
algorithms. It's similar to the existing hash-test-template.h, which
contains shared test logic for hash algorithms.
The initial use cases for this will be AES-CCM and AES-GCM.
It may be used by the existing ChaCha20Poly1305 test suite in the future
as well, though the ChaCha20Poly1305 API currently lacks some of the
functionality that the "template" expects to be present.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make the test cases more self-contained by replacing the test suite
scoped guarded buffer ('test_buf') with buffers allocated by each test
case using the helper functions in test-utils.h.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Move hash-test-template's test RNG code and the buffer allocation code
from various test suites into a header test-utils.h. This allows them
to be shared by other test suites.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upcoming changes will wire up architecture-optimized implementations of
GCM and CCM. FIPS labs can consider such designs to meet the threshold
for separate self-tests to be needed.
Therefore, add FIPS self-tests for encryption and decryption in these
modes.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upcoming changes will wire up architecture-optimized implementations of
ECB, CBC, CBC-CTS, CTR, and XTS. FIPS labs can consider such designs to
meet the threshold for separate self-tests to be needed.
The inverse direction of the block cipher also needs to be exercised,
which the existing CMAC self-test doesn't do.
Therefore, add FIPS self-tests for encryption and decryption in these
modes as well as the "bare" AES.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In preparation for adding FIPS self-tests for AES encryption modes,
split fips.h into separate files for the AES and SHA test vectors.
They are still generated by the same script, but this keeps things a bit
more organized.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The old AES-GCM library code is no longer used, so remove it.
Tested-by: Nikunj A Dadhania <nikunj@amd.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260722025338.33354-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-CCM to the crypto library.
This will be used to provide a streamlined implementation of the
"ccm(aes)" crypto_aead algorithm. Most users of "ccm(aes)" will also be
able to switch to the library, which as usual will be faster and
simpler, e.g.:
- fs/smb/client/
- fs/smb/server/
- net/mac80211/
- net/mac802154/
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As in the AES-GCM API, incremental operation is supported. It has to be
used carefully, especially when decrypting, but it makes the API general
enough to work well for all users.
The AES-CCM library code calls aes_cbcmac_blocks() directly, bypassing
the higher-level aes_cbcmac_init(), aes_cbcmac_update(), and
aes_cbcmac_final(). The latter set of functions is useful only for
AES-CCM, so they don't make sense to keep around and will be removed
once the "ccm(aes)" crypto_aead starts using the AES-CCM library.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-GCM to the crypto library.
This will be used to provide streamlined implementations of the
"gcm(aes)" and "rfc4106(gcm(aes))" crypto_aead algorithms. Most users
of these will also be able to switch to the library, which as usual will
be faster and simpler, e.g.:
- drivers/net/macsec.c
- fs/smb/client/
- fs/smb/server/
- net/ceph/messenger_v2.c
- net/mac80211/ (for both GMAC and GCMP)
- net/tipc/crypto.c
- security/keys/trusted-keys/trusted_dcp.c
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As usual, the architecture-optimized AES-GCM code will be migrated into
the library as well (using the hooks provided in this commit as well as
the GHASH ones), eliminating lots of repetitive boilerplate code.
Incremental en/decryption is supported. Incremental operation is a bit
controversial in AEAD APIs because users have to be careful not to
consume any decrypted data that hasn't been authenticated yet. But I do
think it's the right choice here. It's not fundamentally different from
the existing incremental MAC APIs, and it's the only approach that's
general enough to work well for all users in the kernel:
- An array of virtually-addressed buffers (like that used by
BoringSSL's EVP_AEAD_CTX_sealv() and EVP_AEAD_CTX_openv()) doesn't
work in the kernel in general, since in some cases the data for a
single AES-GCM message is contained in a large number of highmem
pages that each need to be mapped into memory individually. That
can be done efficiently only by using CPU-local mappings, but there
is a limited number of those.
Ceph messenger v2 is a great example, as it can send or receive up
to 32 MiB in a single AES-GCM message. And it needs the
en/decrypted data to go into a (potentially large) number of bvecs
provided by a custom iterator, as well as into four
virtually-addressed buffers, two of which can be large buffers in
the vmalloc region.
Even just allocating an array big enough to store all the pointers
can be problematic in the kernel. There are cases in which
decryption runs in GFP_NOIO context or even in softirq context,
where memory allocations are not as reliable as they normally are.
- Meanwhile, 'struct scatterlist' (the choice of crypto_aead) has
turned out to be really inconvenient for anyone who *does* just have
virtually-addressed buffers. This is especially true if they can be
in the vmalloc region, including the stack, as in that case the
conversion to a scatterlist has to be done page-by-page.
And even for users who have all of their data in bare 'struct page',
none of them actually use 'struct scatterlist' as their native data
structure anyway. They actually use skbs, bvecs, or other formats.
- iov_iter is attractive, but ultimately not general enough either
(considering the Ceph case for example), but also too general in
some ways (like having support for userspace addresses). Additional
iter types like ITER_SKB would help a bit, but bloating iov_iter
with more types would reduce performance elsewhere in the kernel.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-7-ebiggers@kernel.org
Link: https://patch.msgid.link/20260722021730.16897-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-XTS to the crypto library.
This will be used to provide a streamlined implementation of the
"xts(aes)" crypto_skcipher algorithm. I'm also planning to use this
directly in fscrypt and blk-crypto-fallback.
As usual, the architecture-optimized AES-XTS code will be migrated into
the library as well (using the hooks provided in this commit),
eliminating lots of repetitive boilerplate code. Compared to direct
implementation of "xts(aes)", I've also eliminated the requirement for
architectures to implement ciphertext stealing, as the library just
handles it portably instead. That will simplify things considerably.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-CTR and AES-XCTR to the crypto library.
These will be used to provide streamlined implementations of the
"ctr(aes)" and "xctr(aes)" crypto_skcipher algorithms. Most users of
"ctr(aes)" will also be able to switch to the library, which as usual
will be simpler and faster, e.g.:
- net/mac80211/fils_aead.c
- net/mac802154/llsec.c
As usual, the architecture-optimized AES-CTR and AES-XCTR code will be
migrated into the library as well (using the hooks provided in this
commit), eliminating lots of repetitive boilerplate code.
This is also a prerequisite for supporting AES-GCM, AES-CCM, and
AES-HCTR2 in the crypto library.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260715221153.246410-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-CBC and AES-CBC-CTS to the crypto library.
These will be used to provide streamlined implementations of the
"cbc(aes)" and "cts(cbc(aes))" crypto_skcipher algorithms. Most users
of these crypto_skcipher algorithms will also be able to switch to the
library, which as usual will be simpler and faster, e.g.:
- block/blk-crypto-fallback.c (for AES-128-CBC-ESSIV)
- fs/crypto/crypto.c (for AES-128-CBC-ESSIV)
- fs/crypto/fname.c (for AES-256-CTS and AES-128-CBC)
- kernel/bpf/crypto.c
- net/ceph/crypto.c
- security/keys/encrypted-keys/encrypted.c
As usual, the architecture-optimized AES-CBC and AES-CBC-CTS code will
be migrated into the library as well (using the hooks provided in this
commit), eliminating lots of repetitive boilerplate code.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260715221153.246410-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-ECB to the crypto library.
This will be used to provide a streamlined implementation of the
"ecb(aes)" crypto_skcipher algorithm. fs/crypto/keysetup_v1.c will also
use aes_ecb_encrypt() directly.
As usual, the architecture-optimized AES-ECB code will be migrated into
the library as well (using the hooks provided in this commit),
eliminating lots of repetitive boilerplate code.
ECB is obsolete of course, but we need this for parity with the
traditional API and to support some odd users of ECB in the kernel.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Create a documentation file libcrypto-unauth-encryption.rst to hold the
documentation for this and other unauthenticated encryption modes.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260715221153.246410-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Starting in commit 7137cbf2b5c9 ("crypto: aes - Add cmac, xcbc, and
cbcmac algorithms using library"), the aes module (CRYPTO_AES) supports
CBC based MACs using the corresponding library functions.
To avoid including unneeded functionality, that support honors the
existing CRYPTO_CMAC, CRYPTO_XCBC, and CRYPTO_CCM kconfig options. The
dependencies are selected if at least one of those is enabled.
However, the select statements don't correctly handle the case where
CRYPTO_AES=y and (for example) CRYPTO_CMAC=m. In that case the
dependencies get selected at level 'm', due to how the kconfig language
works. That causes a linker error.
Fix this by changing the selection conditions to use '!= n'.
A similar issue also exists for CRYPTO_LIB_AES's conditional selection
of CRYPTO_LIB_UTILS. The same '!= n' would work, but instead just make
CRYPTO_LIB_AES always select CRYPTO_LIB_UTILS. CRYPTO_LIB_UTILS is
lightweight, and it's needed by most AES modes and many other things.
Fixes: 7137cbf2b5c9 ("crypto: aes - Add cmac, xcbc, and cbcmac algorithms using library")
Fixes: 309a7e514da7 ("lib/crypto: aes: Add support for CBC-based MACs")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260709022954.45113-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
No definitions of md5_mod_init_arch() remain, so remove the code that
handles it.
Link: https://patch.msgid.link/20260629032552.26100-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
During randconfig testing, I came across a lot of warnings for the newly
added carryless multiplication function triggering excessive stack usage
from spilling temporary variables to the stack:
lib/crypto/gf128hash.c:166:1: error: stack frame size (1192) exceeds limit (1024) in 'polyval_mul_generic' [-Werror,-Wframe-larger-than]
In addition to the possible risk of overflowing the kernel stack,
the generated object code surely performs very poorly.
This only happens on architectures that don't provide uint128_t
(which should be all 32-bit architectures on modern compilers), but
though I tested random x86 and arm configs, I only saw this with arm's
CONFIG_THUMB2_KERNEL, which adds more pressure to the register allocator.
The testing was done using clang-22, I don't know if gcc has the same
problem. Marking clmul32() as noinline_for_stack experimentally shows
all of the affected builds to completely solve the problem, reducing
the stack usage to a few bytes as expected.
Since u64 arithmetic frequently leads to compilers badly optimizing
32-bit targets, keeping clmul32 out of line is likely to help on
other 32-bit configurations as well when they run into this problem,
though it may also result in a small performance degradation in
configurations that would benefit from inlining.
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Link: https://patch.msgid.link/20260611125952.3387258-1-arnd@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
MD5 is obsolete, is vulnerable to collision attacks, and is being
replaced by SHA-256 in new systems. It doesn't make sense to continue
to maintain architecture-optimized implementations of MD5. Effort
should be spent on modern algorithms.
Indeed, architecture-optimized MD5 code remains only for powerpc. It
was already removed from mips and sparc, and it never existed for any
other architecture (e.g. x86, arm, or arm64) in the first place.
Earlier the decision was made to keep the powerpc MD5 code for a while
anyway because of someone using it via AF_ALG via libkcapi-hasher
(https://lore.kernel.org/r/f0d771d5-ed70-444c-957a-ad4c16f6c115@csgroup.eu/)
However, with AF_ALG itself now being on its way out due to its
continuous stream of security vulnerabilities
(https://lore.kernel.org/r/20260430011544.31823-1-ebiggers@kernel.org/),
it's also time to be a bit more forceful with nudging people towards
userspace crypto code. It's always been the better solution anyway, and
it's much more efficient if properly optimized code is used.
Note that the md5-asm.S file contains no privileged instructions and
could be run in userspace just fine.
Thus, we now have two factors going against keeping the powerpc MD5
code. Different people might weigh these two factors differently, but I
think the two of them together make the removal the clear choice.
Let's remove it.
Acked-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260506030005.9698-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux
Pull more crypto library updates from Eric Biggers:
"Crypto library fix and documentation update:
- Fix an integer underflow in the mpi library
- Improve the crypto library documentation"
* tag 'libcrypto-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux:
lib/crypto: docs: Add rst documentation to Documentation/crypto/
docs: kdoc: Expand 'at_least' when creating parameter list
lib/crypto: mpi: Fix integer underflow in mpi_read_raw_from_sgl()
|
|
Yiming reports an integer underflow in mpi_read_raw_from_sgl() when
subtracting "lzeros" from the unsigned "nbytes".
For this to happen, the scatterlist "sgl" needs to occupy more bytes
than the "nbytes" parameter and the first "nbytes + 1" bytes of the
scatterlist must be zero. Under these conditions, the while loop
iterating over the scatterlist will count more zeroes than "nbytes",
subtract the number of zeroes from "nbytes" and cause the underflow.
When commit 2d4d1eea540b ("lib/mpi: Add mpi sgl helpers") originally
introduced the bug, it couldn't be triggered because all callers of
mpi_read_raw_from_sgl() passed a scatterlist whose length was equal to
"nbytes".
However since commit 63ba4d67594a ("KEYS: asymmetric: Use new crypto
interface without scatterlists"), the underflow can now actually be
triggered. When invoking a KEYCTL_PKEY_ENCRYPT system call with a
larger "out_len" than "in_len" and filling the "in" buffer with zeroes,
crypto_akcipher_sync_prep() will create an all-zero scatterlist used for
both the "src" and "dst" member of struct akcipher_request and thereby
fulfil the conditions to trigger the bug:
sys_keyctl()
keyctl_pkey_e_d_s()
asymmetric_key_eds_op()
software_key_eds_op()
crypto_akcipher_sync_encrypt()
crypto_akcipher_sync_prep()
crypto_akcipher_encrypt()
rsa_enc()
mpi_read_raw_from_sgl()
To the user this will be visible as a DoS as the kernel spins forever,
causing soft lockup splats as a side effect.
Fix it.
Reported-by: Yiming Qian <yimingqian591@gmail.com> # off-list
Fixes: 2d4d1eea540b ("lib/mpi: Add mpi sgl helpers")
Signed-off-by: Lukas Wunner <lukas@wunner.de>
Cc: stable@vger.kernel.org # v4.4+
Reviewed-by: Ignat Korchagin <ignat@linux.win>
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Link: https://lore.kernel.org/r/59eca92ff4f87e2081777f1423a0efaaadcfdb39.1776003111.git.lukas@wunner.de
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Pull bitmap updates from Yury Norov:
- new API: bitmap_weight_from() and bitmap_weighted_xor() (Yury)
- drop unused __find_nth_andnot_bit() (Yury)
- new tests and test improvements (Andy, Akinobu, Yury)
- fixes for count_zeroes API (Yury)
- cleanup bitmap_print_to_pagebuf() mess (Yury)
- documentation updates (Andy, Kai, Kit).
* tag 'bitmap-for-v7.1' of https://github.com/norov/linux: (24 commits)
bitops: Update kernel-doc for sign_extendXX()
powerpc/xive: simplify xive_spapr_debug_show()
thermal: intel: switch cpumask_get() to using cpumask_print_to_pagebuf()
coresight: don't use bitmap_print_to_pagebuf()
lib/prime_numbers: drop temporary buffer in dump_primes()
drm/xe: switch xe_pagefault_queue_init() to using bitmap_weighted_or()
ice: use bitmap_empty() in ice_vf_has_no_qs_ena
ice: use bitmap_weighted_xor() in ice_find_free_recp_res_idx()
bitmap: introduce bitmap_weighted_xor()
bitmap: add test_zero_nbits()
bitmap: exclude nbits == 0 cases from bitmap test
bitmap: test bitmap_weight() for more
asm-generic/bitops: Fix a comment typo in instrumented-atomic.h
bitops: fix kernel-doc parameter name for parity8()
lib: count_zeros: unify count_{leading,trailing}_zeros()
lib: count_zeros: fix 32/64-bit inconsistency in count_trailing_zeros()
lib: crypto: fix comments for count_leading_zeros()
x86/topology: use bitmap_weight_from()
bitmap: add bitmap_weight_from()
lib/find_bit_benchmark: avoid clearing randomly filled bitmap in test_find_first_bit()
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux
Pull crypto library updates from Eric Biggers:
- Migrate more hash algorithms from the traditional crypto subsystem to
lib/crypto/
Like the algorithms migrated earlier (e.g. SHA-*), this simplifies
the implementations, improves performance, enables further
simplifications in calling code, and solves various other issues:
- AES CBC-based MACs (AES-CMAC, AES-XCBC-MAC, and AES-CBC-MAC)
- Support these algorithms in lib/crypto/ using the AES library
and the existing arm64 assembly code
- Reimplement the traditional crypto API's "cmac(aes)",
"xcbc(aes)", and "cbcmac(aes)" on top of the library
- Convert mac80211 to use the AES-CMAC library. Note: several
other subsystems can use it too and will be converted later
- Drop the broken, nonstandard, and likely unused support for
"xcbc(aes)" with key lengths other than 128 bits
- Enable optimizations by default
- GHASH
- Migrate the standalone GHASH code into lib/crypto/
- Integrate the GHASH code more closely with the very similar
POLYVAL code, and improve the generic GHASH implementation to
resist cache-timing attacks and use much less memory
- Reimplement the AES-GCM library and the "gcm" crypto_aead
template on top of the GHASH library. Remove "ghash" from the
crypto_shash API, as it's no longer needed
- Enable optimizations by default
- SM3
- Migrate the kernel's existing SM3 code into lib/crypto/, and
reimplement the traditional crypto API's "sm3" on top of it
- I don't recommend using SM3, but this cleanup is worthwhile
to organize the code the same way as other algorithms
- Testing improvements:
- Add a KUnit test suite for each of the new library APIs
- Migrate the existing ChaCha20Poly1305 test to KUnit
- Make the KUnit all_tests.config enable all crypto library tests
- Move the test kconfig options to the Runtime Testing menu
- Other updates to arch-optimized crypto code:
- Optimize SHA-256 for Zhaoxin CPUs using the Padlock Hash Engine
- Remove some MD5 implementations that are no longer worth keeping
- Drop big endian and voluntary preemption support from the arm64
code, as those configurations are no longer supported on arm64
- Make jitterentropy and samples/tsm-mr use the crypto library APIs
* tag 'libcrypto-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux: (66 commits)
lib/crypto: arm64: Assume a little-endian kernel
arm64: fpsimd: Remove obsolete cond_yield macro
lib/crypto: arm64/sha3: Remove obsolete chunking logic
lib/crypto: arm64/sha512: Remove obsolete chunking logic
lib/crypto: arm64/sha256: Remove obsolete chunking logic
lib/crypto: arm64/sha1: Remove obsolete chunking logic
lib/crypto: arm64/poly1305: Remove obsolete chunking logic
lib/crypto: arm64/gf128hash: Remove obsolete chunking logic
lib/crypto: arm64/chacha: Remove obsolete chunking logic
lib/crypto: arm64/aes: Remove obsolete chunking logic
lib/crypto: Include <crypto/utils.h> instead of <crypto/algapi.h>
lib/crypto: aesgcm: Don't disable IRQs during AES block encryption
lib/crypto: aescfb: Don't disable IRQs during AES block encryption
lib/crypto: tests: Migrate ChaCha20Poly1305 self-test to KUnit
lib/crypto: sparc: Drop optimized MD5 code
lib/crypto: mips: Drop optimized MD5 code
lib: Move crypto library tests to Runtime Testing menu
crypto: sm3 - Remove 'struct sm3_state'
crypto: sm3 - Remove the original "sm3_block_generic()"
crypto: sm3 - Remove sm3_base.h
...
|
|
Since support for big-endian arm64 kernels was removed, the CPU_LE()
macro now unconditionally emits the code it is passed, and the CPU_BE()
macro now unconditionally discards the code it is passed.
Simplify the assembly code in lib/crypto/arm64/ accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401003331.144065-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the SHA-3 code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-9-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the SHA-512 code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the SHA-256 code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-7-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the SHA-1 code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the Poly1305 code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the GHASH and POLYVAL code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the ChaCha code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel mode
NEON at context switch"), kernel-mode NEON sections have been
preemptible on arm64. And since commit 7dadeaa6e851 ("sched: Further
restrict the preemption modes"), voluntary preemption is no longer
supported on arm64 either. Therefore, there's no longer any need to
limit the length of kernel-mode NEON sections on arm64.
Simplify the AES-CBC-MAC code accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260401000548.133151-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since the lib/crypto/ files that include <crypto/algapi.h> need it only
for the transitive inclusion of <crypto/utils.h> (and not all the
traditional crypto API stuff that the rest of <crypto/algapi.h> is
filled with), replace these inclusions with direct inclusions of
<crypto/utils.h>.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260331024438.51783-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
aes_encrypt() now uses AES instructions when available instead of always
using table-based code. AES instructions are constant-time and don't
benefit from disabling IRQs as a constant-time hardening measure.
In fact, on two architectures (arm and riscv) disabling IRQs is
counterproductive because it prevents the AES instructions from being
used. (See the may_use_simd() implementation on those architectures.)
Therefore, let's remove the IRQ disabling/enabling and leave the choice
of constant-time hardening measures to the AES library code.
Note that currently the arm table-based AES code (which runs on arm
kernels that don't have ARMv8 CE) disables IRQs, while the generic
table-based AES code does not. So this does technically regress in
constant-time hardening when that generic code is used. But as
discussed in commit a22fd0e3c495 ("lib/crypto: aes: Introduce improved
AES library") I think just leaving IRQs enabled is the right choice.
Disabling them is slow and can cause problems, and AES instructions
(which modern CPUs have) solve the problem in a much better way anyway.
Link: https://lore.kernel.org/r/20260331024430.51755-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
aes_encrypt() now uses AES instructions when available instead of always
using table-based code. AES instructions are constant-time and don't
benefit from disabling IRQs as a constant-time hardening measure.
In fact, on two architectures (arm and riscv) disabling IRQs is
counterproductive because it prevents the AES instructions from being
used. (See the may_use_simd() implementation on those architectures.)
Therefore, let's remove the IRQ disabling/enabling and leave the choice
of constant-time hardening measures to the AES library code.
Note that currently the arm table-based AES code (which runs on arm
kernels that don't have ARMv8 CE) disables IRQs, while the generic
table-based AES code does not. So this does technically regress in
constant-time hardening when that generic code is used. But as
discussed in commit a22fd0e3c495 ("lib/crypto: aes: Introduce improved
AES library") I think just leaving IRQs enabled is the right choice.
Disabling them is slow and can cause problems, and AES instructions
(which modern CPUs have) solve the problem in a much better way anyway.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260331024414.51545-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Move the ChaCha20Poly1305 test from an ad-hoc self-test to a KUnit test.
Keep the same test logic for now, just translated to KUnit.
Moving to KUnit has multiple benefits, such as:
- Consistency with the rest of the lib/crypto/ tests.
- Kernel developers familiar with KUnit, which is used kernel-wide, can
quickly understand the test and how to enable and run it.
- The test will be automatically run by anyone using
lib/crypto/.kunitconfig or KUnit's all_tests.config.
- Results are reported using the standard KUnit mechanism.
- It eliminates one of the few remaining back-references to crypto/ from
lib/crypto/, specifically a reference to CONFIG_CRYPTO_SELFTESTS.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260327224229.137532-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
MD5 is obsolete. Continuing to maintain architecture-optimized
implementations of MD5 is unnecessary and risky. It diverts resources
from the modern algorithms that are actually important.
While there was demand for continuing to maintain the PowerPC optimized
MD5 code to accommodate userspace programs that are misusing AF_ALG
(https://lore.kernel.org/linux-crypto/c4191597-341d-4fd7-bc3d-13daf7666c41@csgroup.eu/),
no such demand has been seen for the SPARC optimized MD5 code.
Thus, let's drop it and focus effort on the more modern SHA algorithms,
which already have optimized code for SPARC.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260326203341.60393-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
MD5 is obsolete. Continuing to maintain architecture-optimized
implementations of MD5 is unnecessary and risky. It diverts resources
from the modern algorithms that are actually important.
While there was demand for continuing to maintain the PowerPC optimized
MD5 code to accommodate userspace programs that are misusing AF_ALG
(https://lore.kernel.org/linux-crypto/c4191597-341d-4fd7-bc3d-13daf7666c41@csgroup.eu/),
no such demand has been seen for the MIPS Cavium Octeon optimized MD5
code. Note that this code runs on only one particular line of SoCs.
Thus, let's drop it and focus effort on the more modern SHA algorithms,
which already have optimized code for the same SoCs.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260326204824.62010-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since the ChaCha permutation is invertible, the local variable
'permuted_state' is sufficient to compute the original 'state', and thus
the key, even after the permutation has been done.
While the kernel is quite inconsistent about zeroizing secrets on the
stack (and some prominent userspace crypto libraries don't bother at all
since it's not guaranteed to work anyway), the kernel does try to do it
as a best practice, especially in cases involving the RNG.
Thus, explicitly zeroize 'permuted_state' before it goes out of scope.
Fixes: c08d0e647305 ("crypto: chacha20 - Add a generic ChaCha20 stream cipher implementation")
Cc: stable@vger.kernel.org
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260326032920.39408-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
