kernel/linux
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ synced 2026-11-10 20:48:12 +00:00
Code Issues Releases Packages Activity
linux/include/asm-generic/rwonce.h

98 lines
3.1 KiB
Raw Permalink Normal View History

compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
asm/rwonce: Remove smp_read_barrier_depends() invocation Alpha overrides __READ_ONCE() directly, so there's no need to use smp_read_barrier_depends() in the core code. This also means that __READ_ONCE() can be relied upon to provide dependency ordering. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-30 16:51:07 +00:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
asm/rwonce: Allow __READ_ONCE to be overridden by the architecture The meat and potatoes of READ_ONCE() is defined by the __READ_ONCE() macro, which uses a volatile casts in an attempt to avoid tearing of byte, halfword, word and double-word accesses. Allow this to be overridden by the architecture code in the case that things like memory barriers are also required. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 17:30:47 -07:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
asm/rwonce: Allow __READ_ONCE to be overridden by the architecture The meat and potatoes of READ_ONCE() is defined by the __READ_ONCE() macro, which uses a volatile casts in an attempt to avoid tearing of byte, halfword, word and double-word accesses. Allow this to be overridden by the architecture code in the case that things like memory barriers are also required. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 17:30:47 -07:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
asm/rwonce: Remove smp_read_barrier_depends() invocation Alpha overrides __READ_ONCE() directly, so there's no need to use smp_read_barrier_depends() in the core code. This also means that __READ_ONCE() can be relied upon to provide dependency ordering. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-30 16:51:07 +00:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
asm/rwonce: Remove smp_read_barrier_depends() invocation Alpha overrides __READ_ONCE() directly, so there's no need to use smp_read_barrier_depends() in the core code. This also means that __READ_ONCE() can be relied upon to provide dependency ordering. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-30 16:51:07 +00:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
asm/rwonce: Remove smp_read_barrier_depends() invocation Alpha overrides __READ_ONCE() directly, so there's no need to use smp_read_barrier_depends() in the core code. This also means that __READ_ONCE() can be relied upon to provide dependency ordering. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-30 16:51:07 +00:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
rwonce: handle KCSAN like KASAN in read_word_at_a_time() read_word_at_a_time() is allowed to read out of bounds by straddling the end of an allocation (and the caller is expected to then mask off out-of-bounds data). This works as long as the caller guarantees that the access won't hit a pagefault (either by ensuring that addr is aligned or by explicitly checking where the next page boundary is). Such out-of-bounds data could include things like KASAN redzones, adjacent allocations that are concurrently written to, or simply an adjacent struct field that is concurrently updated. KCSAN should ignore racy reads of OOB data that is not actually used, just like KASAN, so (similar to the code above) change read_word_at_a_time() to use __no_sanitize_or_inline instead of __no_kasan_or_inline, and explicitly inform KCSAN that we're reading the first byte. We do have an instrument_read() helper that calls into both KASAN and KCSAN, but I'm instead open-coding that here to avoid having to pull the entire instrumented.h header into rwonce.h. Also, since this read can be racy by design, we should technically do READ_ONCE(), so add that. Fixes: dfd402a4c4ba ("kcsan: Add Kernel Concurrency Sanitizer infrastructure") Signed-off-by: Jann Horn <jannh@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Marco Elver <elver@google.com> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
2025-03-25 17:01:34 +01:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
rwonce: handle KCSAN like KASAN in read_word_at_a_time() read_word_at_a_time() is allowed to read out of bounds by straddling the end of an allocation (and the caller is expected to then mask off out-of-bounds data). This works as long as the caller guarantees that the access won't hit a pagefault (either by ensuring that addr is aligned or by explicitly checking where the next page boundary is). Such out-of-bounds data could include things like KASAN redzones, adjacent allocations that are concurrently written to, or simply an adjacent struct field that is concurrently updated. KCSAN should ignore racy reads of OOB data that is not actually used, just like KASAN, so (similar to the code above) change read_word_at_a_time() to use __no_sanitize_or_inline instead of __no_kasan_or_inline, and explicitly inform KCSAN that we're reading the first byte. We do have an instrument_read() helper that calls into both KASAN and KCSAN, but I'm instead open-coding that here to avoid having to pull the entire instrumented.h header into rwonce.h. Also, since this read can be racy by design, we should technically do READ_ONCE(), so add that. Fixes: dfd402a4c4ba ("kcsan: Add Kernel Concurrency Sanitizer infrastructure") Signed-off-by: Jann Horn <jannh@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Marco Elver <elver@google.com> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
2025-03-25 17:01:34 +01:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00
rwonce: handle KCSAN like KASAN in read_word_at_a_time() read_word_at_a_time() is allowed to read out of bounds by straddling the end of an allocation (and the caller is expected to then mask off out-of-bounds data). This works as long as the caller guarantees that the access won't hit a pagefault (either by ensuring that addr is aligned or by explicitly checking where the next page boundary is). Such out-of-bounds data could include things like KASAN redzones, adjacent allocations that are concurrently written to, or simply an adjacent struct field that is concurrently updated. KCSAN should ignore racy reads of OOB data that is not actually used, just like KASAN, so (similar to the code above) change read_word_at_a_time() to use __no_sanitize_or_inline instead of __no_kasan_or_inline, and explicitly inform KCSAN that we're reading the first byte. We do have an instrument_read() helper that calls into both KASAN and KCSAN, but I'm instead open-coding that here to avoid having to pull the entire instrumented.h header into rwonce.h. Also, since this read can be racy by design, we should technically do READ_ONCE(), so add that. Fixes: dfd402a4c4ba ("kcsan: Add Kernel Concurrency Sanitizer infrastructure") Signed-off-by: Jann Horn <jannh@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Marco Elver <elver@google.com> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
2025-03-25 17:01:34 +01:00
rwonce: fix crash by removing READ_ONCE() for unaligned read When arm64 is built with LTO, it upgrades READ_ONCE() to ldar / ldapr (load-acquire) to avoid issues that can be caused by the compiler optimizing away implicit address dependencies. Unlike plain loads, these load-acquire instructions actually require an aligned address. For now, fix it by removing the READ_ONCE() that the buggy commit introduced. Fixes: ece69af2ede1 ("rwonce: handle KCSAN like KASAN in read_word_at_a_time()") Reported-by: Nathan Chancellor <nathan@kernel.org> Closes: https://lore.kernel.org/r/20250326203926.GA10484@ax162 Signed-off-by: Jann Horn <jannh@google.com> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
2025-03-26 22:04:36 +01:00
compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h In preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'. Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-10-15 16:29:32 -07:00