No description
  • C 96.9%
  • Assembly 0.9%
  • Rust 0.6%
  • Shell 0.6%
  • Python 0.5%
  • Other 0.3%
Find a file
Peng Zhang 1ba3cbf3ec mm: kfence: improve the performance of __kfence_alloc() and __kfence_free()
In __kfence_alloc() and __kfence_free(), we will set and check canary. 
Assuming that the size of the object is close to 0, nearly 4k memory
accesses are required because setting and checking canary is executed byte
by byte.

canary is now defined like this:
KFENCE_CANARY_PATTERN(addr) ((u8)0xaa ^ (u8)((unsigned long)(addr) & 0x7))

Observe that canary is only related to the lower three bits of the
address, so every 8 bytes of canary are the same.  We can access 8-byte
canary each time instead of byte-by-byte, thereby optimizing nearly 4k
memory accesses to 4k/8 times.

Use the bcc tool funclatency to measure the latency of __kfence_alloc()
and __kfence_free(), the numbers (deleted the distribution of latency) is
posted below.  Though different object sizes will have an impact on the
measurement, we ignore it for now and assume the average object size is
roughly equal.

Before patching:
__kfence_alloc:
avg = 5055 nsecs, total: 5515252 nsecs, count: 1091
__kfence_free:
avg = 5319 nsecs, total: 9735130 nsecs, count: 1830

After patching:
__kfence_alloc:
avg = 3597 nsecs, total: 6428491 nsecs, count: 1787
__kfence_free:
avg = 3046 nsecs, total: 3415390 nsecs, count: 1121

The numbers indicate that there is ~30% - ~40% performance improvement.

Link: https://lkml.kernel.org/r/20230403122738.6006-1-zhangpeng.00@bytedance.com
Signed-off-by: Peng Zhang <zhangpeng.00@bytedance.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
History 2023-04-18 16:29:49 -07:00
arch xtensa: reword ARCH_FORCE_MAX_ORDER prompt and help text 2023-04-18 16:29:46 -07:00
block block: remove obsolete config BLOCK_COMPAT 2023-03-16 09:35:44 -06:00
certs Kbuild updates for v6.3 2023-02-26 11:53:25 -08:00
crypto asymmetric_keys: log on fatal failures in PE/pkcs7 2023-03-21 16:23:56 +00:00
Documentation sync mm-stable with mm-hotfixes-stable to pick up depended-upon upstream changes 2023-04-16 12:31:58 -07:00
drivers drm/ttm: remove comment referencing now-removed vmf_insert_mixed_prot() 2023-04-05 19:42:56 -07:00
fs sync mm-stable with mm-hotfixes-stable to pick up depended-upon upstream changes 2023-04-18 14:53:49 -07:00
include mm: move free_area_empty() to mm/internal.h 2023-04-18 16:29:47 -07:00
init init,mm: fold late call to page_ext_init() to page_alloc_init_late() 2023-04-05 19:42:54 -07:00
io_uring block-6.3-2023-03-24 2023-03-24 14:10:39 -07:00
ipc Merge branch 'work.namespace' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2023-02-24 19:20:07 -08:00
kernel sync mm-stable with mm-hotfixes-stable to pick up depended-upon upstream changes 2023-04-18 14:53:49 -07:00
lib lib/test_vmalloc.c: add vm_map_ram()/vm_unmap_ram() test case 2023-04-18 16:29:47 -07:00
LICENSES LICENSES: Add the copyleft-next-0.3.1 license 2022-11-08 15:44:01 +01:00
mm mm: kfence: improve the performance of __kfence_alloc() and __kfence_free() 2023-04-18 16:29:49 -07:00
net mm, treewide: redefine MAX_ORDER sanely 2023-04-05 19:42:46 -07:00
rust Rust fixes for 6.3-rc1 2023-03-03 14:51:15 -08:00
samples kmemleak-test: fix kmemleak_test.c build logic 2023-04-18 16:29:47 -07:00
scripts kasan: remove hwasan-kernel-mem-intrinsic-prefix=1 for clang-14 2023-04-18 16:29:43 -07:00
security mm, treewide: redefine MAX_ORDER sanely 2023-04-05 19:42:46 -07:00
sound ALSA: hda/ca0132: fixup buffer overrun at tuning_ctl_set() 2023-03-14 17:04:53 +01:00
tools sync mm-stable with mm-hotfixes-stable to pick up depended-upon upstream changes 2023-04-18 14:53:49 -07:00
usr usr/gen_init_cpio.c: remove unnecessary -1 values from int file 2022-10-03 14:21:44 -07:00
virt KVM/riscv changes for 6.3 2023-02-15 12:33:28 -05:00
.clang-format cpumask: re-introduce constant-sized cpumask optimizations 2023-03-05 14:30:34 -08:00
.cocciconfig scripts: add Linux .cocciconfig for coccinelle 2016-07-22 12:13:39 +02:00
.get_maintainer.ignore get_maintainer: add Alan to .get_maintainer.ignore 2022-08-20 15:17:44 -07:00
.gitattributes .gitattributes: use 'dts' diff driver for *.dtso files 2023-02-26 15:28:23 +09:00
.gitignore kbuild: rpm-pkg: move source components to rpmbuild/SOURCES 2023-03-16 22:45:56 +09:00
.mailmap mailmap: update jtoppins' entry to reference correct email 2023-04-16 10:41:25 -07:00
.rustfmt.toml rust: add .rustfmt.toml 2022-09-28 09:02:20 +02:00
COPYING COPYING: state that all contributions really are covered by this file 2020-02-10 13:32:20 -08:00
CREDITS There is no particular theme here - mainly quick hits all over the tree. 2023-02-23 17:55:40 -08:00
Kbuild Kbuild updates for v6.1 2022-10-10 12:00:45 -07:00
Kconfig kbuild: ensure full rebuild when the compiler is updated 2020-05-12 13:28:33 +09:00
MAINTAINERS MAINTAINERS: extend memblock entry to include MM initialization 2023-04-05 19:42:55 -07:00
Makefile Linux 6.3-rc4 2023-03-26 14:40:20 -07:00
README Drop all 00-INDEX files from Documentation/ 2018-09-09 15:08:58 -06:00

Linux kernel
============

There are several guides for kernel developers and users. These guides can
be rendered in a number of formats, like HTML and PDF. Please read
Documentation/admin-guide/README.rst first.

In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.  The formatted documentation can also be read online at:

    https://www.kernel.org/doc/html/latest/

There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.

Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.