kernel/linux
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ synced 2026-11-13 07:31:07 +00:00
Code Issues Releases Packages Activity
linux/lib/checksum_kunit.c

640 lines
35 KiB
Raw Permalink Normal View History

x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix again checksum tests on big endian CPUs Commit b38460bc463c ("kunit: Fix checksum tests on big endian CPUs") fixed endianness issues with kunit checksum tests, but then commit 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") introduced new issues on big endian CPUs. Those issues are once again reflected by the warnings reported by sparse. So, fix them with the same approach, perform proper conversion in order to support both little and big endian CPUs. Once the conversions are properly done and the right types used, the sparse warnings are cleared as well. Reported-by: Erhard Furtner <erhard_f@mailbox.org> Fixes: 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Guenter Roeck <linux@roeck-us.net> Acked-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Link: https://lore.kernel.org/r/73df3a9e95c2179119398ad1b4c84cdacbd8dfb6.1708684443.git.christophe.leroy@csgroup.eu Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-23 11:41:52 +01:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
kunit: Fix again checksum tests on big endian CPUs Commit b38460bc463c ("kunit: Fix checksum tests on big endian CPUs") fixed endianness issues with kunit checksum tests, but then commit 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") introduced new issues on big endian CPUs. Those issues are once again reflected by the warnings reported by sparse. So, fix them with the same approach, perform proper conversion in order to support both little and big endian CPUs. Once the conversions are properly done and the right types used, the sparse warnings are cleared as well. Reported-by: Erhard Furtner <erhard_f@mailbox.org> Fixes: 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Guenter Roeck <linux@roeck-us.net> Acked-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Link: https://lore.kernel.org/r/73df3a9e95c2179119398ad1b4c84cdacbd8dfb6.1708684443.git.christophe.leroy@csgroup.eu Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-23 11:41:52 +01:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
lib: checksum: use ARRAY_SIZE() to improve assert_setup_correct() Use ARRAY_SIZE() to simplify the assert_setup_correct() function and improve its readability. Link: https://lkml.kernel.org/r/20240726154946.230928-1-thorsten.blum@toblux.com Signed-off-by: Thorsten Blum <thorsten.blum@toblux.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-07-26 17:49:46 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Fix checksum tests on big endian CPUs On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b47 ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>
2023-08-23 15:21:43 +02:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
kunit: Fix again checksum tests on big endian CPUs Commit b38460bc463c ("kunit: Fix checksum tests on big endian CPUs") fixed endianness issues with kunit checksum tests, but then commit 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") introduced new issues on big endian CPUs. Those issues are once again reflected by the warnings reported by sparse. So, fix them with the same approach, perform proper conversion in order to support both little and big endian CPUs. Once the conversions are properly done and the right types used, the sparse warnings are cleared as well. Reported-by: Erhard Furtner <erhard_f@mailbox.org> Fixes: 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Guenter Roeck <linux@roeck-us.net> Acked-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Link: https://lore.kernel.org/r/73df3a9e95c2179119398ad1b4c84cdacbd8dfb6.1708684443.git.christophe.leroy@csgroup.eu Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-23 11:41:52 +01:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
kunit: Fix again checksum tests on big endian CPUs Commit b38460bc463c ("kunit: Fix checksum tests on big endian CPUs") fixed endianness issues with kunit checksum tests, but then commit 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") introduced new issues on big endian CPUs. Those issues are once again reflected by the warnings reported by sparse. So, fix them with the same approach, perform proper conversion in order to support both little and big endian CPUs. Once the conversions are properly done and the right types used, the sparse warnings are cleared as well. Reported-by: Erhard Furtner <erhard_f@mailbox.org> Fixes: 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Guenter Roeck <linux@roeck-us.net> Acked-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Link: https://lore.kernel.org/r/73df3a9e95c2179119398ad1b4c84cdacbd8dfb6.1708684443.git.christophe.leroy@csgroup.eu Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-23 11:41:52 +01:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
kunit: Fix again checksum tests on big endian CPUs Commit b38460bc463c ("kunit: Fix checksum tests on big endian CPUs") fixed endianness issues with kunit checksum tests, but then commit 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") introduced new issues on big endian CPUs. Those issues are once again reflected by the warnings reported by sparse. So, fix them with the same approach, perform proper conversion in order to support both little and big endian CPUs. Once the conversions are properly done and the right types used, the sparse warnings are cleared as well. Reported-by: Erhard Furtner <erhard_f@mailbox.org> Fixes: 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Guenter Roeck <linux@roeck-us.net> Acked-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Link: https://lore.kernel.org/r/73df3a9e95c2179119398ad1b4c84cdacbd8dfb6.1708684443.git.christophe.leroy@csgroup.eu Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-23 11:41:52 +01:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
lib: checksum: hide unused expected_csum_ipv6_magic[] When CONFIG_NET is disabled, an extra warning shows up for this unused variable: lib/checksum_kunit.c:218:18: error: 'expected_csum_ipv6_magic' defined but not used [-Werror=unused-const-variable=] Replace the #ifdef with an IS_ENABLED() check that makes the compiler's dead-code-elimination take care of the link failure. Fixes: f24a70106dc1 ("lib: checksum: Fix build with CONFIG_NET=n") Suggested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Acked-by: Jakub Kicinski <kuba@kernel.org> Signed-off-by: Arnd Bergmann <arnd@arndb.de> Reviewed-by: Simon Horman <horms@kernel.org> Tested-by: Simon Horman <horms@kernel.org> # build-tested Signed-off-by: David S. Miller <davem@davemloft.net>
2024-04-04 18:36:45 +02:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
kunit: Fix again checksum tests on big endian CPUs Commit b38460bc463c ("kunit: Fix checksum tests on big endian CPUs") fixed endianness issues with kunit checksum tests, but then commit 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") introduced new issues on big endian CPUs. Those issues are once again reflected by the warnings reported by sparse. So, fix them with the same approach, perform proper conversion in order to support both little and big endian CPUs. Once the conversions are properly done and the right types used, the sparse warnings are cleared as well. Reported-by: Erhard Furtner <erhard_f@mailbox.org> Fixes: 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Guenter Roeck <linux@roeck-us.net> Acked-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Link: https://lore.kernel.org/r/73df3a9e95c2179119398ad1b4c84cdacbd8dfb6.1708684443.git.christophe.leroy@csgroup.eu Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-23 11:41:52 +01:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
kunit: Fix again checksum tests on big endian CPUs Commit b38460bc463c ("kunit: Fix checksum tests on big endian CPUs") fixed endianness issues with kunit checksum tests, but then commit 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") introduced new issues on big endian CPUs. Those issues are once again reflected by the warnings reported by sparse. So, fix them with the same approach, perform proper conversion in order to support both little and big endian CPUs. Once the conversions are properly done and the right types used, the sparse warnings are cleared as well. Reported-by: Erhard Furtner <erhard_f@mailbox.org> Fixes: 6f4c45cbcb00 ("kunit: Add tests for csum_ipv6_magic and ip_fast_csum") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Guenter Roeck <linux@roeck-us.net> Acked-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Palmer Dabbelt <palmer@rivosinc.com> Link: https://lore.kernel.org/r/73df3a9e95c2179119398ad1b4c84cdacbd8dfb6.1708684443.git.christophe.leroy@csgroup.eu Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-23 11:41:52 +01:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: Add tests for csum_ipv6_magic and ip_fast_csum Supplement existing checksum tests with tests for csum_ipv6_magic and ip_fast_csum. Signed-off-by: Charlie Jenkins <charlie@rivosinc.com> Link: https://lore.kernel.org/r/20240108-optimize_checksum-v15-5-1c50de5f2167@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-08 15:57:06 -08:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00
kunit: add missing MODULE_DESCRIPTION() macros to lib/*.c make allmodconfig && make W=1 C=1 reports for lib/*kunit: WARNING: modpost: missing MODULE_DESCRIPTION() in lib/bitfield_kunit.o WARNING: modpost: missing MODULE_DESCRIPTION() in lib/checksum_kunit.o WARNING: modpost: missing MODULE_DESCRIPTION() in lib/cmdline_kunit.o WARNING: modpost: missing MODULE_DESCRIPTION() in lib/is_signed_type_kunit.o WARNING: modpost: missing MODULE_DESCRIPTION() in lib/overflow_kunit.o WARNING: modpost: missing MODULE_DESCRIPTION() in lib/stackinit_kunit.o Add the missing invocations of the MODULE_DESCRIPTION() macro. Link: https://lkml.kernel.org/r/20240601-md-lib-kunit-tests-v1-1-4493fe0032b9@quicinc.com Signed-off-by: Jeff Johnson <quic_jjohnson@quicinc.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-06-01 10:53:34 -07:00
x86/csum: Improve performance of `csum_partial` 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com
2023-05-10 20:10:02 -05:00