Commit graph

14 commits

Author SHA1 Message Date
Ricardo B. Marlière
d6ea9f404b selftests/run_kselftest.sh: Allow choosing per-test log directory
The --per-test-log option currently hard-codes /tmp. However, the system
under test will most likely have tmpfs mounted there. Since it's not clear
which filenames the log files will have, the user should be able to specify
a persistent directory to store the logs. Keeping those logs are important
because the run_kselftest.sh runner will only yield KTAP output, trimming
information that is otherwise available through running individual tests
directly.

Allow --per-test-log to take an optional directory argument. Keep the
existing behaviour when the option is passed without an argument, but if
a directory is provided, create it if needed, reject non-directory paths
and non-writable directories, canonicalize it, and have runner.sh write
per-test logs there instead of /tmp.

This also makes relative paths safe by resolving them before the runner
changes into a collection directory.

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Link: https://lore.kernel.org/r/20260320-selftests-fixes-v1-4-79144f76be01@suse.com
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2026-04-13 11:05:39 -06:00
Ricardo B. Marlière
a82e076f4a selftests/run_kselftest.sh: Resolve BASE_DIR with pwd -P
run_kselftest.sh only needs to canonicalize the directory containing the
script itself. Use shell-native path resolution for that by changing into
the directory and calling pwd -P.

This avoids depending on either realpath or readlink -f while still
producing a physical absolute path for BASE_DIR.

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Link: https://lore.kernel.org/r/20260320-selftests-fixes-v1-3-79144f76be01@suse.com
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2026-04-13 11:05:39 -06:00
Ricardo B. Marlière
dbb6153c13 selftests/run_kselftest.sh: Remove unused $ROOT
Fix the following shellcheck warning:

ROOT appears unused. Verify use (or export if used externally). [SC2034]

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Link: https://lore.kernel.org/r/20260320-selftests-fixes-v1-1-79144f76be01@suse.com
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2026-03-31 16:31:07 -06:00
Hangbin Liu
2964f6b816 selftests: Use ktap helpers for runner.sh
Instead of manually writing ktap messages, we should use the formal
ktap helpers in runner.sh. Brendan did some work in commit d9e6269e33
("selftests/run_kselftest.sh: exit with error if tests fail") to make
run_kselftest.sh exit with the correct return value. However, the output
does not include the total results, such as how many tests passed or failed.

Let’s convert all manually printed messages in runner.sh to use the
formal ktap helpers. Here are what I changed:

  1. Move TAP header from runner.sh to run_kselftest.sh, since
     run_kselftest.sh is the only caller of run_many().
  2. In run_kselftest.sh, call run_many() in main process to count the
     pass/fail numbers.
  3. In run_kselftest.sh, do not generate kselftest_failures_file. Just
     use ktap_print_totals to report the result.
  4. In runner.sh run_one(), get the return value and use ktap helpers for
     all pass/fail reporting. This allows counting pass/fail numbers in the
     main process.
  5. In runner.sh run_in_netns(), also return the correct rc, so we can
     count results during wait.

After the change, the printed result looks like:

  not ok 4 4 selftests: clone3: clone3_cap_checkpoint_restore # exit=1
  # Totals: pass:3 fail:1 xfail:0 xpass:0 skip:0 error:0

  ]# echo $?
  1

Fixed change log commit description errors and long lines:
Shuah Khan <skhan@linuxfoundation.org>

Tested-by: Brendan Jackman <jackmanb@google.com>
Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>
Reviewed-by: Brendan Jackman <jackmanb@google.com>
Link: https://lore.kernel.org/r/20260225010833.11301-1-liuhangbin@gmail.com
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2026-03-31 14:48:48 -06:00
Ricardo B. Marlière
3ec6cefc39 selftests/run_kselftest.sh: Add --skip argument option
Currently the only way of excluding certain tests from a collection is by
passing all the other tests explicitly via `--test`. Therefore, if the user
wants to skip a single test the resulting command line might be too big,
depending on the collection. Add an option `--skip` that takes care of
that.

Link: https://lore.kernel.org/r/20260116-selftests-add_skip_opt-v1-1-ab54afaae81b@suse.com
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2026-01-16 13:46:40 -07:00
Brendan Jackman
d9e6269e33 selftests/run_kselftest.sh: exit with error if tests fail
Parsing KTAP is quite an inconvenience, but most of the time the thing
you really want to know is "did anything fail"?

Let's give the user the his information without them needing
to parse anything.

Because of the use of subshells and namespaces, this needs to be
communicated via a file. Just write arbitrary data into the file and
treat non-empty content as a signal that something failed.

In case any user depends on the current behaviour, such as running this
from a script with `set -e` and parsing the result for failures
afterwards, add a flag they can set to get the old behaviour, namely
--no-error-on-fail.

Link: https://lore.kernel.org/r/20251111-b4-ksft-error-on-fail-v3-1-0951a51135f6@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2025-11-19 15:00:22 -07:00
Yosry Ahmed
1107dc4c5b selftests/run_kselftest.sh: Use readlink if realpath is not available
'realpath' is not always available,  fallback to 'readlink -f' if is not
available. They seem to work equally well in this context.

Link: https://lore.kernel.org/r/20250318160510.3441646-1-yosry.ahmed@linux.dev
Signed-off-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2025-05-15 16:52:47 -06:00
Brendan Jackman
103c0b5e82 selftests/run_kselftest.sh: Fix help string for --per-test-log
This is documented as --per_test_log but the argument is actually
--per-test-log.

Link: https://lore.kernel.org/r/20241220-per-test-log-v1-1-de5afe69fdf4@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2025-01-14 17:06:32 -07:00
Hangbin Liu
9d0b4ad82d kselftest/runner.sh: add netns support
Add a variable RUN_IN_NETNS if the user wants to run all the selected tests
in namespace in parallel. With this, we can save a lot of testing time.

Note that some tests may not fit to run in namespace, e.g.
net/drop_monitor_tests.sh, as the dwdump needs to be run in init ns.

I also added another parameter -p to make all the logs reported separately
instead of mixing them in the stdout or output.log.

Nit: the NUM in run_one is not used, rename it to test_num.

Acked-by: David Ahern <dsahern@kernel.org>
Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-12-23 00:26:32 +00:00
Rishabh Bhatnagar
301d6815cd kselftests: Sort the collections list to avoid duplicate tests
If the collections list is not sorted uniq doesn't weed out duplicate
tests correctly. Make sure to sort it before running uniq.

Signed-off-by: Rishabh Bhatnagar <risbhat@amazon.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2023-06-12 16:39:11 -06:00
Luis Chamberlain
f6a01213e3 selftests: allow runners to override the timeout
The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.

To help with all this:

  o Enhance documentation to avoid future increases of insane timeouts
  o Add the option to allow overriding the default timeout with test
    runners with a command line option

Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2023-06-12 16:39:11 -06:00
Hangbin Liu
93f20eff0c selftests/run_kselftest.sh: fix dry-run typo
Should be -d instead of -n for dry-run.

Fixes: 5da1918446 ("selftests/run_kselftest.sh: Make each test individually selectable")
Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-11-10 14:51:39 -07:00
Kees Cook
5da1918446 selftests/run_kselftest.sh: Make each test individually selectable
Currently with run_kselftest.sh there is no way to choose which test
we could run. All the tests listed in kselftest-list.txt are all run
every time. This patch enhanced the run_kselftest.sh to make the test
collections (or tests) individually selectable. e.g.:

$ ./run_kselftest.sh -c seccomp -t timers:posix_timers -t timers:nanosleep

Additionally adds a way to list all known tests with "-l", usage
with "-h", and perform a dry run without running tests with "-n".

Co-developed-by: Hangbin Liu <liuhangbin@gmail.com>
Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>
Signed-off-by: Kees Cook <keescook@chromium.org>
Tested-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-10-07 07:59:15 -06:00
Kees Cook
f0f0a5df4e selftests: Extract run_kselftest.sh and generate stand-alone test list
Instead of building a script on the fly (which just repeats the same
thing for each test collection), move the script out of the Makefile and
into run_kselftest.sh, which reads kselftest-list.txt.

Adjust the emit_tests target to report each test on a separate line so
that test running tools (e.g. LAVA) can easily remove individual
tests (for example, as seen in [1]).

[1] https://github.com/Linaro/test-definitions/pull/208/commits/2e7b62155e4998e54ac0587704932484d4ff84c8

Signed-off-by: Kees Cook <keescook@chromium.org>
Tested-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-10-07 07:58:54 -06:00