unittests: test_tokenizer: check if the tokenizer works

Add extra tests to check if the tokenizer is working properly.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <e4b6e880565f40287a69667652adb393879e039b.1773770483.git.mchehab+huawei@kernel.org>
This commit is contained in:
Mauro Carvalho Chehab 2026-03-17 19:09:28 +01:00 • committed by Jonathan Corbet
commit 5f6fc8ed49