Follow-up: fe25709a9c twister: add unit_testing soc and board
Moving the soc.yml and board.yml to subsys/testsuite tree.
Introduce subsys/testsuite/arch/archs.yml with unit_testing arch.
Update the Zephyr CMake module unittest.cmake to include HWMv2 and
reorder includes to follow same order as zephyr_default.cmake.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Enhance the -s option of twister, used to point to a single scenario in
a testsuite.
- Now accept --scenario on the command line, --test still supported.
- If no testsuite is provided, extract testsuite from scenario
identifier and avoid parsing the whole tree if we only want to address
one scenario in a testsuite.
- If just the identifier of the scenario is provided to
--test/--scenario option, try to find this as well, do not need the
full path for that.
Something like this is now possible:
twister --scenario kernel.threads.init --list-tests
twister -T <path> --scenario kernel.threads.init --list-tests
twister -T <path>/kernel.threads.init --list-tests
All should print the same output.
Fixes#67307
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
As saving tests writes to file, rather than appending to it,
we should indicate that in the --save-tests help.
--load-tests help changed so its grammar is the same as --save-tests's.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Automatically populate the snippet roots from Zephyr modules, instead of
only looking in `ZEPHYR_BASE`.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Additional checks for Twister command line options `--coverage-tool`
and `--coverage-formats`.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister now uses GCOVR by default as the more reliable code
coverage reporting tool instead of LCOV.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
One can not even replace sucessfully pytest basic sample `pytest-args`
with command line "--pytest-args", as all it does is to append a single
string to current list of commands, making it impossible to send several
arguments.
This patch fixes that by allowing several instances of `--pytest-args`
to compose the whole list of arguments to be passed to pytest.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Useful option during development and when applying smaller change to the
code and verifying changes using tests. This works fine as long as no
dependencies or major changes are done, i.e. when editing few files and
in a subsystem. Use with caution and use on your own risk :-)
When building multiple tests, this provide significant boost, up to 300%
faster than when rebuilding everything from scratch or when re-running
cmake every single time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend Twister command line with --pytest-args. This parameter
is passed to pytest subprocess. It allows to select a specific
testcase from a test suite.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
* Replace native_posix references with native_sim
in arguments help messages
* For the seed parameter, correct platform check to accept
native_sim
* Use native_sim in twister tests
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Test suites names are not being handled uniformly for tests not
in zephyr tree. Their names depend on -T arg used in twister's CLI.
The newly added options allow to select if twister should add paths
to suite names. This is needed if test plans are to be used for tests
outside of zephyr tree.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Twister allows us to control maximum execution time for each test
with timeout value in test's .yaml configuration and
platform level timeout multiplier which allows us to tweak
timeout value for specific platform.
However, sometimes we want to additionally adjust tests timeouts
when running twister. This is especially useful in case of
simulation platform as simulation time may depend on the host
speed & load, we may select different (i.e. cycle accurate but
slower one) simulation method, etc...
Let's introduce global (twister-level) timeout multiplier option.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
Add a new option --vendor which allows filtering by vendors tracked in
the board/platform yaml file. The vendor string is compatible with DTS
and is what we have in dts/bindings/vendor-prefixes.txt.
Providing multiple vendors is also supported.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
While the code already includes some provisions to allow running outside
of a git checkout, attempting that right now causes a failure to generate
the XML report, as the JSON report creates a null `zephyr_version` field.
The reason for that is that the original code doesn't set `self.version`
when the subprocess returns a non-zero status (and also doesn't log.)
Instead of having to set `self.version` for all failure cases, initialize
it to the failure state, and log if it hasn't changed from that.
Signed-off-by: Diego Elio Pettenò <flameeyes@meta.com>
If size is given then twister prnits the size information for the ELF
but returns one which if used in automation fails the command.
Return zero as the command is successful. Also, fix the unit test case.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
When using twister to generate coverage with the coverage tool lcov,
allow using --coverage-formats to pick if you want lcov or html,
defaulting to both. Picking html will also use lcov, since that is
required for geninfo. This will allow callers to avoid the potentially
slow and disk intensive html reports if they only wanted the lcov info
file.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Fixes Issue #60522 by guarding member access against None parameters.
Shortens the init by using the if...else assignment.
Fixes two typos.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fix `--device-flash-with-test` command line options check logic
to be effective with `--device-testing`.
Remove `--device-flash-timeout` check for `--device-testing`
presence as marginal.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We have many tests that are marked as slow that do not run in CI and any
known workflow, to isolate them and be able to run them on their own,
add a new option --enable-slow-only which causes only those tests to build
and execute.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When calling twister with --coverage and no platform specified on the
command line, set the the value of coverage platforms to an empty list.
Fixes#57534
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Making the necessary changes to enable the new pytest plugin.
By default Twister should work without the pytest-twister-harness
plugin installed. To achieve this, each time Twister calls pytest,
the PYTHONPATH environment variable is expanded and the
`-p twister_harness.plugin` option is added to the pytest command.
Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a new option to Twister that allows to load alternative
configuration files, `--alt-config-root`. This new option takes an
arbitrary number of root folders where alternative configuration files
can be stored. Twister will check if a test configuration file exists in
any of the alternative test configuration folders. For example, given
`$test_root/tests/foo/testcase.yaml`, Twister will use
`$alt_config_root/tests/foo/testcase.yaml` if it exists.
This feature can be useful if an out-of-tree project needs to run
upstream tests in different configurations, potentially not available
upstream (e.g. an out-of-tree board, or Kconfig setting).
Note that overlaying has been discarded because we can't easily delete
fields from the original configuration file, something that in certain
cases could be required. Current approach may lead to some dupplication,
but guarantees full control of the test configuration.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Make twister to raise error by default whenever duplicated test ids
are found. Remove redundant option to list duplicates.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Twister does not handle out-of-tree boards automatically. Modules like
e.g. example-application that define their own board root folder, are
required to inform twister via `--board-root`. In contrast, west is able
to to this automatically. This patch makes twister a bit smarter by
automatically filling the board root list with module paths as well.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Add option to shuffle tests (randomly) before they are split into subsets.
This allow to fairly distribute tests across subsets,
so that long tests are not crowded in a single group.
This also allows to detect unwanted dependencies in test execution order.
Also, added option to provide custom seed to random generator
used to shuffle tests.
This will allow to reproduce certain test order if needed.
Used seed is printed at console.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
Use explicit flash command execution timeout at DeviceHandler
instead of the hardcoded value of 60 sec.
When a HW platform executes test cases right after the flash command,
the test case remaining timeout is affected by how much time the flash
operation consumes. Some simulation platforms need to spend significant
amount of time on each 'flash' cycle, usually adding the same delay on
each test case.
This improvement adds two new command line options and device map fields:
--device-flash-timeout - for the flash operation timeout
('flash-timeout' device map field).
--device-flash-with-test - to indicate that the platform flash
command also runs a test case, so the overall timeout should be
calculated as a sum of the flash timeout and the current test case
timeout to receive all console output from the platform
('flash-with-test' device map field).
The device map field values override command line values for the
particular platform where configured.
Default behavior is backward compatible: flash operation fixed timeout
is 60 sec. not including the test case timeout.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Add support test levels and the ability to assign a specific test to one
or more levels. Using command line options of twister it is then possible
to select a level and just execute the tests included in this level.
Additionally, a test configuration allows definiing level
dependencies and additional inclusion of tests into a specific level if
the test itself does not have this information already.
In the configuration file you can include complete components using
regular expressions and you can specify which test level to import from
the same file, making management of levels easier.
To help with testing outside of upstream CI infrastructure, additional
options are available in the configuration file, which can be hosted
locally. As of now, those options are available:
- Ability to ignore default platforms as defined in board definitions
(Those are mostly emulation platforms used to run tests in upstream
CI)
- Option to specify your own list of default platforms overriding what
upstream defines.
- Ability to override build_onl_all options used in some testscases.
This will treat tests or sample as any other just build for default
platforms you specify in the configuation file or on the command line.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Disables allowing the python argparse library from automatically
shortening command line arguments, this prevents issues whereby
a new command is added and code that wrongly uses the shortened
command of an existing argument which is the same as the new
command being added will silently change script behaviour.
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
This reverts commit 21eb27c5c0.
This change has been a source of much trouble and ends up preventing us
from tracking test results across tree. It uses the repo name as the
namespace, and that is not always the same and does not have to be
called 'zephyr', depending on where you are running and in which
environment.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Implementation ported from TwisterV2.
- quarantine handled by separate module
- multiple yaml allowed from args: --quarantine-list
- scenarios, platforms, architectures keywords in
quarantine yaml are optional, if not given - means take it all
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.
The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
When using llvm we need to set the gcov-tool to "llvm-cov gcov" but
the lcov tool is incapable of passing arguments to the gcov-tool. i.e.
the following cannot work:
$ lcov --gcov-tool "llvm-cov gcov" ...
Instead, create a symlink to llvm-cov prefixed as `gcov` which by the
documentation of llvm-cov will alias to `llvm-cov gcov` subcommand.
Signed-off-by: Yuval Peress <peress@google.com>
This commit expands Twister's CLI so that arguments passed after
a double dash (`--`) get forwarded to the ztest test executable when
run. When built for native_posix, ztest provides a useful CLI that
allows filtering individual suites and tests, adjusting timing settings,
and controlling the shuffling seed. Currently there is no easy way to
use these (with the exception of `--seed`) without needing to dig out
the resulting `zephyr.exe` binary from the build dir and invoke it
manually.
### Examples
Run a specific ztest suite only (useful when writing tests and you don't
want to run a long testcase in its entirety)
```
$ scripts/twister \
-p native_posix \
-s
zephyr/tests/lib/cmsis_dsp/complexmath/libraries.cmsis_dsp.complexmath \
-- -test=complexmath_q31::*
```
Unrecognized arguments that precede the double dash will result in an
error message:
```
$ scripts/twister \
-p native_posix \
--foobar 123 \
-- -test=basic_math_q7::*
...
Unrecognized arguments found: '--foobar 123'. Use -- to delineate extra
arguments for test binary or pass -h for help.
```
Signed-off-by: Tristan Honscheid <honscheid@google.com>
This commit calculate memory footprint from build.log and
proposes an alternative approach to #2465.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
add 2 choices for '--runtime-artifact-cleanup' option,
'passing' to delete artifacts of passing tests,
'all' to delete artifacts of both passing tests
and failing tests
keep testcase_extra.config, it contains some extra Kconfig
needed for the tests and they is only generated first time
when run retry failed
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
With the patch a relative path between a test and west's project top
directory is always calculated instead of being relative to zephyr
base. The test names and results structur change by addition of one
level. This way the names/structure conventon for tests from zephyr and
modules is alligned.
For external tests ".." parts are replaced with "external".
This allows to maintain unified folder structure also for results of
external tests.
Without this patch the folder/name allignment is lost for
external and module test becaming depandant on a way how
twister was called.
.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Version is not enough, we need the run date but also the commit date, to
ease reporting and use in dashboards.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some scenarios we might end up with a null version value in the json
file, if the version can't be determined, set it to Unkown instead.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Follow-up: #41301
This commit is a rework and cleanup of the tools handling in Zephyr
CMake build system.
Instead of directly loading code a CMake modules for tool lookup, the
host tools now follows the CMake `find_package()` pattern for finding
programs / tools in module mode.
This makes it more clear which modules are responsible for finding tools
and which modules provides build integration / features.
The following tools can now be found using `find_package()`:
- Zephyr-sdk : find_package(Zephyr-sdk <version>)
- Generic host tools: find_package(HostTools)
This further allows us to decouple the `verify-toolchain` CMake script
part required by `twister` into a tool lookup module and a dedicated
CMake script which utilizes the lookup module.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
With verbose output enabled, twister logs the cmake arguments as a
python array whenever it invokes cmake to generate a build system.
This is clunky output for manual reproduction, because it forces the
developer to manually join the arguments together for consumption by
the shell.
As an enhancement for this use case, use the standard library
shlex.join() function to produce a string that can be copy/pasted
directly into the shell on POSIX platforms.
We should use this function instead of str.join, because it correctly
handles things like shell quoting for options with spaces in them.
This function is not available on Windows, so we fall back on behavior
that is similar to the old one there.
The main difference on Windows is that the output now includes the
path to cmake as well, and not just its arguments. (This is the same
on POSIX, and is intended to help debug if multiple cmakes are
installed on the same machine.)
We are running cmake from a couple of different places, so doing this
cleanly requires adding a shared module with the platform abstraction.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This commit makes the `--ninja` argument default to `true` when neither
`-k` nor `--make` is specified (i.e. when Make generator is not
selected).
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
When `--short-build-path` argument is specified to the twister, the
following error message is displayed because the Python argparse module
does not allow specifying a mutually exclusive argument as required:
ValueError: mutually exclusive arguments must be optional
This commit removes the `required` parameter when adding the `--ninja`
argument, which is a mutually exclusive argument, and adds a manual
check to validate this condition instead.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Generates an error if --coverage-formats option is supplied without
coverage tool being set to gcovr.
Signed-off-by: Jason Wright <jwright@synchron.com>
Torsten Rasmussen
Anas Nashif
Lukasz Mrugala
Jordan Yates
Dmitrii Golovanov
Ederson de Souza
Grzegorz Chwierut
Alberto Escolar Piedras
Øyvind Rønningstad
Maciej Perkowski
Evgeniy Paltsev
Diego Elio Pettenò
Chaitanya Tata
Jeremy Bettis
Gerard Marull-Paretas
Piotr Kosycarz
Jamie McCrae
Al Semjonovs
Tom Burdick
Yuval Peress
Tristan Honscheid
Katarzyna Giądła
Cong Nguyen Huu
Martí Bolívar
Stephanos Ioannidis
Bartosz Bilas
Jason Wright