Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
In case of unknown platforms in filters such as plaform_allow,
integration_plaforms or platform_exclude, report an error and exit.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
bsim requires a special setup for it to be usuable and run by twister.
If this environment is not available, we should not error when the
platform is an integration platform.
Most developers and users of twister will not have the babblesim setup
and reporting errors is misleading and confusing. So if such env. is not
available, just filter as usual and do not error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fixes ruff linting errors UP030, UP031, UP032,
which make you use format specifiers (fstrings)
instead of printf formatting or str.format().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error F541,
where an fstring is used when a normal
string would suffice.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error B023,
where function definitions use
loop variables incorrectly.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error B904,
where an error raised inside an except
clause does not specify from where it
should start.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM102,
where multiple ifs were used instead
of combining them with and or similar
operators.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error E741,
where a variable name is ambiguous
or hard to read (e.g. l).
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM1101,
where a builtin, like any() or all()
has been reimplemented.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes errors E402 - module import
not at top of file.
One error in testplan.py left in,
as it is a part of a obscure 2020 fix
that would need to be re-tested.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
#79174 incorectly interpreted the condition. If no simulator is found
if cannot be "renode" and therefore robot should be considered unsupported.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
add vendor_allow and vendor_exclude to the list of filters. The vendor
data is maintained in the board.yml files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Detect duplicate TestSuites on load, and raise error when it happens
with `--no-detailed-test-id` option which shortens TestSuite name
excluding the test project path prefix, thus increasing chances
for duplicates. Without this check, only the last duplicated
test configuration was selected while others silently ignored.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix duplicate test scenario error reporting to show paths to all
twister.yaml configuration files where these duplicates were found.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Fix wrong reporting about where testcases were executed.
We have:
INFO - 1130 of 1130 executed test cases passed (100.00%) on 0 out of
total 860 platforms (0.00%).
which is obviously wrong.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When using the --build-only into --test-only
Twister setup, NOTRUN statuses were not properly rerun.
Now they are properly run again if runnable.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fixes: #69548
Support extending an existing board with new board variants.
This commit introduces the following changes to allow a board to be
extended out-of-tree.
The board yaml schema is extended to support an extend field which
will be used to identify the board to be extended.
A board 'plank' can be extended like this:
> board:
> extend: plank
> variants:
> - name: ext
> qualifier: soc1
For the rest of the build system this means that there is no longer a
single board directory.
The existing CMake variable BOARD_DIR is kept and reference the
directory which defines the board.
A new CMake variable BOARD_DIRECTORIES provides a list of all
directories which defines board targets for the board.
This means the directory which defines the board as well as all
directories that extends the board.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
if alias or shorthand name is provided on the command line, we need
convert this to complete target name for the filters to work.
Fixes#80332
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
--device-testing implies runnable tests, this was the case before the
rework and was missed in one spot, so instead do that directly in the
argument parser.
Fixes#80428
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We now use hwmv2 to list boards instead of relying on twister specific
config files.
One yaml files (twister.yaml for now) will have all the data needed for
all possible targets and variations of a board reusing most of the data
where possible and variations can override the top level data.
Twister keeps track of 'aliases' of boards and identifies that for
example native_sim is the same as native_sim/native, so either names
will be possible in both test yaml files or on the command line,
however, the reporting will always use the full name, so no there is no
confusion about what is being tested/built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some dict.get() calls did not use a TwisterStatus
as a default value, thus using a NoneType where
TwisterStatus should appear.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Get data for various roots from modules and do not hardcode location
when retrieving list of boards in testplan.
Fixes#71761
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.
Code guarding Twister's Properties was shortened to a
value check only.
QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This change is removing some deprecation warnings
which for some reason causing failing tests with
Python 3.12 on CI #76877.
Also, it fixes warnings from pytest like:
PytestCollectionWarning: cannot collect test class 'TestPlan'
because it has a __init__ constructor
(from: scripts/tests/twister/test_testplan.py)
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Twister scans C-files to find testcases that are implemented
using ZTest framework. Also runs scanning of Elf files
after building. Skip scanning files if it is not required.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Added a switch that show failed/error report from the last run.
Default shows all items found. However, you can specify the number of items
(e.g. --report-summary 15).
It also works well with the --outdir switch
Signed-off-by: Kamil Paszkiet <kamilx.paszkiet@intel.com>
sometime we need use quarantine at runtime when --test-only
add this will help to skip problematic cases in test cycle
to reenable the quarantined case you need add
--quarantine-list quarantine.yaml --quarantine-verify
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Current find_v2_boards() has a lot of unnecessarry fluff.
We can make a new version of that function without it
given its use in TestPlan.add_configurations().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Sped up Twister via name prefix loading.
If the only thing influencing the platform selection
is the platform filter (-p), then we only load *.yaml
files that start the same way as those selected platforms.
We split the platform name to the first '/' or '@'.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Make the --list-tests and --tests-tree options outputs
respect the --tag and --exclude-tag options,
so that only the tests for the specified tags are listed.
Also update the TestPlan report testcases for this change.
Signed-off-by: Declan Snyder <declan.snyder@nxp.com>
Fail a test run gracefully when an unknown test level
is requested to execute with the current test plan.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Follow-up: fe25709a9c twister: add unit_testing soc and board
Moving the soc.yml and board.yml to subsys/testsuite tree.
Introduce subsys/testsuite/arch/archs.yml with unit_testing arch.
Update the Zephyr CMake module unittest.cmake to include HWMv2 and
reorder includes to follow same order as zephyr_default.cmake.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Dmitrii Golovanov
Anas Nashif
Grzegorz Chwierut
Lukasz Mrugala
Wilfried Chauveau
Torsten Rasmussen
Lukasz Fundakowski
Kamil Paszkiet
Hake Huang
Declan Snyder