We now use hwmv2 to list boards instead of relying on twister specific
config files.
One yaml files (twister.yaml for now) will have all the data needed for
all possible targets and variations of a board reusing most of the data
where possible and variations can override the top level data.
Twister keeps track of 'aliases' of boards and identifies that for
example native_sim is the same as native_sim/native, so either names
will be possible in both test yaml files or on the command line,
however, the reporting will always use the full name, so no there is no
confusion about what is being tested/built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds a new NOTRUN status, which indicates
that a test was successfully built, but
not run on account of being not
runnable in given test instance.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
like extra_confs, add extr_args filterable by
soc:<arch>:<extra_args> or
platfrom:<platform name>: <extra_args>
simulation:<simulator name>: <extra_args>
which will only apply to given platform or arch
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Testcase execution time doesn't match between twister.xml and
twister.log. Testcase execution time is the sum of the previous
testcases' execution time plus its own execution time in
twister.xml.
This patch fixes the issue above.
Signed-off-by: Ioannis Damigos <ioannis.damigos.uj@renesas.com>
Fix trailing `\\r\\n` (escaped CR/LF) didn't cut off because of rstrip()
removed by #58338, so the CR/LF suffix was never found as the actual line
end was `\\r\\n\n`.
Add ANSI code sequence to `test_handlers` Twister unit test.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Separate the pickled EDT generation from the C-Macro header
generation in gen_defines.py to have a more clear responsibility
of the scripts in the DTS parsing process.
Signed-off-by: Benedikt Schmidt <benedikt.schmidt@embedded-solutions.at>
Currently, debug logging in the console and verbosity
are tightly coupled - verbosity of level 2 and higher
enables logging at the debug level.
This change introduces a separate Twister flag
responsible for controlling the debug logging,
while leaving the rest of verbosity unchanged.
This allows for controlling the verbosity on
both logging levels, according to one's needs.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
In case of problem with parsing hex data from coverage dump,
do not create empty gcda file.
Such empty file will break gcovr parsing.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
This small change concerns the following filter functions:
1. `dt_compat_enabled(C)`:
There's a node with compatible `C` and status "okay".
2. `dt_enabled_alias_with_parent_compat(A, C)`:
There's a node with alias `A` and status "okay", and its parent
has compatible `C`.
3. `dt_label_with_parent_compat_enabled(L, C)`:
There's a node with label `L`, and its parent has compatible `C`
and status "okay".
All three functions involve checking whether some node or its parent has
a given compatible, but the way this has been checked is inconsistent.
Function (1) has done it with this Python conditional:
compat in node.compats
while (2) and (3) have used:
parent.matching_compat == compat
The first check works well with nodes that have multiple compatibles,
and it is more aligned with the notion of "has_compat" as seen in the
devicetree macros for C, CMake, and Kconfig.
Arguably, `matching_compat` shouldn't have been used here, because it is
actually a property of a node's binding, moreso than of the node itself.
In practice, it's usually equal to the first compatible for which edtlib
has found a binding, which at first glance is just more constrained than
the `node.compats` check. However, there also exist obscure cases where
the `node.compats` are empty, while the `node.matching_compat` is not.
For now, the three functions can use a combined check, to improve
consistency and utility while avoiding breakage:
node.matching_compat == compat or compat in node.compats
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
It has been deprecated since Zephyr v2.6.0.
The filter that replaced it - `dt_enabled_alias_with_parent_compat` -
had shared code with it, which can now be inlined into `ast_expr()` to
match the parser's overall code structure.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
When at verbosity 1, we print out the status of TestInstances.
This makes it harder to notice changes at TestCase level,
which require perusing the logs.
This adds TestCase status and reason printing
if verbosity level is 2 or more.
Reason printing is suppressed if the reason is empty or None.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Running sysbuilt tests fails because of missing "run" target.
This adds the default domain context to the command.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Some dict.get() calls did not use a TwisterStatus
as a default value, thus using a NoneType where
TwisterStatus should appear.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
If you interrupt process() operation, we want Twister
to exit as gracefully as it can. This avoids the
UnboundLocalError that could appear e.g.
when interrupting the operation via SIGINT.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This changes how some arguments are set in the `Handler`s.
`options`, `generator_cmd` and `suite_name_check` are now passed as
arguments to the constructor rather than injected from an other module.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Python 3.12 warns that
> Testing an element's truth value will raise an exception in future
> versions. Use specific 'len(elem)' or 'elem is not None' test instead.
> if elem_ts := root.find('testsuite'):
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Space separated lists are deprecated but this notice is not checked for.
extract_fields_from_arg_list also converts lists back to space-separated
lists causing a warning on get_scenario
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Status errors previously logged an error, but didn't fail the running test.
This commit changes that
and introduces a new StatusAttributeError to use there.
One test is modified so it follows proper status form.
One test for the new error has been added.
Status errors now will properly mark the Instance as ERROR
and not run TestCases as SKIP.
This necessitated some code layout changes in runner.py
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Get data for various roots from modules and do not hardcode location
when retrieving list of boards in testplan.
Fixes#71761
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Re-arrange Twister command line options for test plan reporting
to a dedicated group with mutally-exclusive options to reflect
actual implementation and its dry-run execution mode.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Readline method sometimes receives more lines in buffer.
Split them to avoid misinterpreting data in harness module.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use pytest-args parameters provided in command line after
parameters taken from yaml file. The last occurence is
considered by argparse, so parameters from command line
will be used.
Fixes: #77319
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The gtest 1.15 has slightly different output from the original one that
was used to set up the harness. With this change we support both output
formats.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The serial number for debugger selection over USB
can be selected with the dev-id. This change
reflects also more the workflow of west flash
with J-Link.
The usage of SelectEmuBySN breaks the support for
J-Link over IP with twister.
Signed-off-by: Michael Arnold <marnold@baumer.com>
Extend pytest-args from configuration yaml file with args
from command line instead of overwriting.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix confusing `TypeError: 'type' object is not subscriptable`
at _get_installed_packages() if twister runs on python version < 3.9
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Improve DUT selection at DeviceHandler: for each DUT it counts
how many test instances have been failed on it during the current
twister execution, so the next available DUT will be chosen
ordering the eligible DUTs by less failures occured so far.
The new selection mechanism should increase chances to retry failed
tests on different DUTs, for instance to resolve ploblems when some
DUTs have connectivity or HW issues slowing down test plan execution,
or even block the execution when only one test suite runs whereas
the same first DUT candidate in the list is not working and others
were not chosen.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Change Twister PyTest plugin's test finalizing sequence to release
the DUT it is used as the very last operation, after the Test Instance
status becomes fully updated from the execution results.
This also fix a race condition possible when pytest plugin releases
the DUT and it becomes acquired by another test while the current test
is not yet finalized completely.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister DeviceHandler - add test failure counter for how many
test instances have been failed on each DUT (Device Under Test)
when it executes the current test plan.
Output DUT falure counter summary at the end of Twister run.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix Twister DeviceHandler exit on SerialException when
it connects to the serial device in 'flash before' mode.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Several improvements at Twister DeviceHandler when it releases
current DUT (Device Under Test):
- release the exact DUT which is used for the test instance
instead of all configured DUTs which happened to have
the same serial device configured.
- Twister PyTtest harness plugin adjustment to the above.
- additional debug logging to track DUT waiting/retain/release.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Updated the Pytest-Harness methods to capture and log error messages from
the subprocess, when pytest command fails.
Ensured that error messages are logged with `--inline-log` option
and placed in twister log reports.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
While the option to inform pytest about which platform a test is for
(platform being the board, such as mps2/an385) is available, it wasn't
being used. This patch fixes that, by building a pytest command which
includes the platform.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Fix after #71401 merge at Twister Reporting:
* filter status value type mismatch;
* tesplan.json generation incorrect error logs.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Incorrect status assignments were previously just warnings in logs.
Now they are elevated to errors.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Anas Nashif
Mara Furland
Lukasz Mrugala
Hake Huang
Ioannis Damigos
Dmitrii Golovanov
Benedikt Schmidt
Piotr Kosycarz
Grzegorz Swiderski
Wilfried Chauveau
Pieter De Gendt
Maciej Perkowski
Grzegorz Chwierut
Yuval Peress
Michael Arnold
Arkadiusz Cholewinski
Ederson de Souza