Setup logging per process to fix issue on both mac and windows where
handlers are not available to the processes.
Fixes#86237
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
native_sim only works and builds on Linux, when running twister on the
Mac, this platform fails to build and reports errors.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Introduce a new command-line option `--keep-artifacts` in twister that
allows users to specify which artifacts should be preserved
during test cleanup in addition to the default set.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Co-authored-by: Anas Nashif <anas.nashif@intel.com>
These files can be useful to troubleshoot a test that's failing,
and they're really small, too.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
NOTE: Even though previous commits indicate, that this can only happen on
MacOS, that's actually not true. It happens on Linux as well.
The constructor of `psutil.Process` can throw an exception as well, so we
need to wrap the whole loop in another try, unfortunately.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
Problem
-------
Board & SoC extensions are used to define out-of-tree board variants or
SoC qualifiers. When a board is extended, it has multiple directories
associated with it (each with its own `board.yml`), where twister should
be able to find additional platform files to support these qualifiers.
Currently, this doesn't work, because twister only traverses the primary
BOARD_DIR and ignores the rest.
The fix would've been trivial in the case of "legacy" platform files,
i.e. those of the form `<normalized_board_target>.yaml`, but it's less
straightforward for the newly introduced `twister.yaml` format.
A `twister.yaml` file contains platform configuration that can be shared
by multiple board targets and tweaked for specific targets by using the
top-level `variants` key. Normally, there is at most one `twister.yaml`
per board, but the file isn't necessarily unique to one board. Instead,
it's unique to one directory, which may define multiple boards (as is
the case with e.g. `boards/qemu/x86/`).
With extensions in the picture, the goal is to initialize platforms when
given multiple `twister.yaml` per board. The OOT files are expected to
only provide information about OOT board targets, without being able to
override in-tree targets (same principle as in the Zephyr build system).
Solution
--------
The `twister.yaml` handling is broken up into multiple passes - first
loading all the files, then splitting the `variants` keys apart from the
shared configuration, before constructing the Platform instances.
The purpose of the split is to treat the variant information as global,
instead of making unnecessary or faulty assumptions about locality.
Remember that the build system can derive board target names not only
from `board.yml`, but from `soc.yml` too. Considering that any board may
end up using an OOT-extended SoC (and hence multiple `soc.yml` files),
not every board target can be said to belong to some board dir.
Unlike the variant data, the remaining top-level config is still rooted
to the primary BOARD_DIR and inherited by the extension dirs from there.
This is quite intuitive in most imagined cases, but there is a caveat:
if a `twister.yaml` resides in an extension dir, then it is allowed to
have a top-level config of its own, but it will be silently ignored.
This is to support corner cases where, much like how a single board dir
can define multiple boards, a single board dir can also extend multiple
boards, or even do both. In those cases, the primary BOARD_DIR rule
should make it unambiguous which config belongs to which board, even if
it may seem counter-intuitive at first.
For concrete examples of what this means, please see the newly added
platform unit tests.
As part of these functional changes, a good chunk of logic is moved out
of `TestPlan.add_configurations()` into a new function in `platform.py`.
This is because recombining the top-level and variant configs requires
direct manipulation of the loaded YAML contents, which would be improper
to do outside of the module responsible for encapsulating this data.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
Allow user to add shell commands in testcase/sample yaml file
alongside the harness_config like in the console harness.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Extended the reason field in Twister report to include
more detailed information for 'Build failure' and
'CMake build failure'
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Introduce a new harness based on pytest that does basic shell command
handling. The harness is enabeld using:
harness: shell
and expects a file with parameters in the form:
test_shell_harness:
- command: "kernel version"
expected: "Zephyr version .*"
- ...
Multiple commands and their expected output can be tested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When using --coverage-per-instance, collect the test names in lcov, and
add --show-details to the html report to display those test names.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Instead of capturing the coverage data twice with
--coverage-per-instance, just merge the lcov files in the aggregation
report.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
When pytest is executed, it only writes logging messages to the twister
harness, and does not include information about any test assertion
failures. This information is printed to stdout, so simply write
the contents stdout to the harness log.
Remove the log file arguments previously passed to pytest, as these are
no longer needed.
Signed-off-by: Daniel DeGrasse <ddegrasse@tenstorrent.com>
In `--coverage-per-instance` mode for 'gcovr' merge individual
coverage.json reports into the aggregate code coverage report
instead of processing .gcda data files again.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
With this change, the coverage data (GCOV dump) is extracted from the
test log files after each of the test instance parallel execution,
instead of post processing all the logs at the end of the Twister run.
The new `--coverage-per-instance` mode extends Twister coverage operations
to report coverage statistics on each test instance execution individually
in addition to the default reporting mode which aggregates data to one
report with all the test instances in the current scope of the Twister run.
The split mode allows to identify precisely what amount of code coverage
each test suite provides and to analyze its contribution to the overall
test plan's coverage. Each test configuration's output directory will have
its own coverage report and data files, so the overall disk space and
the total execution time increase.
Another new `--disable-coverage-aggregation` option allows to execute
only the `--coverage-per-instance` mode when the aggregate coverage
report for the whole Twister run scope is not needed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Group Twister code coverage command line arguments for better usability.
Add more details in descriptions.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
In situations where a test won't be run because there is a gap
in the hardware support, add the gap information to the "reason"
field to aid in debugging test cases.
With this change, the reason field for the error in test case
will show the gap:
e.g. reason field now updated with gaps
"No hardware support {'usbd'} but is one of the integration platforms"
Signed-off-by: David Leach <david.leach@nxp.com>
Actually using the error count before clearing it
would probably be more useful.
ExecutionCounter's error count was set to 0
before being used to lower the done count properly.
This should fix in-progress overcounting,
where Twister claimed to execute more tests than planned.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Refactor Reporting to use custom ReportingJSONEncoder and encode
all pathlib.Path derived instances there which are possible
e.g. in 'environment.options' received from command line.
Fixes: #83823
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This commit removes all references to the `xtools` toolchain variant in the
twister scripts.
Note that the `xtools` toolchain variant has been deprecated since Zephyr
v3.3.0 and now removed.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.
Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.
Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.
Adjust Twister documentation and test suite to the above changes.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
not force host variant if environment is setting something different in
the case of posix arch.
Fixes#83766
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.
This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
using "dt_node_prop_enabled" to filter out supported platforms
1. check node existing.
2. check prop in node.
3. check the prop is True.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Pass a test failure reason text from harness to the instance
to convey more details on the failure.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Set missing TestCase statuses when a test under the BinaryHandler failed
(crashed), so remaining 'STARTED' and 'NONE' are now 'BLOCK' instead.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Pretty print XML reports with tab identation on levels for better
readability and ease text diff there as well.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
The testcase.classname properties in `twister_report.xml` and
`twister_suite_report.xml` now contain the full testsuite name
(without its path prefix) instead of just two its dot-separated
leftmost components. This way `twister_report.xml` testsuite container
has no duplicate testcase elements with the same `classname` and `name`
properties executing with --no-detailed-test-id for a project with same
testcase names in its 'sibling' suites, for example`tests/kernel/sleep`.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Milliseconds are logged as is, without leading zeros. This means that
following log content is produced, which is not aligned by columns:
12:54:27.972:INFO:root: Test message
12:54:27.992:INFO:root: Test message
12:54:28.12:INFO:root: Test message
12:54:28.32:INFO:root: Test message
12:54:28.52:INFO:root: Test message
12:54:28.73:INFO:root: Test message
12:54:28.93:INFO:root: Test message
12:54:28.113:INFO:root: Test message
12:54:28.133:INFO:root: Test message
Change milliseconds logging to include leading zeroes. This allows to print
nicely aligned log messages:
12:54:27.972:INFO:root: Test message
12:54:27.992:INFO:root: Test message
12:54:28.012:INFO:root: Test message
12:54:28.032:INFO:root: Test message
12:54:28.052:INFO:root: Test message
12:54:28.073:INFO:root: Test message
12:54:28.093:INFO:root: Test message
12:54:28.113:INFO:root: Test message
12:54:28.133:INFO:root: Test message
Additionally it is much easier to configure programs simplifying
timestamped log browsing (like in logview-mode for Emacs), as it is more
standard and does not require custom regex matching.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Pass additional args to test binary. Twister
passes it only for native_sim.
Fixes#82463
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use with statement to open stdout/stderr and resolve current warning
when running twister:
ResourceWarning: unclosed file....
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In case of unknown platforms in filters such as plaform_allow,
integration_plaforms or platform_exclude, report an error and exit.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
in CI, we may need to quit if there is any failure
to save time, so add this --quit-on-failure so that
any failure will quit the test.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Anas Nashif
Benjamin Cabé
Michael Zimmermann
Grzegorz Swiderski
Grzegorz Chwierut
Armin Kessler
Jeremy Bettis
Hake Huang
Dmitrii Golovanov
Erwan Gouriou
Daniel DeGrasse
David Leach
Lukasz Mrugala
Stephanos Ioannidis
Sylvio Alves
Pieter De Gendt
Marcin Niestroj
Artur Dobrynin