With junitparser 4.0.2, the JunitXml.fromfile() now returns a JUnitXml
instance. In order to access the junit.TestSuite instance, we must use
TestSuite.fromelem() to convert it back.
Signed-off-by: Daniel DeGrasse <ddegrasse@tenstorrent.com>
When we have skips in ctest, the overall result should not be marked as
a SKIP unless all tests have been skipped. If we have passing tests,
mark the test as PASS. Any failures/errors obviously will fail the
tests.
Also pass the skip reason from ctest to twister to give a hint about
what is going on in case of skips.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some simulators - like simics - may end up adding extraneous suffixes to
logged lines. This may cause some regex that match too much fail. This
patch fixes two such cases:
- regex to find RunID changed to only match valid hexadecimal
characters;
- regex to match start of testsuite changed to only match valid word
characters (0-9A-Za-z_).
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Extend the testcase schema to provide a way to communicate that a given
test is expected to reboot during execution. The generic harness takes
that information and suppresses `already STARTED` warnings if the
restart is expected.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Fix missing match on Ztest test case summary log entries, so even
if the test case 'END' log entry is missed or corrupted, its status
will be updated from the Ztest summary log entries, if present.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Removed logger.setLevel(...) from all twister modules, becasue
there should be only one place where twister loggers are configured,
and the log level should be set in that place.
This should be done in twister_main.py in setup_logging function.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Allow user to add shell commands in testcase/sample yaml file
alongside the harness_config like in the console harness.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Introduce a new harness based on pytest that does basic shell command
handling. The harness is enabeld using:
harness: shell
and expects a file with parameters in the form:
test_shell_harness:
- command: "kernel version"
expected: "Zephyr version .*"
- ...
Multiple commands and their expected output can be tested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When pytest is executed, it only writes logging messages to the twister
harness, and does not include information about any test assertion
failures. This information is printed to stdout, so simply write
the contents stdout to the harness log.
Remove the log file arguments previously passed to pytest, as these are
no longer needed.
Signed-off-by: Daniel DeGrasse <ddegrasse@tenstorrent.com>
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.
Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.
Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.
Adjust Twister documentation and test suite to the above changes.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Milliseconds are logged as is, without leading zeros. This means that
following log content is produced, which is not aligned by columns:
12:54:27.972:INFO:root: Test message
12:54:27.992:INFO:root: Test message
12:54:28.12:INFO:root: Test message
12:54:28.32:INFO:root: Test message
12:54:28.52:INFO:root: Test message
12:54:28.73:INFO:root: Test message
12:54:28.93:INFO:root: Test message
12:54:28.113:INFO:root: Test message
12:54:28.133:INFO:root: Test message
Change milliseconds logging to include leading zeroes. This allows to print
nicely aligned log messages:
12:54:27.972:INFO:root: Test message
12:54:27.992:INFO:root: Test message
12:54:28.012:INFO:root: Test message
12:54:28.032:INFO:root: Test message
12:54:28.052:INFO:root: Test message
12:54:28.073:INFO:root: Test message
12:54:28.093:INFO:root: Test message
12:54:28.113:INFO:root: Test message
12:54:28.133:INFO:root: Test message
Additionally it is much easier to configure programs simplifying
timestamped log browsing (like in logview-mode for Emacs), as it is more
standard and does not require custom regex matching.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Pass additional args to test binary. Twister
passes it only for native_sim.
Fixes#82463
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fixes ruff linting errors UP030, UP031, UP032,
which make you use format specifiers (fstrings)
instead of printf formatting or str.format().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error F541,
where an fstring is used when a normal
string would suffice.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error F811,
where a module is reimported or otherwise
redefined without being used.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error B904,
where an error raised inside an except
clause does not specify from where it
should start.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM102,
where multiple ifs were used instead
of combining them with and or similar
operators.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM300,
where constant values are on the left
side of an expression.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Fix a problem of Ztest suite names not taken into account by Twister
to identify a TestCase, so in some situations a Ztest test's status
was not assigned to the proper TestCase and it remains 'None'
whereas the actual status value lost, eventually the resulting total
execution counters not correct.
The issue was observed in these situations:
* Ztest application with multiple test suites having same test names.
* Ztest suite is 'skipped' entirely on execution with all its tests.
The proposed solution extends Twister test case name for Ztest to
include Ztest suite name, so the resulting identifier looks like:
`<test_scenario_name>.<ztest_suite_name>.<ztest_name>`
The above naming scheme now requires ztest_suite_name part to be
provided for `--sub-test` command line option.
Testcase identifiers in twister.json and testplan.json will also
include ztest_suite_name component.
The Twister Ztest(Test) Harness is improved to track all state changes
known from the test application's log for Ztest suites and test cases,
so now it parses log output from a Ztest application more scurpulously.
Regular expressions to match log records are extended and optimized
to compile them only once and, in some cases, fixed (suite summary).
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Python 3.12 warns that
> Testing an element's truth value will raise an exception in future
> versions. Use specific 'len(elem)' or 'elem is not None' test instead.
> if elem_ts := root.find('testsuite'):
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Status errors previously logged an error, but didn't fail the running test.
This commit changes that
and introduces a new StatusAttributeError to use there.
One test is modified so it follows proper status form.
One test for the new error has been added.
Status errors now will properly mark the Instance as ERROR
and not run TestCases as SKIP.
This necessitated some code layout changes in runner.py
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Use pytest-args parameters provided in command line after
parameters taken from yaml file. The last occurence is
considered by argparse, so parameters from command line
will be used.
Fixes: #77319
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The gtest 1.15 has slightly different output from the original one that
was used to set up the harness. With this change we support both output
formats.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
Extend pytest-args from configuration yaml file with args
from command line instead of overwriting.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Change Twister PyTest plugin's test finalizing sequence to release
the DUT it is used as the very last operation, after the Test Instance
status becomes fully updated from the execution results.
This also fix a race condition possible when pytest plugin releases
the DUT and it becomes acquired by another test while the current test
is not yet finalized completely.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Several improvements at Twister DeviceHandler when it releases
current DUT (Device Under Test):
- release the exact DUT which is used for the test instance
instead of all configured DUTs which happened to have
the same serial device configured.
- Twister PyTtest harness plugin adjustment to the above.
- additional debug logging to track DUT waiting/retain/release.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Updated the Pytest-Harness methods to capture and log error messages from
the subprocess, when pytest command fails.
Ensured that error messages are logged with `--inline-log` option
and placed in twister log reports.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
While the option to inform pytest about which platform a test is for
(platform being the board, such as mps2/an385) is available, it wasn't
being used. This patch fixes that, by building a pytest command which
includes the platform.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Incorrect status assignments were previously just warnings in logs.
Now they are elevated to errors.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.
Code guarding Twister's Properties was shortened to a
value check only.
QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This change is removing some deprecation warnings
which for some reason causing failing tests with
Python 3.12 on CI #76877.
Also, it fixes warnings from pytest like:
PytestCollectionWarning: cannot collect test class 'TestPlan'
because it has a __init__ constructor
(from: scripts/tests/twister/test_testplan.py)
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
The default base timeout for pytest is statically set by
the TwisterHarnessConfig class to be 60 seconds. However,
sometimes it takes longer than 60s before the app starts
to run, especially on emulator/simulator where it takes
quite some time to start. So pass the test timeout as
the base timeout via pytest command line argument.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The gTest harness asssumed that the lines end with the test name, but
some gTest implementations include the test duration in the line. Update
both the tests and regex to allow this and also avoid capturing
characters into the `test_name` that cannot be valid test name chars.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.
Fixes: #74563
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
Pass Twister pytest plugin's log output, as well as output from
a test image running with pytest, up to Twister log irregardless
of the current verbosity level set at Twister.
This allows to collect full test output and also ensures that
recording data embedded in the test log is passed to the Twister
pytest harness for export.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend Twister Harness recording feature to allow selected data fields,
extracted from the log by a regular expression, to be parsed into JSON
objects and eventually reported in `twister.json` as `recording` list
property of the test suite.
With this extension, log records can convey layered data structures
passed from a test image as summary results, traces, statistics, etc.
This extension also allows flexible recording structure: a test image
can output different types of data records incapsulated into a fixed
set of fields, so `recording.csv` file columns are respected, whereas
some of the columns keep strings with json-encoded semi-structured data.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Daniel DeGrasse
Anas Nashif
Ederson de Souza
Jordan Yates
Dmitrii Golovanov
Arkadiusz Cholewinski
Lukasz Fundakowski
Benjamin Cabé
Grzegorz Chwierut
Pieter De Gendt
Marcin Niestroj
Lukasz Mrugala
Wilfried Chauveau
Yuval Peress
Daniel Leung
Gerson Fernando Budke