For some boards, like up_squared and ehl_crb, they need do a
power off/on operation and use bios to load and execute zephyr
test binary, during the power off, it may meet this SerialException,
but the serial object still works well after power on, we can continue
to get all results from serial, so we don't need to close it,
it's ok to ignore this exception.
BTW, even if we ignore this exception here, it doesn't influence twister
function, because if it was really caused by disconnection, the test would
still be judged as failed due to timeout and serial would be closed later
in main thread, it doesn't impact results.
On the contrary, close serial directly in the monitor serial thread, it
will cause later tests failed due to mismatch errors.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
(cherry picked from commit ca08d2e4a6)
When testcase written in ztest is skipped by ztest_test_skip() function,
then "SKIP" information is printed in output data. This "SKIP" result is
set as result of performed testcase, but the information about reason is
still "Unknown". This patch fix this problem and if all test scenario
has state "passed" (so testcases have result "PASS" or "SKIP"), reason
is still set as "Unknown" and if any of testcase has result "SKIP" then
set reason as "ztest skip".
Fixes#42414
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
sometimes there are some unencodable unicode chars from output
of west flash command, need to ignore them rather than reporting
an error.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
This reports the warning fixed in commit 61fe69dded ("sys:
device_mmio.h replace <toolchain/common.h> with <toolchain.h>")
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Testing shows cmake is invoked directly, not through a shell.
Fixes commit 5092541e2b ("sanitycheck: disable erroring on warnings").
Note EXTRA_AFLAGS was quoted only on the left side!
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Changes will raise error and break program in following three situations:
1. Platform name pass in --platform option does not exist.
2. During using --all option, platform from platform_allow list does not
exist.
3. During using --integration option, platform from
integration_platforms list does not exist.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
When some testcase is skip in their source code by ztest_test_skip()
function in final json report it is marked as pass what is wrong. Due to
the changes in this commit those skips can be marked properly.
Fixes: #41440
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
When generating new map we store serial_baud, but the schema has 'baud'
only, so call this baud in the class and resolve the issue of creating
new maps.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Changes will log error message in following three situations:
1. Platform name pass in --platform option does not exist.
2. During using --all option, platform from platform_allow list does not
exist.
3. During using --integration option, platform from
integration_platforms list does not exist.
Fixes#31868
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
In order to enable use of stm32cubeprogrammer runner with twister,
add "sn" tool specific option which allows to provide target serial
number and hence select the target to flash when multiple ones are
connected to the host.
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
The -DCMAKE_EXPORT_COMPILE_COMMANDS=1 is removed from twisterlib.py as
it is now always set by the Zephyr build system.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Fix few limitation which do not allow to run Twister on Windows OS.
Especially turn off "import pty" module, due to the fact, that this
module can be used only on POSIX OS.
Fixes#40007
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Introduce a weak implementation of test_main() which calls:
* ztest_run_registered_test_suites(NULL);
* ztest_verify_all_registered_test_suites_ran();
This will attempt to run all registered test suites and verify that
they each ran.
Signed-off-by: Yuval Peress <peress@chromium.org>
Add new functionality to ztest to improve test modularity. The two
primary new entry points are:
* ztest_register_test_suite
* ztest_run_registered_test_suites
When registering a new test suite, users provide the name as well as
an optional predicate used to filter the tests for each run. Using NULL
as the predicate ensures that the test is run exactly once (after which
it is automatically filtered from future runs).
Calls to ztest_run_registered_test_suites take a state pointer as an
argument. This allows the the pragma functions to decide whether the
test should be run.
The biggest benefit of this system (other than the ability to filter
tests and maintain a larger test state) is the ability to better
modularize the test source code. Instead of all the various tests
having to coordinate and the main function having to know which tests
to run, each source file manages registering its own test
suite and handling the conditions for running the suite.
Signed-off-by: Yuval Peress <peress@chromium.org>
Fixes a bug causing tests with error status get silently skipped
when --subset is used.
Fixes: #39619
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fixes#38046
Document the changes in twister.rst
Add baud rate to debug message in twister
Add baud parameter to twister's add_device function
Set the twister baud rate from input parameters
Use 115200 as the default baud rate if not specified
Add baud to the hwmap-schema.yaml file
Add --device-serial-baud to twister arguments
Fix compliance issues
Fix mistake in parameter name from device-baud to device-serial-baud
Refactoring of the code in orded to simplify the logic
and clear multiple definitions of the default baud rate.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Signed-off-by: Dennis Ruffer <daruffer@gmail.com>
If timeout error was occured during performing testsuite on QEMU or
other simulator (e.g. Renode), information about this error will be not
placed in final report. It can be fixed, by set status of testcases,
which were not performed due to this error as "BLOCK".
Simmilar solution is alredy used in DeviceHandler when Twister works
with real platform. So I think that it is resonable to use the same
approach, when tests are performed on simulators.
I move loop which iterate through all testcases and set their tests
status as BLOCK into the parent Handler class, because the same
activity is performed in each children class (BinaryHandler,
DeviceHandler and QEMUHandler).
Fixes#38756
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Fixes: #38924
When the `verify-toolchain.cmake` script fails, then twister will print
a standard message to the user regardless of the reason.
> E: Variable ZEPHYR_TOOLCHAIN_VARIANT is not defined
The `verify-toolchain.cmake` already prints detailed information
regarding the cause of the failure, so twister should just pass that
message as-is.
For example, the message that is provided by verify-toolchain.cmake
when Zephyr SDK 0.13.0 is installed but 0.13.1 is required looks like:
> CMake Error at cmake/verify-toolchain.cmake:75 (find_package):
> Could not find a configuration file for package "Zephyr-sdk" that is
> compatible with requested version "0.13.1".
>
> The following configuration files were considered but not accepted:
>
> /opt/zephyr-sdk-0.13.0/cmake/Zephyr-sdkConfig.cmake, version: 0.13.0
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Tests are reported as skipped if they are only being built. Fix this by
checking for the correct status.
Fixes#37475
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In the case of native_posix (and other "host" toolchain based
platforms) Twister shouldn't filter tests by ZEPHYR_TOOLCHAIN_VARIANT.
The PR replaces `- zephyr` with `- host` in native_posix(_64).yaml
and modify the filter in twisterlib.py to not block building
if "host" is under the platform supported toolchains.
Fixes: #38418
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
For twister, when simulation is mdb-nsim, the platform is
nsim_hs_smp. Before the twister will call west falsh
when platform is nsim_hs_smp, because twister can't kill
cld process, now this problem has been fixed
Signed-off-by: Jingru Wang <jingru@synopsys.com>
When run by a makefile twister might fail to decode the text results of
commands, based upon the environment settings of the system. With this
change escape characters are removed before the string is passed to the
json decoder.
Signed-off-by: Benedikt Schmidt <benedikt.schmidt@embedded-solutions.at>
When using both test-only and retry-failed flags, all tests would be
added to the test queue regardless if they failed or not. This commit
updates the logic to process test-only runs similar as other runs,
by adding tests to the queue with the 'run' operation directly.
Signed-off-by: Andreas Vibeto <andreas.vibeto@nordicsemi.no>
Create a "global" gen_defines.py option and edtlib.EDT constructor
kwarg that turns edtlib-specific warnings into errors. This applies to
edtlib-specific warnings only. Warnings that are just dupes of dtc
warnings are not affected.
Use it from twister to increase DT testing coverage in upstream zephyr.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make correction on the terms "intput" appearing
in sciripts/pylib/twister/twisterlib.py to "input" for all.
This type of misspellings may cause another unnoticeable misspellings,
when propagated by code completions.
(assuming that the "intput" was intended
to have the opposite meaning of "output",
which also appear on the same areas)
Signed-off-by: Kentaro Sugimoto <tarotene@gmail.com>
This function takes a node label (not a label property) and returns
true if a node with such label exists and the node is enabled.
Signed-off-by: Andrzej Głąbek <andrzej.glabek@nordicsemi.no>
When integration mode is on, any skips on integration_platforms are
treated as errors. This patch adds an exemption for quarantined tests.
They will stay as skipped.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If "Cmake build failure" is detected test instance get status "error".
Despite this in final report this error is counted as failure. It can
be fix, by set proper results of each testcase in instance from None
to "BLOCK" after found Cmake error. After this fix, error is counted
properly in final report.
Fixes#37140
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
In dt_label_with_parent_compat_enabled, it maybe that there is
no node matching expected label.
In this case don't generate error, but return False.
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
sometimes ninja fails to launch qemu, the sub-thread
can't read anything from qemu pipeline, then the
corresponding testcase will timeout. Then the
sub-thread will get blocked if it call join()
set terminate as the Handler's method, then
Handler's children class can call it
Signed-off-by: Jingru Wang <jingru@synopsys.com>
Function used for filtering "dt_compat_enabled_with_label" was not
working as expected as it was not taking into account that we're
looking for a children/parent combination:
Provided "compat" with enabled status should be the one of the parent
of the node matching given label.
Function is then reworked to take this into account.
And to make it's usage clear:
- function name is changed to be clearer on the intention
- args order is reversed to be more logical wrt the intention
Users of the function are also updated to take the change into
account.
Fixes#36093
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
when generating a new map, set the status of the device and whether it
is connected or now. The available key is runtime only and is not part
of the map anymore.
Adapt documentation and remove available as a key in the generated map.
Fixes#35341
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
On ARC some platforms utilize ICCM/DCCM for their "flash" and "ram"
storage. So we might get errors about overflowing one of these regions.
So treat them similar to how we treat FLASH and SRAM.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
1. Add a class of Pytest(Harness) which can handle pytest script
in harness.py
2. Use running_dir to store current test directory which be
used in pytest scrpt.
3. Add usage of this feature into zephyr doc.
Signed-off-by: YouhuaX Zhu <youhuax.zhu@intel.com>
All skips for platforms listed in the .yaml's integration_platforms
will be treated as errors in the integration mode ('-G/--integration').
This feature serves to guarantee a proper testing scope in CI as no
integration platform will be skipped silently.
Fixes#33874
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
With the previous logic memory footprint can only be saved in a report
if a given test/sample was fully executed and passed (built and run),
hence build-only tests were not providing these metrics. This commit
modifies the logic so that it is enough to have the build successful
to be able to get the memory footprint.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Upon discovery of a sample/testcase.yaml file, twister will disregard
further subfolders. Given the folder structure below, the testcase.yaml
will currently not be discovered:
samples/sample1/
samples/sample1/sample.yaml
samples/sample1/tests/
samples/sample1/tests/testcase.yaml
This is desirable to allow placing test code closer to sample code.
Signed-off-by: Jorgen Kvalvaag <jorgen.kvalvaag@nordicsemi.no>
Add arm fvp emulator in order to use ninja run or west build -t run.
Add armfvp in order to run twister.
Set env ARMFVP_BIN_PATH before using it,
e.g. export ARMFVP_BIN_PATH=<path/to/fvp/dir>
NOTE: ARMFVP_BIN_PATH is the dir path.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Adds feature allowing to use yaml file with dictionaries defining
tests to be quarantined (extra arg "--quarantine-list FILENAME").
The dictionaries are validated according to the proper schema
and loaded.
A flat list is created containing quarantined configurations
(configuration = platform + scenario). Configurations under quarantine
are skipped and get "Quarantine" as a reason in the results reports.
A "comment" can be added to a quarantine entry in the quarantine yaml
with more details (e.g. issue #) and it will be also added to
the report.
The status of tests under quarantine can be verify if
`--quarantine-verify` is used in addition to
"--quarantine-list FILENAME". Using these args will make twister skip
all tests which are not on the quarantine list.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In CI we typically build for "default" platforms. However some
testcases that set "platform_allow" may have no overlap with the default
platforms and thus thoses testcases will not get built.
Change the platform selection logic in these cases to set the platforms
to the list of allowed platforms ("platform_allow") in the testcase.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
We are now in the process of extracting edtlib and dtlib into a
standalone source code library that we intend to share with other
projects.
Links related to the work making this standalone:
https://pypi.org/project/devicetree/https://python-devicetree.readthedocs.io/en/latest/https://github.com/zephyrproject-rtos/python-devicetree
This standalone repo includes the same features as what we have in
Zephyr, but in its own 'devicetree' python package with PyPI
integration, etc.
To avoid making this a hard fork, move the code that's being made
standalone around in Zephyr into a new scripts/dts/python-devicetree
subdirectory, and handle the package and sys.path changes in the
various places in the tree that use it.
From now on, it will be possible to update the standalone repository
by just recursively copying scripts/dts/python-devicetree's contents
into it and committing the results.
This is an interim step; do NOT 'pip install devicetree' yet.
The code in the zephyr repository is still the canonical location.
(In the long term, people will get the devicetree package from PyPI
just like they do the 'yaml' package today, but that won't happen for
the foreseeable future.)
This commit is purely intended to avoid a hard fork for the standalone
code, and no functional changes besides the package structure and
location of the code itself are expected.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Remove dead code from twisterlib.py which is a copy of code
from twister script. This redundancy causes pylint errors in
compliance checks.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The function `handle()` from class `Test(Harness)` had most of its
code being the direct copy of the code from
`Harness.process_test(line)`. This patch replaces the copied code
with the call for `process_test(line)` and removes repetition of
this function call.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Use edt.label2node in dt_compat_enabled_with_label DT filter.
Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
Co-authored-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The patch improves handling runners errors from flashing. It
add thread termination in case of process returning non 0 exit
code. Before, even though the error was returned,
a thread was not terminated and test ended
only after timeout termination. The patch also adds information about
the failure reason to the reports
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The function returns true if the node with the referenced
node label is compatible and enabled.
Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
Chen Peng1
Piotr Golyzniak
Marc Herbert
Anas Nashif
Erwan Gouriou
Torsten Rasmussen
Yuval Peress
Maciej Perkowski
Dennis Ruffer
Henrik Brix Andersen
Jingru Wang
Benedikt Schmidt
Andreas Vibeto
Martí Bolívar
Kentaro Sugimoto
Andrzej Głąbek
Kumar Gala
YouhuaX Zhu
Jorgen Kvalvaag
Jaxson Han
Johann Fischer