To make possible to build bsim tests by Twister, it is necessary to
copy executables to BabbleSim bin directory.
Signed-off-by: Piotr Golyzniak <metody159@gmail.com>
Twister DeviceHandler now checks `--enable-coverage` command line argument
instead of `--coverage` when it deals with device output.
This resolves potential problem when only `--enable-coverage` argument
is given and the coverage report is not needed. In this case the test image
which is built for code coverage works slower also producing additional
console output, so the additional DeviceHandler timeout still have to be
applied and the output with coverage data correctly processed by Harness.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
lcov 2.0 added support for processing coverage data in parallel, which
provides a large speedup when processing many files, at the cost of some
additional overhead. When running the Chrome EC tests with coverage,
parallel reporting on a 36C72T machine reduces the time spent generating
coverage reports by 40 minutes (from approximately 1 hour to 20 minutes
total runtime), at the cost of about 3x greater CPU time overall
(assumed to be overhead for parallel processing, likely from spawning
much larger numbers of subprocesses).
The level of lcov parallelism is taken from the --jobs option passed to
twister, allowing lcov to choose if unspecified.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
This moves handling of changed lcov arguments in lcov 2.0 into helper
functions, significantly simplifying Lcov._generate() to reduce the
visual noise of extra arguments that are required but unimportant to the
task at hand.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
Make sure we set the gcov tool in a consistent way and avoid issues
where path is set as Path instead of a string.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
BinaryHandler.hanlde() method was trying to call GCOV and fails
silently because of incorrect call parameters. Moreover, even
being fixed, this call is not needed here as it attempts to
find and process .gcno and .gcna files (using wrong paths)
to create .gcov coverage text reports.
Currently the CoverageTool class does all data processing
and report generation using GCOVR or LCOV tools instead of
direct calls to GCOV.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Verify we have the coverage tool we want to use, otherwise we will end
up with many warnings and errors during coverage data collection.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Additional checks for Twister command line options `--coverage-tool`
and `--coverage-formats`.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister now uses GCOVR by default as the more reliable code
coverage reporting tool instead of LCOV.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When scanning ports the manufacture field is not always
filled. It must be checked before using.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In some cases genhtml incorrectly built reports from LCOV
coverage data using full path for some of the source files
and relative paths for other files.
This fix adds `--prefix` parameter to shorten paths explicitly
relative to the ZEPHYR_BASE directory.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This gives us the flexibility to exclude certain platforms'
simulator that's known to fail from running in Twister, but
allows real hardware to be tested.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
NXP boards with CMSID-DAP are not detected by twister
--generate-hardware-map, because serial device name 'mbed' is compared
with upper case 'MBED' in a list of supported manufacturers.
Fix it by making the comparison case-insensitive.
Tested using mimxrt1020_evk.
Fixes#63765
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Fixes#65477
Platform key checking seemed to be erroneous;
now the variable names, comments and code seem in line with each other.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Support lcov > 2.0 tool which has strict error checking and some new
configuration options deprecating syntax used in 1.4 versions.
Fixes#62202
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
as in windows platfrom the format by default is not utf-8,
and we will see below error
Traceback (most recent call last):
File "...zephyr\scripts\twister", line 211, in <module>
ret = main(options)
^^^^^^^^^^^^^
File "...scripts/pylib/twister\twisterlib\twister_main.py",
tplan.load()
File "...scripts/pylib/twister\twisterlib\testplan.py",
self.load_from_file(last_run, filter_platform=connected_list)
File "...scripts/pylib/twister\twisterlib\testplan.py",
instance.create_overlay(platform, self.options.enable_asan,
self.options.enable_ubsan, self.options.enable_coverage,
self.options.coverage_platform)
File "...scripts/pylib/twister\twisterlib\testinstance.py"
f.write(content)
UnicodeEncodeError: 'gbk' codec can't encode character '\xf8'
in position 64: illegal multibyte sequence
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
The Console Harness is able to parse its log with patterns to compose
extracted fields into records in 'recording.csv' file in the test's build
directory. This feature allows to extract custom test results like
performance counters.
With this change the extracted records are also written into 'twister.json'
as a part of each test suite object. This makes easier to store
all the data collected by the test for its further processing.
Other improvements:
- compile parsing pattern only once instead of at each input line;
- quote fields in '.csv' to avoid unexpected field separators;
- make 'regex' a required schema field of 'harness_config';
- Twister documentation update.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
A race condition was possible at QEMUHandler which didn't wait for
its harness to complete causing false negatives and test suite
retries, for example when the console harness has got rather
long output from a test application to check for patterns.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Using --platform-reports will also generate json reports
with results from a single platform. Needed for on-target
results publishing process.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
One can not even replace sucessfully pytest basic sample `pytest-args`
with command line "--pytest-args", as all it does is to append a single
string to current list of commands, making it impossible to send several
arguments.
This patch fixes that by allowing several instances of `--pytest-args`
to compose the whole list of arguments to be passed to pytest.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
If a platform is fitlered, do not add it to keyed test map, otherwise we
will end up skipping all platforms from the same class without any
coverage.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Useful option during development and when applying smaller change to the
code and verifying changes using tests. This works fine as long as no
dependencies or major changes are done, i.e. when editing few files and
in a subsystem. Use with caution and use on your own risk :-)
When building multiple tests, this provide significant boost, up to 300%
faster than when rebuilding everything from scratch or when re-running
cmake every single time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Measure both cmake and make/ninja build times and log them in debug mode
and additionally put the result in the json report for tracking build
times.
Cleanup what build/cmake function return and remove unused keys in the
result map.
Remove some excessive logging of launched jobs or default platforms that
gets in the way when in debug mode.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend Twister command line with --pytest-args. This parameter
is passed to pytest subprocess. It allows to select a specific
testcase from a test suite.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
On MacOS, ps utils raises a `NoSuchProcess` error rather than a
`ProcessLookupError` when a pid no longer exists.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Implement a workaround for Console harness to compose TestCase
identifier correctly when a Ztest suite with a single testcase
uses this harness type. Normally, a Ztest suite should use the
Ztest Twister harness.
Without this workaround each Ztest TestCase result on Console is
duplicated (and written into twister.json) with its 'identifier'
attribute set to TestSuite id only, no TestCase suffix added;
the resulting entry with the full TestCase id is also stored,
but its values are empty with defaults.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
If Console Harness 'harness_config' properties have no 'regex'
patterns or no correct 'type' set, then ConfigurationError exception
is raised, handled, and the test instance error is accounted
in the summary results.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix the Twister Console harness unordered pattern matching
to treat the ztest as failed when not all of the
expected patterns were found in the console output, but the ztest
application itself reports 'PROJECT EXECUTION SUCCESSFUL'.
Unify debug logging on pattern match for ordered and unordered patterns.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
* Replace native_posix references with native_sim
in arguments help messages
* For the seed parameter, correct platform check to accept
native_sim
* Use native_sim in twister tests
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Create copys of the common configuration values when constructing test
scenarios, to avoid mutating the common value with test specific
appends.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
when load test plan it is possible the plan is built in another os
so the case key would be
'samples/hello_world/samples.baseic.hello_world'
but the testsuite is scaned in current os may in window
and the key is like
'samples\\hello_world\\samples.baseic.hello_world'
so update the uniq path with only backslash in path
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Some platforms prefix extra logging information before the standard
[] blocks so I've added `.\*` to the regex. Also, removed the static
values so they're only referenced using 'self.' and stopped parsing
lines after the FINISHED_PATTERN is matched since some versions of
gTest also print out a test summary after and it's not useful for
the processing.
Signed-off-by: Yuval Peress <peress@google.com>
When pytest scenario fails, then 'handler.log' is printed.
Changed to print 'twister_harness.log' that is a log from
pytest execution. That file tells much more when test fails.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Apply a workaround for an issue introduced with bc8b7dd
for code coverage collection at qemu_x86 when gcov data
was broken because of optimized out memcpy() in write_u64()
static inline function, so gcovr parser fails with
"gcovr.gcov_parser.NegativeHits: Got negative hit value
in gcov line 'branch 1 taken -1'".
Add gcovr command line option to ignore such errors (since v.6.0):
"--gcov-ignore-parse-errors=negative_hits.warn_once_per_file"
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added pytest_dut_scope keyword under harness_config section.
New keyword is used to determine the scope of dut and shell
fixtures in pytest-twister-harness plugin.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Do not break processing when incorrect coverage data are received.
Instead, report failures but still process valid data.
Include coverage processing status within twister retrun code.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
When coverage is enabled, gather console logs longer
to receive gcov data which are send after test output.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
Very verbose log message without too much context that should be a debug
message rather than a info().
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Piotr Golyzniak
Dmitrii Golovanov
Peter Marheine
Anas Nashif
Keith Short
Grzegorz Chwierut
Yong Cong Sin
Piotr Kosycarz
Michal Smola
Lukasz Mrugala
Maciej Perkowski
Hake Huang
Ederson de Souza
Wilfried Chauveau
Alberto Escolar Piedras
Jordan Yates
Yuval Peress
Øyvind Rønningstad