When running purely with Python, the extra modules set with environment
variables are not taken into account.
If left empty, check the environment variables.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
We now return "unknown" as the revision, and only concat onto
the revision if the revision is a valid git has value.
Signed-off-by: Andy Lee <andy@chiefmarley.com>
Add runners entry to the module schema and import the file specified.
Every class that inherits from ZephyrBinaryRunner will be discovered and
added to runners list.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Add a new west command as helper for package management of Zephyr
and modules.
The first package manager to get supported is pip, where west projects
can list individual packages or requirements files in their module file.
A convenience --install argument is added to install the packages instead.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
For any module that defines blobs, add a new Kconfig symbol to indicate
whether the blobs have been fetched or not. Example output for the
hal_silabs module:
# (no blobs present)
$ scripts/zephyr_module.py --kconfig-out=/dev/stdout \
-m ../modules/hal/silabs
menu "hal_silabs (../modules/hal/silabs)"
osource "/Users/johedber/src/zephyr/modules/hal/silabs/zephyr/Kconfig"
config ZEPHYR_HAL_SILABS_MODULE
bool
default y
config ZEPHYR_HAL_SILABS_MODULE_BLOBS
bool
endmenu
$ west blobs fetch hal_silabs
# (blob fetching output)
$ scripts/zephyr_module.py --kconfig-out=/dev/stdout \
-m ../modules/hal/silabs
menu "hal_silabs (../modules/hal/silabs)"
osource "/Users/johedber/src/zephyr/modules/hal/silabs/zephyr/Kconfig"
config ZEPHYR_HAL_SILABS_MODULE
bool
default y
select TAINT_BLOBS
config ZEPHYR_HAL_SILABS_MODULE_BLOBS
bool
default y
endmenu
The generated output for modules which do not define blobs is not affected.
Having this additional symbol for blobs lets us specify Kconfig
dependencies for features which require the blobs to be present.
Signed-off-by: Johan Hedberg <johan.hedberg@silabs.com>
Improve the SPDX with the current values:
- URL: extracted from `git remote`. If more than one remote, URL is not
set.
- Version: extracted from `git rev-parse` (commit id).
- PURL and CPE for Zephyr: generated from URL and version.
For zephyr, the tag is extracted, if present, and replace the commit id for
the version field.
Since official modules does not have tags, tags are not yet extracted for
modules.
To track vulnerabilities from modules dependencies, a new SBOM,
`modules-deps.spdx` was created. It contains the `external-references`
provided by the modules. It allows to easily track vulnerabilities from
these external dependencies.
Signed-off-by: Thomas Gagneret <thomas.gagneret@hexploy.com>
Fixes an issue with zephyr modules not being listed in sysbuild if
they did not have a Kconfig file set
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
Define the ZEPHYR_{MODULE_NAM}_MODULE option for all available modules.
Right now only modules defined as 'kconfig-ext' has this symbol defined
which may cause problem if the build wants to check if a module not
using this options is available.
This allow us to do something like:
comment "mbedTLS module not available."
depends on !ZEPHYR_MBEDTLS_MODULE
or make a feature depends on the module be present.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Currently, this script will blow up when given a symlink for a path (e.g.,
to a kconfig file).
This is annoying, as when wrapping the build system in a hermetic build
system like Bazel, Bazel likes to limit the input file set by creating
symlinks.
Resolve the path, if it is a file once resolved, then it's OK.
Signed-off-by: Jack Rosenthal <jrosenth@chromium.org>
Commit ce2a7d9a1a
("scripts: zephyr_module: handle expected west errors")
introduced some better error handling that requires
west version v0.14.0 or later to work.
Bump the west version in requirements-base.txt accordingly.
Due to the way zephyr_module.py is handling imports, this API change
resulted in zephyr_module.py running on older versions of west
reacting as if west was not installed, instead of erroring out.
Fix that so users who are on older west will get a hard error.
(We're about to force everyone to move to west v1.1 as soon as I can
get that release done, but this hotfix should still be helpful in the
interim as well.)
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Errors can occur if the user's workspace is not set up properly.
Output a better error message instead of dumping stack in these cases.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This key may appear in a module.yml's 'settings' map, and works
analogously to the already existing 'board_root', 'dts_root', etc.
keys: if set, its value is interpreted as a directory that will be
added to SNIPPET ROOT.
For example, if a module.yml contains:
settings:
snippet_root: foo
Then 'foo', relative to the module's base directory, will be added to
SNIPPET_ROOT. This then means that any snippets in foo/snippets will
be discovered by the build system.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This commit extends the Zephyr module yaml scheme with additional
entries for sysbuild in the build section.
This allows for Zephyr modules to extend the sysbuild infrastructure
by providing additional CMake and Kconfig files to be included in
sysbuild.
The new settings are:
build:
sysbuild-cmake: <path>
sysbuild-kconfig: <path>/<file>
sysbuild-ext: <true>|<false>
sysbuild-kconfig-ext: <true>|<false>
those settings follow the same pattern as the equivalent Zephyr build
settings but are processed by sysbuild.
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Static code analyser (SCA) tools are important in software development.
CMake offers built-in support for some tools, such as cppcheck and
clang-tidy.
Other tools, such as sparse, are not directly supported.
This commit provides a uniform way for users to specify a supported
SCA using `ZEPHYR_SCA_VARIANT=<tool>` which is consistent with how
toolchains are specified.
ZEPHYR_SCA_VARIANT can be set using `-D` or in environment.
Support for an SCA tool is done in `cmake/sca/<tool>/sca.cmake`.
SCA_ROOT can be used to specify additional search paths when looking up
implementation for a tool. SCA_ROOT can also be specified in
`zephyr/module.yml` as setting. This makes it possible to provide SCA
tool implementation as part of a Zephyr module.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Disables allowing the python argparse library from automatically
shortening command line arguments, this prevents issues whereby
a new command is added and code that wrongly uses the shortened
command of an existing argument which is the same as the new
command being added will silently change script behaviour.
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
Whenever binary blobs are present in the filesystem (i.e. fetched,
regardless of whether they are up-to-date or not) we consider the Zephyr
build as tainted. In order to ensure that it is marked as so, add a
Kconfig `select` statement whenever this is the case.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
In preparation for the handling of taint flags in zephyr_module.py, move
blob-processing code from the west command to it.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
This patch introduces the basic infrastructure to list and fetch binary
blobs. This includes:
- The new 'blobs' extension command
- An implementation of the `west blobs list` command
with custom formatting
- A very simple mechanism for loading fetchers
- A basic implementation of an HTTP fetcher
In order to ensure consistency among the west extension commands in the
main zephyr tree, we reuse a similar class factory pattern that is present
for ZephyrBinaryRunner instances in the ZephyrBlobFetcher case. This
could be achieved with a simpler mechanism, but opted for consistency
before simplicity.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Instead of using strings, use pathlib's functionality to handle the
fact that it can have two different extensions.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
The boards command was not properly using the zephyr_module
functionality to obtain the board roots of all modules. Fix that by
moving the functionality required to the core zephyr_module file and
reuse it from external scripts.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Twister should be partially usable even without West installed, however,
Twister unconditionally tries to import it when initializing to check
for west projects and crashes if it is not there.
This PR changes the behavior to return None if the west imports failed,
which appears to be more in line with what was intended. This allows
Twister to proceed and successfully run tests in a non-west environment.
There is no impact on environments that do have west installed.
Fixes#45355
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Although CMake scripts already support duplicated module
names, if two modules with the same name provide different
Kconfig files, then both files will be loaded leading to
potential conflicts.
Modify zephyr_module.py to enforce that all modules are
uniquely named so that it is possbile to override some of
the built-in modules using ZEPHYR_EXTRA_MODULES variable.
Signed-off-by: Damian Krolik <damian.krolik@nordicsemi.no>
The PR #39382 raised a discussion on build reproducibility and knowledge
of west projects being out of sync with the west manifest.
Similar to how `git submodules` will report the working tree dirty if
any of the submodules HEAD points to a SHA different than the one
recorded in the super project.
Based on this discussion this commit extends the Zephyr and manifest
repo (when `west` is used) revisions to include the concatenated states
of the workspace.
The workspace states are:
> dirty: false / true
> extra: false / true
> off: false / true
the final revision will become: <SHA>-<state1>-<state2>-<state3>
For example:
> zephyr:
> path: /.../zephyr
> revision: <SHA>-dirty-extra-off
or
> zephyr:
> path: /.../zephyr
> revision: <SHA>-extra
The `BUILD_OUTPUT_META_STATE_PROPAGATE` Kconfig setting is introduced
to provide user control of this behavior.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The PR #39382 raised a discussion on build reproducibility and knowledge
of west projects being out of sync with the west manifest.
Similar to how `git submodules` will report the working tree dirty if
any of the submodules HEAD points to a SHA different than the one
recorded in the super project.
Based on this discussion this commit extends the Zephyr build meta file
with overall workspace status included in the meta file.
It adds the following meta info to the build meta file:
> workspace:
> dirty: false / true
> extra: false / true
> off: false / true
A project using west and having an extra Zephyr module loaded not
controlled using git and a west project at a SHA different than the
SHA referenced by the manifest can look like:
zephyr:
path: /.../zephyr
revision: 863600cd0e
modules:
- name: mcuboot
path: /.../bootloader/mcuboot
revision: c61538748ead773ea75a551a7beee299228bdcaf
- name: local_module
path: /.../local_module
revision: null
west:
manifest: /.../zephyr/west.yml
projects:
- path: /.../zephyr
revision: 863600cd0e
- path: /.../bootloader/mcuboot
revision: c61538748ead773ea75a551a7beee299228bdcaf-off
- path: /.../tools/net-tools
revision: f49bd1354616fae4093bf36e5eaee43c51a55127
workspace:
dirty: false
extra: true
'off': true
And without west:
zephyr:
path: /.../zephyr
revision: 863600cd0e
modules:
- name: hal_nordic
path: /.../modules/hal/nordic
revision: a6e5299041f152da5ae0ab17b2e44e088bb96d6d
- name: local_module
path: /.../local_module
revision: null
workspace:
dirty: false
extra: true
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
kconfiglib doesn't render backslashes in menu strings, and the prompts
in the module menu will be unintelligible on Windows. Print these
menus as posix paths on Windows as well, so they don't look broken in
menuconfig.
Signed-off-by: Trond Einar Snekvik <Trond.Einar.Snekvik@nordicsemi.no>
This commit is motivated by the west discussion in:
https://github.com/zephyrproject-rtos/west/issues/548
The commit provides the ability to generate a build meta info file
containing lists of:
- Zephyr: path and revision
- Zephyr modules: name, path, and revision
- West: manifest path
path and revision for each project
For Zephyr or Zephyr modules the revision will be `null` if it is not
under git version control.
If Zephyr, a modules, or a project has uncommitted changes, the revision
will be marked dirty.
If west is not installed or used for the build process, the
west-projects list will be empty.
If a project is both a Zephyr module and a west project it will show up
in both lists.
Similar to Zephyr, which is independently referred as the Zephyr in use
but also listed as west project when west is used.
This is important in case ZEPHYR_BASE was manually set and pointing to
a different Zephyr repository.
The build meta file is not created per default but can be enabled with
the BUILD_OUTPUT_META Kconfig setting.
A project using west and having an extra Zephyr module loaded not
controlled using git can look like:
zephyr:
path: /.../zephyr
revision: 863600cd0e3c0a271e86629c5089821e5e4380cc-dirty
modules:
- name: mcuboot
path: /.../bootloader/mcuboot
revision: c61538748ead773ea75a551a7beee299228bdcaf
- name: local_module
path: /.../local_module
revision: null
west:
manifest: /.../zephyr/west.yml
projects:
- path: /.../zephyr
revision: 863600cd0e3c0a271e86629c5089821e5e4380cc-dirty
- path: /.../bootloader/mcuboot
revision: c61538748ead773ea75a551a7beee299228bdcaf
- path: /.../tools/net-tools
revision: f49bd1354616fae4093bf36e5eaee43c51a55127
And without west:
zephyr:
path: /.../zephyr
revision: 863600cd0e3c0a271e86629c5089821e5e4380cc-dirty
modules:
- name: hal_nordic
path: /.../modules/hal/nordic
revision: a6e5299041f152da5ae0ab17b2e44e088bb96d6d
west: null
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
I saw a user wondering what this is for and and where it comes from.
Provide a comment header with some explanation and a pointer to where
to find out more.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add parse_modules() function, which will offload most of the work in
main() and additionally allow external Python code to use that function.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
West has introduced support for group tags in:
https://github.com/zephyrproject-rtos/west/pull/454
This means that manifest files might start containing groups.
Zephyr itself only requires west>=0.7.2 where groups are not supported
but other Zephyr based projects might start using the group feature.
When using a west version with group support, then only active projects
will be processed as Zephyr modules.
West versions without group support will consider all projects active.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The introduction of Zephyr module glue code in the Zephyr repository
introduces a Kconfig variable in the form of:
`config ZEPHYR_<MODULE_NAME>_MODULE`.
All Kconfig variables go into `autoconf.h`, therefore it is necessary
to sanitize the Kconfig variable, so that it does not contain special
characters. To ensure consistent variable name, then the module name
will be sanitized in all variable use in both Kconfig and CMake.
The sanitization is done be replacing all special characters with an
underscore, `_`.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The folder name of a Zephyr module is also used as its module name
when integrating it into the build system.
This means that a Zephyr module, BAR, located in:
<workspace>/modules/foo
|--- zephyr
|--- CMakeLists.txt
|--- Kconfig
will be referred to as FOO in the build system, that is:
ZEPHYR_FOO_MODULE_DIR==<workspace>/modules/foo
ZEPHYR_FOO_CMAKE_DIR==<workspace>/modules/foo/zephyr
The `name` field allows the module to specify its module name,
independent of its location like:
<workspace>/modules/foo/zephyr/module.yml
```
name: bar
build:
cmake: zephyr
```
will instead be referred to as BAR in the build system, that is:
ZEPHYR_BAR_MODULE_DIR==<workspace>/modules/foo
ZEPHYR_BAR_CMAKE_DIR==<workspace>/modules/foo/zephyr
This allows for greater flexibility of relocating Zephyr modules in
other folders and at the same time be guaranteed that other modules
depending on `ZEPHYR_BAR_MODULE_DIR` is still working.
If `name` field is not specified in `module.yml`, then the existing
behavior of using the folder name will be used.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This commit introduces MODULE_EXT_ROOT which allows CMake and Kconfig
glue code to be placed outside of the Zephyr module repository.
This allows for placing glue code in Zephyr, but also allows users to
specify custom MODULE_EXT_ROOTs for glue code using either
`-DMODULE_EXT_ROOT` or `zephyr/module.yml` with
`build:settings:module_ext_root` settings.
MODULE_EXT_ROOT' is a list of directories, similar to other roots such
as BOARD_ROOT, DTS_ROOT, etc.
The Zephyr repo folder ${ZEPHYR_BASE} is always to the MODULE_EXT_ROOT
list as lowest priority.
For each MODULE_EXT_ROOT, the file
`<module_ext_root>/modules/modules.cmake` will be processed.
In Zephyr repo, the folder `modules/<module>/` contains CMakeLists.txt
and Kconfig glue code for the Zephyr module.
A Zephyr module can specify that CMakeLists.txt and Kconfig glue code is
placed in an external module root by specifying:
```
build:
cmake-ext: True
kconfig-ext: True
```
It is still possible to place the CMakeLists.txt and Kconfig files
directly in the Zephyr module using the existing:
```
build:
cmake: <path>
kconfig: <file>
```.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Fixes: #29235
This commit ensures that roots are being converted to posix paths.
This fixes the issue where windows style path (containing `\`) would
result in DTS dependency file to contain mixed style path separator and
thus causing Ninja to re-invoke CMake in an endless loop.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This commit improves Zephyr modules support in CMake.
All Zephyr modules will now have a corresponding
`ZEPHYR_${module_name}_MODULE_DIR` set, regardless of whether they
define a cmake folder in module.yml.
This results in:
`ZEPHYR_${module_name}_MODULE_DIR` defined for all modules and refers to
the root of the modle.
`ZEPHYR_${module_name}_CMAKE_DIR` defined for modules that specifies
cmake in module.yml, or is having both a CMakeLists.txt and Kconfig file
in the zephyr folder.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This commit has no functional changes but is simply a cleanup so that
flake8 will pass without warnings.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Fixes: #25215
This commit introduces the possibility of defining ROOTs in a Zephyr
module and have it automatically appended to list of other ROOTs.
Supported with this commit:
- BOARD_ROOT
- SOC_ROOT
- DTS_ROOT
- ARCH_ROOT
In order to support this in Zephyr module files, the detection of west
has been moved to dedicated west.cmake file and included immediately
after python.cmake.
Also the inclusion of zephyr_modules.cmake has moved before first use
of BOARD_ROOT.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The projects variable needs to be initialized with the list of modules
provided via cmd-line arguments.
This is a regression introduced by
ef3c5e5516.
Fixes#26948.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
When running CMake, then Python3 will be used.
This is detected through FindPython3, with a preference for using the
python or python3 in path, if any of those matches the required Python
minimal version in Zephyr.
It is also possible for users to specify a different Python, as example
by using:
`cmake -DPYTHON_PREFER=/usr/bin/python3.x`
However, when running `west` as native command, then west will be
invoked on linux based on the python defined in:
`west` launcher, which could be: `#!/usr/bin/python3.y`
Thus there could be mismatch in Pythons used for `west` and the python
used for other scripts.
This is even worse on windows, where a user might experience:
```
>.\opt\bin\Scripts\west.exe --version
Traceback (most recent call last):
File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
...
File "C:\Python37\lib\socket.py", line 49, in <module>
import _socket
ImportError: Module use of python38.dll conflicts with this version of
Python.
```
when testing out a newer Python, but the python in path is still a 3.7.
By importing `west` into zephyr_module.py and by using, as example
`python -c "from west.util import west_topdir; print(topdir())"`
we ensure the same python is used in all python scripts.
Also it allows the user to control the python to use for west.
It also ensures that the west version being tested, is also the version
being used, where old code would test the version imported by python,
but using the west in path (which could be a different version)
If the west version installed in the current Python, and west invocation
is using a different Python interpreter, then an additional help text
is printed, to easier assist users with debugging.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This commit introduces the possibility of adding dependencies between
Zephyr modules to ensure that a Zephyr module on which other modules
depends is processed first.
The dependency chain is ordered using a topological sort.
This allows to add dependencies to a zephyr/module.yml as:
build:
cmake: .
depends:
- fatfs
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
West utilizes Zephyr base when invoked out-of-tree in order to determine
west topdir.
This commit ensures that zephyr base when invoking west from CMake is
set to current zephyr base.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The west command line output is not stable and has changed for 0.7.
Match it in zephyr_module.py's check for whether we are in a
workspace.
The real fix is to start using 'west topdir' whenever the west version
is at least 0.7.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add error checking in zephyr_module.py so that if the user manually
specifies ZEPHYR_EXTRA_MODULES and the list contains something that
isn't in fact a valid module, we scream and die.
This should help with diagnosing module errors.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It is supported to have a zephyr module that does not have a
module.yml, but zephyr_module.py does not support it and will drop
such modules.
To fix this we add support in zephyr_module.py.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Pieter De Gendt
Yangbo Lu
Andy Lee
Johan Hedberg
Christophe Dufaza
Thomas Gagneret
Lukasz Mrugala
Jamie McCrae
Flavio Ceolin
Jack Rosenthal
Martí Bolívar
Torsten Rasmussen
Carles Cufi
Tristan Honscheid
Alexander Mihajlovic
Damian Krolik
Trond Einar Snekvik
Marcin Niestroj
Andrzej Głąbek
Anas Nashif
Sebastian Bøe