When using the code and data relocation feature, every relocated symbol
would be marked with `KEEP()` in the generated linker script. Therefore,
if any input files contained unused code, then it wouldn't be discarded
by the linker, even when invoked with `--gc-sections`.
This can cause unexpected bloat, or other link-time issues stemming from
some symbols being discarded and others not.
On the other hand, this behavior has been present since the feature's
introduction, so it should remain default for the users who rely on it.
This patch introduces support for `zephyr_code_relocate(... NOKEEP)`.
This will suppress the generation of `KEEP()` statements for all symbols
in a particular library or set of files.
Much like `NOCOPY`, the `NOKEEP` flag is passed to `gen_relocate_app.py`
in string form. The script is now equipped to handle multiple such flags
when passed from CMake as a semicolon-separated list, like so:
"SRAM2:NOCOPY;NOKEEP:/path/to/file1.c;/path/to/file2.c"
Documentation and tests are updated here as well.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
For applications relocating big parts of the code with many sections,
builds were failing on Windows due to hitting the max command-line
length on that platform.
Fix this by using a file to store the dictionary passed to the python
script.
Fixes https://github.com/zephyrproject-rtos/zephyr/issues/60994.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Use the "device_deps" naming scheme to emphasize we are storing device
dependencies. The fact we are using device handles to store them is an
implementation detail.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
The code_relocation feature creates generic section names that sometimes
conflict with already existing names.
This patch adds a '_reloc_' word to the created names to reduce the risk
of conflict.
This solves #54785.
Signed-off-by: Björn Stenberg <bjorn@haxx.se>
This refactors the script to introduce a SectionKind enum that
represents each of the kinds of sections handled by this system (text,
data, rodata or bss), using that to improve the code structure by
reducing the use of strings in favor of indicating the use of values by
their types.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
When using code_data_relocation the script identifies sections that
should be relocated, but cannot guarantee that section names are unique
across all linked files. If a section name is not unique then matching
all sections with that name across all input files will relocate more
data than intended.
As a simple example, if both file1 and file2 contain .rodata.strings
sections, then if gen_relocate_app inspects only file1 it will generate
a linker script fragment `*(.rodata.strings)` that matches both object
files.
This commit changes gen_relocate_app to make the linker match on object
files as well (`*file1.o(.rodata.strings)`) to ensure unwanted sections
don't get relocated.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
Disables allowing the python argparse library from automatically
shortening command line arguments, this prevents issues whereby
a new command is added and code that wrongly uses the shortened
command of an existing argument which is the same as the new
command being added will silently change script behaviour.
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
Update gen_relocate_app.py to use "|" to separate code relocation
directives, and ";" to separate multiple files in a relocation directive.
This will enable multiple files to be passed to zephyr_code_relocate,
as well as multiple files to be passed from a CMake generator expression.
The script will then seperate these files and relocate each according to
the arguments given to zephyr_code_relocate.
Note! This commit will break support for zephyr_code_relocate until
the CMake function is updated
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
This is a follow-up to commit b1a3ce4016.
`parse_input_string()` needs to use `:NOCOPY:`, not `:NOCOPY`, when
partitioning input lines, otherwise, when a line contains the NOCOPY
flag, the file name returned by the function starts with `:` and the
file cannot be then found by the script.
Such problem can be observed in the code_relocation_nocopy sample,
which without this fix does not actually relocate code from ext_code.c.
Signed-off-by: Andrzej Głąbek <andrzej.glabek@nordicsemi.no>
Some linker scripts also associate memory regions with program headers
(PHDRS [1]). This patch adds support on gen_relocate_app.py to also
place a phdr alongside the memory region.
To keep things simple (and more natural), the format is basically the
same used in linker scripts - an space followed by ':<phdr_name>', like:
SRAM2\ :phdr0:COPY:/home/xyz/zephyr/samples/hello_world/src/main.c
(Note the escape char before the space.)
[1] https://sourceware.org/binutils/docs/ld/PHDRS.html
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
As of today <zephyr/zephyr.h> is 100% equivalent to <zephyr/kernel.h>.
This patch proposes to then include <zephyr/kernel.h> instead of
<zephyr/zephyr.h> since it is more clear that you are including the
Kernel APIs and (probably) nothing else. <zephyr/zephyr.h> sounds like a
catch-all header that may be confusing. Most applications need to
include a bunch of other things to compile, e.g. driver headers or
subsystem headers like BT, logging, etc.
The idea of a catch-all header in Zephyr is probably not feasible
anyway. Reason is that Zephyr is not a library, like it could be for
example `libpython`. Zephyr provides many utilities nowadays: a kernel,
drivers, subsystems, etc and things will likely grow. A catch-all header
would be massive, difficult to keep up-to-date. It is also likely that
an application will only build a small subset. Note that subsystem-level
headers may use a catch-all approach to make things easier, though.
NOTE: This patch is **NOT** removing the header, just removing its usage
in-tree. I'd advocate for its deprecation (add a #warning on it), but I
understand many people will have concerns.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This implements support for relocating code to chosen memory regions via
the `zephyr_code_relocate` CMake function for RISC-V SoCs. ARM-specific
assumptions that were made by gen_relocate_app.py need to be corrected,
in particular not assuming any particular name for the default RAM
section (which is 'SRAM' for most ARM pltaforms) and not assuming 32-bit
pointers (so the test works on RV64).
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
Most ARM platforms name their ROM region `FLASH`, but this assumption is
not portable. Have gen_relocate_app refer to ROMABLE_REGION instead of
hard-coding the platform-specific memory region name.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
Only the elftools SymbolTableSection section type provides an
iter_symbols() method, and compilers are free to emit sections that
include the substring '.symtab' in their name which causes errors if
gen_relocate_app examines only the section name. Instead check whether a
section is a symbol table to skip attempting to inspect sections that
are not actually symbol tables.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
Move scripts needed by the build system and not designed to be run
individually or standalone into the build subfolder.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
COPY/NOCOPY flag was added to the end of the string with ':'
as separators. The python script then split the line on ':'
which breaks on Windows build because the string was
[MEM]:[FILE]:[COPY]
In windows you will have 'c:\' in the file path so the
line.split() in script will split on the 'c:'.
Move the COPY/NOCOPY to:
[MEM]:[COPY/NOCOPY]:[FILE]
Note that the comments at top of gen_relocate_app.py also
indicates this format.
Fixes#43950
Signed-off-by: David Leach <david.leach@nxp.com>
* Use case of interest:
Some platforms are shipping in parallel to the internal FLASH some other
storage / external FLASH (usually a QSPI FLASH) that can be used to
execute (XIP) code from.
The content of this external FLASH can usually be written at flash time
using proper tools (see for example the case of the external FLASH on
the nRF5340DK that can be written at flash time using nrfjprog).
The external FLASH is a nice addition that is extremely useful when a
large application code doesn't entirely fit on the internal FLASH so
that we could want to move part of it in the auxiliary FLASH to XIP the
code from there.
* The problem:
Right now Zephyr doesn't have a formal and generic way to move at build
time part of the code to a different memory region.
* The current status:
Zephyr is indeed shipping a code_relocation feature but that doesn't
entirely match our needs.
When XIP is enabled, the code_relocation feature is used in Zephyr to
move the selected code (that is to copy text section, to initialize data
and zero bss) from FLASH to SRAM at run time and execute code from SRAM.
The relocation is done by a generated snippet of code that is
memcpy()-ing the right content to the destination region also using some
build-time generated portions of linker script to retrieve start and
destination addresses and regions.
* This patch:
This patch is leveraging the code_relocation code and adding a NOCOPY
feature. This feature is using the code_relocation feature to
dynamically build the linker script snippets but entirely skipping the
run-time code relocation so that the code can be XIP-ed from the
destination region.
* Example:
Let's say that we have a big file called `huge_file.c` that we want to
XIP from the external FLASH that is mapped in an EXTFLASH region.
In this case we should enable `CONFIG_XIP` and
`CONFIG_CODE_DATA_RELOCATION` and instruct cmake as follows:
zephyr_code_relocate(src/huge_file.c EXTFLASH_TEXT NOCOPY)
zephyr_code_relocate(src/huge_file.c SRAM_DATA)
this means that:
- The .text section of the `huge_file.c` must reside in the EXTFLASH
memory region and we do not need to copy the section there because we
are going to XIP it (and we assume that the file is going to be placed
in the external FLASH at flash time).
- The .data section of the `huge_file.c` must still reside in the SRAM
memory region.
* TODOs:
It's desirable to have the possibility to relocate libraries and
pre-build files instead of source code files.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Zeroing the BSS and copying data to RAM with regular memset/memcpy may
cause problems when those functions are assuming a fully initialized
system for their optimizations to work e.g. some instructions require
an active MMU, but turning the MMU on needs the .bss section to be
cleared first, etc.
Commit c5b898743a ("aarch64: Fix alignment fault on z_bss_zero()")
provides a detailed explanation of such a case.
Replacing z_bss_zero() with an architecture specific one is problematic
as the former may see new sections added to it that would be missed by
the later. The same reasoning goes for z_data_copy().
Let's make maintenance much easier by providing weak versions of
memset/memcpy that can be overridden by architecture-specific safe
versions when needed.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The problem with append is that when doing incremental (non-clean)
build, the content of these files are appended each time to the previous
leftover content.
Example:
# west build -p -b qemu_cortex_m3 \
samples/application_development/code_relocation/
# // disable for example CONFIG_XIP
# west build -b qemu_cortex_m3 \
samples/application_development/code_relocation/
at this point the content of the generated files (i.e.
linker_relocate.ld) is wrong.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Check each line whether contains the correct pattern
${location}:${file} before splitting because it
turns out that script passes also compile
definitions added in project CMakeLists.txt such as:
add_compile_definitions(ABCD="XYZ")
which results in the build failure.
Signed-off-by: Bartosz Bilas <b.bilas@grinn-global.com>
As pylint says "Consider iterating the dictionary directly
instead of calling .keys()" don't use .keys() method.
Signed-off-by: Bartosz Bilas <bartosz.bilas@hotmail.com>
With the addition of #34185, it is not longer gauranteed that for a
memory region `NAME` there exists a symbol `NAME_ADDR`. Use the linker
builtin function `ORIGIN` to instead directly get the start address of
the selected memory region.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Fixes: #28847
This commit fixes two places that was causing gen_relocate_app to fail
in windows.
gen_relocate_app.py now splits only on first `:` in
`<MEM_REGION>:<file>`.
Windows contains `:` in path, thus only first `:` is valid for splitting
mem region and file path.
Second part of the issue is fixed in CMake where `'` was used for
quoting of command arguments.
This causes a file not found on Windows because the final `'` would be
treated as part of the filename. Similar the first `'` would be treated
as path of the mem region name.
This is fixed by using `"` for quoting, which works correctly on all
platforms.
gen_relocate_app.py:403: UserWarning: File: .../kernel/sem.c' Not found
Note the stray `'` ^^^
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
We have a collection of python scripts that are part of our build
system. This PR collects docstring comments added to these scripts into
a summary document. Previous references to just the script name in
other documentation are updated to point to this build tool
documentation.
Some of the scripts needed an update to be processed (via include
directives) consistently.
Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit refactors kernel and arch headers to establish a boundary
between private and public interface headers.
The refactoring strategy used in this commit is detailed in the issue
This commit introduces the following major changes:
1. Establish a clear boundary between private and public headers by
removing "kernel/include" and "arch/*/include" from the global
include paths. Ideally, only kernel/ and arch/*/ source files should
reference the headers in these directories. If these headers must be
used by a component, these include paths shall be manually added to
the CMakeLists.txt file of the component. This is intended to
discourage applications from including private kernel and arch
headers either knowingly and unknowingly.
- kernel/include/ (PRIVATE)
This directory contains the private headers that provide private
kernel definitions which should not be visible outside the kernel
and arch source code. All public kernel definitions must be added
to an appropriate header located under include/.
- arch/*/include/ (PRIVATE)
This directory contains the private headers that provide private
architecture-specific definitions which should not be visible
outside the arch and kernel source code. All public architecture-
specific definitions must be added to an appropriate header located
under include/arch/*/.
- include/ AND include/sys/ (PUBLIC)
This directory contains the public headers that provide public
kernel definitions which can be referenced by both kernel and
application code.
- include/arch/*/ (PUBLIC)
This directory contains the public headers that provide public
architecture-specific definitions which can be referenced by both
kernel and application code.
2. Split arch_interface.h into "kernel-to-arch interface" and "public
arch interface" divisions.
- kernel/include/kernel_arch_interface.h
* provides private "kernel-to-arch interface" definition.
* includes arch/*/include/kernel_arch_func.h to ensure that the
interface function implementations are always available.
* includes sys/arch_interface.h so that public arch interface
definitions are automatically included when including this file.
- arch/*/include/kernel_arch_func.h
* provides architecture-specific "kernel-to-arch interface"
implementation.
* only the functions that will be used in kernel and arch source
files are defined here.
- include/sys/arch_interface.h
* provides "public arch interface" definition.
* includes include/arch/arch_inlines.h to ensure that the
architecture-specific public inline interface function
implementations are always available.
- include/arch/arch_inlines.h
* includes architecture-specific arch_inlines.h in
include/arch/*/arch_inline.h.
- include/arch/*/arch_inline.h
* provides architecture-specific "public arch interface" inline
function implementation.
* supersedes include/sys/arch_inline.h.
3. Refactor kernel and the existing architecture implementations.
- Remove circular dependency of kernel and arch headers. The
following general rules should be observed:
* Never include any private headers from public headers
* Never include kernel_internal.h in kernel_arch_data.h
* Always include kernel_arch_data.h from kernel_arch_func.h
* Never include kernel.h from kernel_struct.h either directly or
indirectly. Only add the kernel structures that must be referenced
from public arch headers in this file.
- Relocate syscall_handler.h to include/ so it can be used in the
public code. This is necessary because many user-mode public codes
reference the functions defined in this header.
- Relocate kernel_arch_thread.h to include/arch/*/thread.h. This is
necessary to provide architecture-specific thread definition for
'struct k_thread' in kernel.h.
- Remove any private header dependencies from public headers using
the following methods:
* If dependency is not required, simply omit
* If dependency is required,
- Relocate a portion of the required dependencies from the
private header to an appropriate public header OR
- Relocate the required private header to make it public.
This commit supersedes #20047, addresses #19666, and fixes#3056.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Promote a handy and often-overlooked sys.exit() feature: Passing it a
string (or any other non-int object) prints it to stderr and exits with
status 1.
See the documentation at
https://docs.python.org/3/library/sys.html#sys.exit.
This indirectly prints some errors to stderr that previously went to
stdout.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Getting slightly subjective, but fixes this pylint warning:
scripts/gen_relocate_app.py:228:38: R1714: Consider merging these
comparisons with "in" to "region in ('data', 'bss')"
(consider-using-in)
Use a set literal instead of a tuple literal, as recent Python 3
versions optimize set literals with constant keys nicely.
Getting rid of pylint warnings for a CI check. I could disable any
controversial ones (it's already a list of warnings to enable anyway).
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Dictionaries are not ordered in Python 3.5 and before, so building twice
${ZEPHYR_BASE}/samples/application_development/code_relocation/
in a row could lead to a different sections order, different
build/zephyr/include/generated/linker_relocate.d and code_relocation.c
and different binaries.
Fix with a minor change to three "for" loops in the output functions:
make them iterate on sorted(list of sections) instead of the raw and
randomly ordered dictionaries of sections.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Found a few annoying typos and figured I better run script and
fix anything it can find, here are the results...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
on non-XIP system, SRAM is the default region, and relocated .data
section and .bss section of SRAM shouldn't be inserted between
_image_rom_start and _image_rom_end, because the memory region between
_image_rom_start and _image_rom_end will construct the mpu ro region.
Also for the newly added memory region on non-XIP system, the
relocated .text secition and .rodata section should also be mpu aligned.
Fixes: #16090.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
Discovered with pylint3.
Use the placeholder name '_' for unproblematic unused variables. It's
what I'm used to, and pylint knows not to flag it.
Python tip:
for i in range(n):
some_list.append(0)
can be replaced with
some_list += n*[0]
Similarly, 3*'\t' gives '\t\t\t'.
(Relevant here because pylint flagged the loop index as unused.)
To do integer division in Python 3, use // instead of /.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
(OPTIONAL) was a vestiage from the initial import of the Zephyr code
base and we dont utilize it with the GNU linker. Additionally, the way
(OPTIONAL) gets defined to nothing creates a linker script that lld
(from llvm) doesn't like.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
This seems to solve the issue with GH-12033. There seems to be
some compiler optimization that was causing this issue. This
occurs only when the build is re-run by invoking cmake without
clearing the previous build.
Fixes:GH-12033
Signed-off-by: Adithya Baglody <adithya.nagaraj.baglody@intel.com>
This script will relocate .text .data and .bss sections from
required files and places it in the required memory region. This
memory region and file are given to this python script in the form
of a string. Refer to the script for the format of this string and
the procedure to invoke it.
The main goal of this script is to provide a robust way to re-order
the memory contents without actually having to modify the code
(C source code and the linker code).
In simple terms this script will do the job of
__attribute__((section("name"))) for a bunch of files together.
Signed-off-by: Varun Sharma <varun.sharma@intel.com>
Signed-off-by: Adithya Baglody <adithya.nagaraj.baglody@intel.com>
Grzegorz Swiderski
Carles Cufi
Gerard Marull-Paretas
Björn Stenberg
Peter Marheine
Carlo Caione
Jamie McCrae
Daniel DeGrasse
Andrzej Głąbek
Ederson de Souza
Anas Nashif
Fabio Baltieri
David Leach
Nazar Kazakov
Nicolas Pitre
Bartosz Bilas
Bartosz Bilas
Mahesh Mahadevan
Jordan Yates
Torsten Rasmussen
Kumar Gala
David B. Kinder
Stephanos Ioannidis
Wentong Wu
Håkon Øye Amundsen
Ulf Magnusson
Marc Herbert
Adithya Baglody