From 05cb5b3ef09049afb0d5d252b5dbdaf1a95c620f Mon Sep 17 00:00:00 2001 From: Flavio Ceolin Date: Mon, 28 Oct 2019 15:02:57 -0700 Subject: [PATCH] docs: guide: Add build system information A detailed overview of Zephyr's build system. This is a thorough view of the low level build process starting from CMake and using Make as the build system tool. Things missing here that will be further documented: - west - external modules/libraries Signed-off-by: Flavio Ceolin --- .gitignore | 1 + doc/guides/build/build-build-phase-1.svg | 3 + doc/guides/build/build-build-phase-2.svg | 3 + doc/guides/build/build-build-phase-3.svg | 3 + doc/guides/build/build-build-phase-4.svg | 3 + doc/guides/build/build-config-phase.svg | 3 + doc/guides/build/index.rst | 155 +++++++++++++++++++++++ doc/guides/index.rst | 1 + 8 files changed, 172 insertions(+) create mode 100644 doc/guides/build/build-build-phase-1.svg create mode 100644 doc/guides/build/build-build-phase-2.svg create mode 100644 doc/guides/build/build-build-phase-3.svg create mode 100644 doc/guides/build/build-build-phase-4.svg create mode 100644 doc/guides/build/build-config-phase.svg create mode 100644 doc/guides/build/index.rst diff --git a/.gitignore b/.gitignore index fb82082db09..e9131429038 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ *.swo *~ build*/ +!doc/guides/build cscope.* .dir diff --git a/doc/guides/build/build-build-phase-1.svg b/doc/guides/build/build-build-phase-1.svg new file mode 100644 index 00000000000..464c591a233 --- /dev/null +++ b/doc/guides/build/build-build-phase-1.svg @@ -0,0 +1,3 @@ + + +

Makefile
(various)

[Not supported by viewer]
Build Stage I (make) – Pre-build
[Not supported by viewer]
offsets.c
[Not supported by viewer]
GNU cc
(cpp, as, ld)
[Not supported by viewer]
offsets.c.obj
[Not supported by viewer]
scripts/gen_offset_header.py
[Not supported by viewer]
offsets.h
[Not supported by viewer]
scripts/gen_syscall_header.py
[Not supported by viewer]
syscall_macros.h
[Not supported by viewer]
scripts/parse_syscalls.py
[Not supported by viewer]
syscalls.json
[Not supported by viewer]

System Headers (*.h)

[Not supported by viewer]
scripts/gen_syscall_header.py
[Not supported by viewer]
syscall_list.h
syscalls/*.h
syscall_dispatch.c
[Not supported by viewer]

System Headers (*.h)

[Not supported by viewer] \ No newline at end of file diff --git a/doc/guides/build/build-build-phase-2.svg b/doc/guides/build/build-build-phase-2.svg new file mode 100644 index 00000000000..44073f36705 --- /dev/null +++ b/doc/guides/build/build-build-phase-2.svg @@ -0,0 +1,3 @@ + + +
kernel/*.c
[Not supported by viewer]
arch/x86/*.c
arch/x86/*.S
[Not supported by viewer]

Makefile
(various)

[Not supported by viewer]
Build Stage II (make) – First-pass binary
[Not supported by viewer]
Other sources
[Not supported by viewer]

Headers (*.h)
Includes outputs from Configuration Phase and Build Stage I (e.g., autoconf.h, syscall_list.h)

[Not supported by viewer]
kernel/*.c.obj
[Not supported by viewer]
arch/x86/*.c.obj
arch/x86/*.S.obj
[Not supported by viewer]
Other *.obj files
[Not supported by viewer]
linker.cmd
[Not supported by viewer]
libkernel.a
[Not supported by viewer]
arch__x86__core.a
[Not supported by viewer]
Other archives
[Not supported by viewer]
GNU ar
GNU ranlib
[Not supported by viewer]
GNU ar
GNU ranlib
[Not supported by viewer]
GNU ar
GNU ranlib
[Not supported by viewer]
GNU cc
(cpp, cc1, as)
[Not supported by viewer]
GNU cc
(cpp, cc1, as)
[Not supported by viewer]
GNU cc
(cpp, cc1, as)
[Not supported by viewer]
*.h
[Not supported by viewer]
*.h
[Not supported by viewer]
GNU ld
[Not supported by viewer]
zephyr_prebuilt.elf
[Not supported by viewer]

Linker scripts (*.ld)
(various, scattered)

[Not supported by viewer]
GNU cpp
[Not supported by viewer]
scripts/gen_app_partitions.py
[Not supported by viewer]
app_smem linker files
[Not supported by viewer] \ No newline at end of file diff --git a/doc/guides/build/build-build-phase-3.svg b/doc/guides/build/build-build-phase-3.svg new file mode 100644 index 00000000000..b37bfde83d8 --- /dev/null +++ b/doc/guides/build/build-build-phase-3.svg @@ -0,0 +1,3 @@ + + +

Makefile
(various)

[Not supported by viewer]
Build Stage III (make) – Final binary
[Not supported by viewer]
zephyr_prebuilt.elf
[Not supported by viewer]
scripts/gen_kobject_list.py
(Find all kernel objects)
[Not supported by viewer]
kobject_hash.gperf
[Not supported by viewer]

otype-to-str.h
otype-to-size.h
kobj-types-enum.h
driver-validation.h

[Not supported by viewer]
GNU gperf
[Not supported by viewer]
kobject_hash_preprocessed.c
[Not supported by viewer]
scripts/process_gperf.py
(Optimize hashing)
[Not supported by viewer]
kobject_hash.c
[Not supported by viewer]
GNU cc
(cpp, cc1, as)
[Not supported by viewer]
kobject_hash.c.obj
[Not supported by viewer]
kobject_hash_renamed.o
[Not supported by viewer]
GNU objdump
[Not supported by viewer]
linker.cmd
[Not supported by viewer]
GNU ld
[Not supported by viewer]
zephyr.elf
[Not supported by viewer]
Other archives from previous page
[Not supported by viewer]
zephyr.map
[Not supported by viewer]

Linker scripts (*.ld)
(various, scattered)

[Not supported by viewer]
GNU cpp
[Not supported by viewer]
app_smem linker files
[Not supported by viewer] \ No newline at end of file diff --git a/doc/guides/build/build-build-phase-4.svg b/doc/guides/build/build-build-phase-4.svg new file mode 100644 index 00000000000..2b38e432c18 --- /dev/null +++ b/doc/guides/build/build-build-phase-4.svg @@ -0,0 +1,3 @@ + + +

Makefile
(various)

[Not supported by viewer]
Build Stage IV (make) – Post-Processing
[Not supported by viewer]
GNU objdump
[Not supported by viewer]
zephyr.elf
[Not supported by viewer]
zephyr.elf
[Not supported by viewer]
GNU objdump
[Not supported by viewer]
zephyr.bin
[Not supported by viewer]
zephyr.hex
[Not supported by viewer] \ No newline at end of file diff --git a/doc/guides/build/build-config-phase.svg b/doc/guides/build/build-config-phase.svg new file mode 100644 index 00000000000..d8488fb11b0 --- /dev/null +++ b/doc/guides/build/build-config-phase.svg @@ -0,0 +1,3 @@ + + +
GNU cpp
[Not supported by viewer]
dtc
[Not supported by viewer]
scripts/dts
(DT postprocessing)
[Not supported by viewer]
parse configuration
(scripts/kconfig)
[Not supported by viewer]

*.dts/dtsi (from board/soc/app/dts)

[Not supported by viewer]

*.yaml in
dts/bindings/

[Not supported by viewer]

Kconfig
(various, scattered)

[Not supported by viewer]
*.dts.pre.tmp
[Not supported by viewer]
*.dts_compiled
[Not supported by viewer]

generated_dts_board_unfixed.h

[Not supported by viewer]
autoconf.h
[Not supported by viewer]
merge fixups
[Not supported by viewer]

dts_fixup.h (from board/SoC/app)

[Not supported by viewer]
generated_dts_board_fixups.h
[Not supported by viewer]

CMakeLists.txt
(various)

[Not supported by viewer]

Makefile
(various)

[Not supported by viewer]

Configuration Phase (cmake)

[Not supported by viewer] \ No newline at end of file diff --git a/doc/guides/build/index.rst b/doc/guides/build/index.rst new file mode 100644 index 00000000000..b9fb2b3d8f2 --- /dev/null +++ b/doc/guides/build/index.rst @@ -0,0 +1,155 @@ +.. _build_overview: + +Build Overview +############## + +The Zephyr build process can be divided into two main phases: a +configuration phase (driven by *CMake*) and a build phase (driven by +*Make* or *Ninja*). We will descibe the build phase using *Make* as +example. + + +Configuration Phase +******************* + +The configuration phase begins when the user invokes *CMake*, +specifying a source application directory and a board target. + +.. figure:: build-config-phase.svg + :align: center + :alt: Zephyr's build configuration phase + :figclass: align-center + :width: 80% + +*CMake* begins by processing the *CMakeLists.txt* file in the application +directory, which refers to the *CMakeLists.txt* file in the Zephyr +top-level directory, which in turn refers to *CMakeLists.txt* files +throughout the build tree (directly and indirectly). Its primary +output is a set of Makefiles to drive the build process, but *CMake* +scripts do some build processing of their own: + +Device tree + Using *cpp*, device-tree specifications (*.dts/.dtsi* files) are + collected from the target’s architecture, SoC, board, and + application directories and compiled with *dtc*. Then the build + tool (scripts/dts) convert this into *.h* files for later + consumption. + +Device tree fixup + Files named *dts_fixup.h* from the target’s architecture, SoC, + board, and application directories are concatenated into a single + *dts_fixup.h*. Its purpose is to normalize constants output in the + previous step so they have the names expected by the source files + in the build phase. + +Kconfig + The build tool reads the *Kconfig* files for the target + architecture, the target SoC, the target board, the target + application, as well as *Kconfig* files associated with subsystems + throughout the source tree. It incorporates the device tree outputs + to allow configurations to make use of that data. It ensures the + desired configuration is consistent, outputs *autoconf.h* for the + build phase. + +Build Phase +*********** + +The build phase begins when the user invokes *make*. Its ultimate +output is a complete Zephyr application in a format suitable for +loading/flashing on the desired target board (*zephyr.elf*, +*zephyr.hex*, etc.) The build phase can be broken down, conceptually, +into four stages: the pre-build, first-pass binary, final binary, and +post-processing. + +Pre-build occurs before any source files are compiled, because during +this phase header files used by the source files are generated. + +Pre-build +========= + +Offset generation + Access to high-level data structures and members is sometimes + required when the definitions of those structures is not + immediately accessible (e.g., assembly language). The generation of + *offsets.h* (by *gen_offset_header.py*) facilitates this. + +System call boilerplate + The *gen_syscall_header.py*, *parse_syscalls.py* and + *gen_syscall_header.py* scripts work together to bind potential + system call functions with their implementations. + +.. figure:: build-build-phase-1.svg + :align: center + :alt: Zephyr's build stage I + :figclass: align-center + :width: 80% + +First-pass binary +================= + +Compilation proper begins with the first-pass binary. Source files (C +and assembly) are collected from various subsystems (which ones is +decided during the configuration phase), and compiled into archives +(with reference to header files in the tree, as well as those +generated during the configuration phase and the pre-build stage). + +If memory protection is enabled, then: + +Partition grouping + The gen_app_partitions.py script scans all the + generated archives and outputs linker scripts to ensure that + application partitions are properly grouped and aligned for the + target’s memory protection hardware. + +Then *cpp* is used to combine linker script fragments from the target’s +architecture/SoC, the kernel tree, optionally the partition output if +memory protection is enabled, and any other fragments selected during +the configuration process, into a *linker.cmd* file. The compiled +archives are then linked with *ld* as specified in the +*linker.cmd*. + +In some configurations, this is the final binary, and the next stage +is skipped. + +.. figure:: build-build-phase-2.svg + :align: center + :alt: Zephyr's build stage II + :figclass: align-center + :width: 80% + +Final binary +============ + +In some configurations, the binary from the previous stage is +incomplete, with empty and/or placeholder sections that must be filled +in by, essentially, reflection. When :ref:`usermode` is enabled: + +Kernel object hashing + The *gen_kobject_list.py* scans the *ELF DWARF* + debug data to find the address of the all kernel objects. This + list is passed to *gperf*, which generates a perfect hash function and + table of those addresses, then that output is optimized by + *process_gperf.py*, using known properties of our special case. + +Then, the link from the previous stage is repeated, this time with the +missing pieces populated. + +.. figure:: build-build-phase-3.svg + :align: center + :alt: Zephyr's build stage III + :figclass: align-center + :width: 80% + + +Post processing +=============== + +Finally, if necessary, the completed kernel is converted from *ELF* to +the format expected by the loader and/or flash tool required by the +target. This is accomplished in a straightforward manner with *objdump*. + +.. figure:: build-build-phase-4.svg + :align: center + :alt: Zephyr's build final stage + :figclass: align-center + :width: 80% diff --git a/doc/guides/index.rst b/doc/guides/index.rst index 3b0c5ec8f2a..22b484dc3a6 100644 --- a/doc/guides/index.rst +++ b/doc/guides/index.rst @@ -30,3 +30,4 @@ User and Developer Guides tracing/index west/index optimizations/index + build/index