Previously, dtlib would fail to parse the following:
/delete-node/ &{/};
This is accepted by dtc, so dtlib should be aligned.
The expected behavior is that the contents of the "deleted" root node
are emptied, but the node itself remains in the tree. This means that
it's possible to put that statement at the end of a DTS file and still
get a valid output. A small test case for this scenario is included.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
This helper lets you place a node (really the entire subtree rooted at
that node) elsewhere in the devicetree. This will be useful when
adding system devicetree support, when we'll want to be able to, for
example, move the CPU cluster node selected by the current execution
domain to /cpus.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Introduce a context manager that will save some typing
when dealing with expected exceptions.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The standard library copy module allows you to implement shallow and
deep copies of objects. See its documentation for more details on
these terms.
Implementing copy.deepcopy() support for DT objects will allow us to
"clone" devicetree objects in other classes. This in turn will enable
new features, such as native system devicetree support, within the
python-devicetree.
It is also a pure feature extension which can't harm anything and is
therefore safe to merge now, even if system devicetree is never
adopted in Zephyr.
Note that we are making use of the move from OrderedDict to regular
dict to make this implementation more convenient.
See https://github.com/devicetree-org/lopper/ for more information on
system devicetree. We want to add system devicetree support to dtlib
because it seems to be a useful way to model modern, heterogeneous
SoCs than traditional devicetree, which can really only model a single
CPU "cluster" within such an SoC.
In order to create 'regular' devicetrees from a system devicetree, we
will want a programming interface that does the following:
1. parse the system devicetree
2. receive the desired transformations on it
3. perform the desired transformations to make
a 'regular' devicetree
Step 3 can be done as a destructive modification on an object-oriented
representation of a system devicetree, and that's the approach we will
take in python-devicetree. It will therefore be convenient to have an
efficient deepcopy implementation to be able to preserve the original
system devicetree and the derived regular devicetree in memory in the
same python process.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Attempts to define two nodes with the same name within a single set of
curly brackets should fail.
For example, this is invalid DTS according to dtc:
/ { foo {}; foo {}; };
By contrast, this is valid since the node named 'foo' appears twice in
two different sets of curly brackets:
/ { foo {}; };
/ { foo {}; };
Zephyr's dtlib currently does not error out on the invalid condition.
Now that Zephyr itself has been updated to not include such nodes (to
the best of my ability), we can fix this divergence from current dtc
behavior and add a regression test in dtlib.
Fixes: #49590
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Node names are subject to the rules in table 2.1 of the devicetree
specification v0.3, while properties are subject to rules in table
2.2. These rules mean that some property names are invalid node names.
However, the same regular expression is being used to validate the
names of nodes and properties in dtlib. This leads to invalid node
names being allowed to pass. Fix this issue by moving the node name
handling code to the Node constructor and checking against the
characters in table 2.1.
The test cases claim that the existing behavior matches dtc. I can't
reproduce that. I get errors when I use invalid characters (like "?")
in a node name. For example:
foo.dts:3.8-11: ERROR (node_name_chars): /node?: Bad character '?' in
node name
Try to make the dtlib error message reminiscent of that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As a first step towards being more forgiving on invalid inputs, allow
string-valued aliases properties that do not point to valid nodes when
the user requests permissiveness.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The documentation says DT.__init__ takes any iterable for the
include_path, but this leads to bad results when you pass it something
other than a 'real' sequence (list/tuple/etc), like a generator:
>>> dt = DT('/tmp/foo.dts', (x for x in ['a', 'b', 'c']))
>>> repr(dt)
"DT(filename='/tmp/foo.dts', include_path=<generator object ...>)"
Make a copy in list form just to avoid things like this.
Add a test for this and relax the regular expression in the existing
test case related to this.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Instead of hard-coding constants, use an IntEnum.
These is still a subclass of 'int', but is both easier to import and
easier to read during debugging.
For example, compare:
>>> Type.BYTES
<Type.BYTES: 1>
with:
>>> TYPE_BYTES
1
However, 'Type.BYTES == 1' is still True, and the enum values
otherwise behave like you would expect.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We are now in the process of extracting edtlib and dtlib into a
standalone source code library that we intend to share with other
projects.
Links related to the work making this standalone:
https://pypi.org/project/devicetree/https://python-devicetree.readthedocs.io/en/latest/https://github.com/zephyrproject-rtos/python-devicetree
This standalone repo includes the same features as what we have in
Zephyr, but in its own 'devicetree' python package with PyPI
integration, etc.
To avoid making this a hard fork, move the code that's being made
standalone around in Zephyr into a new scripts/dts/python-devicetree
subdirectory, and handle the package and sys.path changes in the
various places in the tree that use it.
From now on, it will be possible to update the standalone repository
by just recursively copying scripts/dts/python-devicetree's contents
into it and committing the results.
This is an interim step; do NOT 'pip install devicetree' yet.
The code in the zephyr repository is still the canonical location.
(In the long term, people will get the devicetree package from PyPI
just like they do the 'yaml' package today, but that won't happen for
the foreseeable future.)
This commit is purely intended to avoid a hard fork for the standalone
code, and no functional changes besides the package structure and
location of the code itself are expected.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use the pytest test framework in the dtlib.py and edtlib.py test
suites (testdtlib.py and testedtlib.py respectively).
The goal here is not to change what is being tested. The existing test
suite is excellent and very thorough.
However, it is made up of executable scripts where all of the tests
are run using a hand-rolled framework in a single function per file.
This is a bit all-or-nothing and prevents various nice features
available in the de-facto standard pytest test framework from being
used.
In particular, pytest can:
- drop into a debugger (pdb) when there is a problem
- accept a pattern which specifies a subset of tests to run
- print very detailed error messages about the actual and expected
results in various traceback formats from brief to very verbose
- gather coverage data for the python scripts being tested (via plugin)
- run tests in parallel (via plugin)
- It's easy in pytest to run tests with temporary directories
using the tmp_path and other fixtures. This us avoid
temporarily dirtying the working tree as is done now.
Moving to pytest lets us leverage all of these things without any loss
in ease of use (in fact, some things are nicer in pytest):
- Any function that starts with "test_" is automatically picked up and
run. No need for rolling up lists of functions into a test suite.
- Tests are written using ordinary Python 'assert'
statements.
- Pytest magic unpacks the AST of failed asserts to print details on
what went wrong in really nice ways. For example, it will show you
exactly what parts of two strings that are expected to be equal
differ.
For the most part, this is a pretty mechanical conversion:
- extract helpers and test cases into separate functions
- insert temporary paths and adjust tests accordingly to not match
file names exactly
- use 'assert CONDITION' instead of 'if not CONDITION: fail()'
There are a few cases where making this happen required slightly
larger changes than that, but they are limited.
Move the checks from check_compliance.py to a new GitHub workflow,
removing hacks that are no longer needed.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
API oversight. This was meant to be there all along together with
Device.parent, for navigating the devicetree, but since a need for it
never came up in gen_defines.py, it got overlooked.
Devices are just devicetree nodes augmented with binding information and
some interpretation of devicetree properties. I wonder if the name
should be changed to something like edtlib.Node to make that clearer.
Calling something like a flash partition a "device" is a bit weird, as
Galak pointed out.
I think I went with Device originally to avoid confusion with
dtlib.Node, but since edtlib users don't directly interact with dtlib,
it might not be that confusing in practice.
Piggyback some documentation clarifications.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Similar to edtlib._err(). Just saves a bunch of 'raise DTError'.
Also add tests for some errors from the global to_num() and to_nums()
functions that were untested.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
The public DT.get_node() function was used during parsing to look up
paths in references like &{/foo/bar}, along with an ugly
'DT._is_parsing' flag to adapt its behavior (to not mention aliases in
error messages).
Split out common node lookup code needed during parsing and by
get_node() instead, and stop using get_node() during parsing. This
allows '_is_parsing' to be removed and untangles things a bit.
Piggyback some other small reference-related cleanups, and fix an issue
with the filename/linenr being given twice in some error messages.
This commit also removes the index of path components from error
messages, but just the string is probably good enough.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Add two new type-checked property types 'phandles' and 'phandle-array'
to edtlib.
'phandles' is for pure lists of phandles, with no other data, like
foo = < &bar &baz ... >
'phandle-array' is for lists of phandles and (possibly) numbers, like
foo = < &bar 1 2 &baz 3 4 ... >
dt-schema also has the 'phandle-array' type.
Property.val (in edtlib) is set to an array of Device objects for the
'phandles' type.
For the 'phandle-array' type, no Property object is created. This type
is only used for type checking.
Also refactor how types that do not create a Property object
('phandle-array' and 'compound') are handled. Have _prop_val() return
None for them.
The new types are implemented with two new TYPE_PHANDLES and
TYPE_PHANDLES_AND_NUMS types at the dtlib level. There is also a new
Property.to_nodes() functions for fetching the Nodes for an array of
phandles, with type checking.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Previously, Property.to_node() would allow assignments like
x = < 1 >;
as long as 1 happened to be a valid phandle. This was deliberate, but
might hide errors, and would make the planned 'phandles' (list of
phandles) and 'phandle-array' (list of phandles and numbers) types a bit
too similar to 'type: array'.
Change Property.to_node() to only accept
x = < &foo >;
This is probably all we need, and if you really need to accept manually
specified phandles, it can be worked around in other ways.
Piggyback some consistency nits in error messages from the
Property.to_*() functions.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
dtlib.py and guiconfig.py do some hackery to build token and image
variable names, which triggers spurious pylint warnings like
scripts/dts/dtlib.py:243:28: E0602: Undefined variable '_T_LABEL'
(undefined-variable)
Suppress the warning for those files. The generated names get used in
lots of places.
Also suppress some warnings in doc/conf.py ('tags' is from Sphinx), and
fix a legitimate issue in scripts/dts/testdtlib.py.
This pylint check is useful enough to want enabled in the upcoming CI
check.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This was allowed due to a misunderstanding:
foo = 'x';
In reality, 'x' works like an integer literal, and is used like this:
foo = < 'x' >;
Fix character literal parsing to match the C tools.
Also fix backslash escape parsing to match the C tools exactly
(get_escape_char() in util.c): \<char> should be turned into <char> if
<char> isn't recognized as a special escape character, instead of being
left alone. This fixes parsing of e.g. '\'' (a character literal with a
single quote in it).
Piggyback some more tests for weird property/node names.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Property type-checking has been pretty rudimentary until now, only
checking things like the length being divisible by 4 for 'type: array',
and strings being null-terminated. In particular, no checking was done
for 'type: uint8-array', letting
jedec-id = < 0xc8 0x28 0x17 >;
slip through when
jedec-id = [ 0xc8 0x28 0x17 ];
was intended.
Fix it by adding a syntax-based type checker:
1. Add Property.type, which gives a high-level type for the property,
derived from the markers added in the previous commit.
This includes types like TYPE_EMPTY ('foo;'),
TYPE_NUM ('foo = < 3 >;'), TYPE_BYTES ('foo = [ 01 02 ];'),
TYPE_STRINGS ('foo = "bar", "baz"'),
TYPE_PHANDLE ('foo = < &bar >;'), and TYPE_COMPOUND (everything not
recognized).
See the Property.type docstring in dtlib for more info.
2. Use the high-level type in
Property.to_num()/to_string()/to_node()/etc. to verify that the
property was assigned in an expected way for the type.
If the assignment looks bad, give a helpful error:
expected property 'nums' on /foo/bar in some.dts to be assigned
with 'nums = < (number) (number) ... >', not 'nums = "oops";'
Some other related changes are included as well:
- There's a new Property.to_bytes() function that works like accessing
Property.bytes, except with an added check for the value being
assigned like 'foo = [ ... ]'.
This function solves problems like the jedec-id one.
- There's a new Property.to_path() function for fetching the
referenced node for assignments like 'foo = &node;', with type
checking. (Strings are accepted too, as long as they give the path
to an existing node.)
This function is used for /chosen and /aliases.
- A new 'type: phandle' type can now be given in bindings, for
properties that are assigned like 'foo = < &node >;'.
- Property.__str__() now displays phandles and path references as they
were written (e.g. '< &foo >' instead of '< 0x1 >', if the
allocated phandle happened to be 1).
- Property.to_num() and Property.to_nums() no longer take a 'length'
parameter, because it makes no sense with the type checking.
- The global dtlib.to_string() and dtlib.to_strings() functions were
removed, because they're not that useful.
- More tests were added, along with misc. minor cleanup in various
places.
- Probably other stuff I forgot.
The more strict type checking in dtlib indirectly makes some parts of
edtlib more strict as well (wherever Property.to_*() is used).
Fixes: #18131
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Previously, dtlib just stored the raw 'bytes' value for each property,
along with some markers in Property._markers for phandle and path
references.
Extend Property._markers to also remember where different data blocks
start, so that e.g.
foo = <1 2 3>, "bar", [00 01];
can be reproduced as written.
Use the new information to reproduce properties as written in
Property.__str__(). This gives good test coverage as well, since the
test suite checks literal __str__() output.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Add a new DTS/binding parser to scripts/dts/ for generating
generated_dts_board.conf and generated_dts_board_unfixed.h.
The old code is kept to generate some deprecated defines, using the
--deprecated-only flag. It will be removed later.
The new parser is implemented in three files in scripts/dts/:
dtlib.py:
A low-level .dts parsing library. This is similar to devicetree.py in
the old code, but is a general robust DTS parser that doesn't rely on
preprocessing.
edtlib.py (e for extended):
A library built on top of dtlib.py that brings together data from DTS
files and bindings and creates Device instances with all the data for
a device.
gen_defines.py:
A script that uses edtlib.py to generate generated_dts_board.conf and
generated_dts_board_unfixed.h. Corresponds to extract_dts_includes.py
and the files in extract/ in the old code.
testdtlib.py:
Test suite for dtlib.py. Can be run directly as a script.
testedtlib.py (uses test.dts and test-bindings/):
Test suite for edtlib.py. Can be run directly as a script.
The test suites will be run automatically in CI.
The new code turns some things that were warnings (or not checked) in
the old code into errors, like missing properties that are specified
with 'category: required' in the binding for the node.
The code includes lots of documentation and tries to give helpful error
messages instead of Python errors.
Co-authored-by: Kumar Gala <kumar.gala@linaro.org>
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Lingao Meng
Grzegorz Swiderski
Martí Bolívar
Ulf Magnusson